Forklift control method and device, electronic equipment and computer readable storage medium

Abstract

The invention discloses a forklift control method and device, electronic equipment and a computer readable storage medium. The method relates to the field of vehicle control and comprises the steps that environmental data in front of a forklift are obtained through a plurality of different types of sensors installed on the forklift, and the different types of sensors at least comprise a laser radar sensor and a visual sensor; the method comprises the following steps: carrying out tray slot detection based on environmental data to obtain a first distance between a gear shaper of a forklift and a tray slot and first deviation information of the gear shaper and the tray slot, carrying out goods and shelf vacancy detection based on the environmental data to obtain second deviation information of goods and shelf vacancy, and carrying out obstacle identification based on the environmental data. Obtaining a second distance between the forklift and the obstacle; and the forklift is adjusted based on the first distance, the deviation information and the second distance. The technical problem that due to the fact that the tray cannot be accurately detected, the safety of a forklift in the tray inserting and taking operation process is low is solved.

Description

Technical Field

[0001] The present invention relates to the field of vehicle control, and in particular to a forklift control method, device, electronic equipment and computer-readable storage medium. Background Art

[0002] In modern logistics and warehousing industries, forklifts are indispensable equipment for logistics operations. With the continuous development of automation technology, the automation and intelligence of forklifts are also constantly improving, and the pallet detection system on forklifts is one of the key technologies to realize the automation and intelligence of forklifts. The pallet detection system needs to obtain the environmental information in front of the forklift in real time and accurately to determine the position and status of the pallet, so as to guide the insertion and extraction operation of the forklift. At present, the main technologies for detecting pallets are machine vision technology, millimeter wave radar, and ultrasonic radar technology. Machine vision technology obtains image information through a camera and uses image processing algorithms to identify and locate objects. It has the advantages of low cost and easy deployment. Millimeter wave radar can sense the relative distance and speed of objects in complex environments by emitting electromagnetic waves and receiving reflected signals. It has the advantages of strong anti-interference ability and high adaptability. Ultrasonic radar measures the distance of objects by emitting sound waves and receiving echoes, which is suitable for close-range detection. However, existing pallet detection methods also have problems in accuracy and safety. For example, if the camera is interfered by strong light or the lens is blocked, the pallet detection system may not be able to correctly identify the pallet, causing the inserter to accidentally push the shelf down; the ultrasonic radar may cause too large an error due to external interference, which may lead to the inability to correctly identify obstacles and increase the risk of forklift safety. This leads to a technical problem that the forklift is less safe during the operation of inserting and removing pallets due to the inability to accurately detect the pallet.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] The embodiments of the present invention provide a forklift control method, device, electronic device and computer-readable storage medium to at least solve the technical problem that the forklift has low safety during the operation of inserting and removing pallets due to the inability to accurately detect the pallet.

[0005] According to one aspect of an embodiment of the present invention, a forklift control method is provided, comprising: acquiring environmental data in front of the forklift through a plurality of different types of sensors installed on the forklift, the plurality of different types of sensors comprising at least: a lidar sensor and a visual sensor; performing pallet slot detection based on the environmental data to obtain a first distance between a forklift's tooth and the pallet slot and first deviation information between the tooth and the pallet slot; performing cargo and shelf space detection based on the environmental data to obtain second deviation information between the cargo and shelf space; and performing obstacle identification based on the environmental data to obtain a second distance between the forklift and the obstacle; and adjusting the forklift based on the first distance, the deviation information, and the second distance.

[0006] Optionally, pallet slot detection is performed based on environmental data to obtain a first distance between the forklift's tooth and the pallet slot and first deviation information between the tooth and the pallet slot, cargo and shelf space detection is performed based on environmental data to obtain second deviation information between the cargo and shelf space, and obstacle identification is performed based on environmental data to obtain a second distance between the forklift and the obstacle, including: determining the position information of the forklift's tooth based on the forklift's position information; pallet slot detection is performed based on environmental data to obtain position information of the pallet slot, cargo and shelf space detection is performed based on environmental data to obtain position information of the cargo and position information of the shelf space, and obstacle identification is performed based on environmental data to obtain position information of the obstacle; determining the first distance and first deviation information based on the position information of the tooth and the position information of the pallet slot; determining the second deviation information based on the position information of the cargo and the position information of the shelf space; determining the second distance based on the position information of the tooth and the position information of the obstacle.

[0007] Optionally, pallet slot detection is performed based on environmental data to obtain position information of the pallet slot, including: acquiring initial point cloud data of the pallet using a lidar sensor installed on a forklift; processing the initial point cloud data to obtain point cloud data after removing outliers; clustering the point cloud data to obtain at least one cluster; extracting geometric features from at least one cluster to obtain at least one geometric feature; obtaining at least one pixel coordinate of the pallet based on the at least one geometric feature and a preset size range of the pallet; and obtaining position information of the pallet slot based on the at least one pixel coordinate and image data of the pallet.

[0008] Optionally, the geometric features include: the width, depth and height of the pallet, and the preset size range includes: a preset width range, a preset depth range and a preset height range of the pallet; based on the geometric features and the preset size range of the pallet, obtaining at least one pixel coordinate of the pallet includes: in response to the width of the pallet satisfying the preset width range, the depth of the pallet satisfying the preset depth range, and the height of the pallet satisfying the preset height range, determining at least one cluster corresponding to the at least one geometric feature as at least one candidate cluster; obtaining at least one geometric center coordinate of the at least one candidate cluster, performing coordinate transformation on the at least one geometric center coordinate, and obtaining at least one pixel coordinate.

[0009] Optionally, based on at least one pixel coordinate and an image of the pallet, the position information of the pallet is obtained, including: obtaining the center pixel coordinates of the image data; obtaining the distance between the center pixel coordinates and at least one pixel coordinate to obtain at least one third distance; determining the distance with the smallest absolute value in the at least one third distance as the target distance; and performing coordinate conversion on the pixel coordinates corresponding to the target distance to obtain the position information.

[0010] Optionally, the position information of the pallet slot includes at least: the three-dimensional coordinates of the pallet slot and the cluster coordinates of the pallet slot, the cluster coordinates are the coordinates obtained after clustering the point cloud data of the pallet slot; based on the position information of the tooth and the position information of the pallet slot, determining the first distance includes: obtaining the width and height of the pallet slot based on the three-dimensional coordinates of the pallet slot and the cluster coordinates; judging whether the width of the pallet slot is greater than or equal to the width of the tooth, and whether the height of the pallet slot is greater than or equal to the thickness of the tooth; in response to the width of the pallet slot being greater than or equal to the width of the tooth, and the height of the pallet slot being greater than or equal to the thickness of the tooth, determining the first distance based on the three-dimensional coordinates of the tooth and the three-dimensional coordinates of the slot.

[0011] Optionally, based on the position information of the insert tooth and the position information of the pallet slot, the first deviation information is determined, including: obtaining the horizontal coordinate value in the position information of the insert tooth, and the difference between the horizontal coordinate value in the position information of the pallet slot, to obtain the first deviation; obtaining the vertical coordinate value in the position information of the insert tooth, and the difference between the vertical coordinate value in the position information of the pallet slot, to obtain the second deviation; summarizing the first deviation and the second deviation to obtain the first deviation information.

[0012] Optionally, based on the location information of the goods and the location information of the empty shelf, second deviation information is determined, including: based on the location information of the goods, obtaining the left position and the right position of the goods; based on the location information of the empty shelf, obtaining the left boundary and the right boundary of the empty shelf; based on the left position of the goods and the left boundary of the empty shelf, obtaining a third deviation between the left position of the goods and the left boundary of the empty shelf; based on the right position of the goods and the right boundary of the empty shelf, obtaining a fourth deviation between the right position of the goods and the right boundary of the shelf; based on the third deviation and the fourth deviation, obtaining second deviation information.

[0013] Optionally, the forklift is adjusted based on the first distance, the deviation information and the second distance, including: adjusting the position of the forklift based on the first distance; adjusting the position of the insert tooth horizontally and vertically based on the first deviation information; adjusting the position of the insert tooth horizontally based on the second deviation information; and adjusting the speed of the forklift based on the second distance.

[0014] Optionally, the position of the forklift is adjusted based on the first distance, including: in response to the first distance being greater than a first preset distance, controlling the forklift to move toward the slot; in response to the first distance being less than the first preset distance, controlling the inserting teeth to align with the slot; in response to the first distance being less than a second preset distance, controlling the inserting teeth to perform an insertion operation, wherein the second preset distance is less than the first preset distance.

[0015] Optionally, adjusting the position of the insert tooth in the horizontal direction and the vertical direction based on the first deviation information includes: adjusting the position of the insert tooth in the horizontal direction based on the first deviation; and adjusting the position of the insert tooth in the vertical direction based on the second deviation.

[0016] Optionally, based on the first deviation, the position of the insert tooth is adjusted in the horizontal direction, including: in response to the first deviation being less than the first preset deviation, controlling the insert tooth to move rightward; in response to the first deviation being greater than the first preset deviation, controlling the insert tooth to move leftward.

[0017] Optionally, based on the second deviation, the position of the insert tooth is adjusted in a vertical direction, including: in response to the second deviation being greater than a second preset deviation, controlling the insert tooth to move downward.

[0018] Optionally, the position of the insert tooth is adjusted horizontally based on the second deviation information, including: in response to the second deviation information being greater than a preset threshold, controlling the insert tooth to move leftward; in response to the second deviation information being less than a preset threshold, controlling the insert tooth to move rightward.

[0019] Optionally, adjusting the speed of the forklift based on the second distance includes: in response to the second distance being greater than a preset safety distance, not adjusting the speed; in response to the second distance being less than or equal to the preset safety distance, controlling the forklift to decelerate.

[0020] Optionally, the method also includes: fusing the point cloud data acquired by the lidar sensor and the image data acquired by the visual sensor to obtain processed point cloud data and processed image data; and adjusting the forklift based on the processed point cloud data and the processed image data.

[0021] Optionally, information fusion is performed on the point cloud data acquired by the lidar sensor and the image data acquired by the visual sensor to obtain processed point cloud data and processed image data, including: time alignment of the point cloud data and the image data to obtain first point cloud data and first image data, wherein the first point cloud data is time synchronized with the first image data; coordinate transformation of the first point cloud data and the first image data to obtain processed point cloud data and processed image data.

[0022] According to another aspect of an embodiment of the present invention, a forklift control device is also provided, including: an acquisition module, used to acquire environmental data in front of the forklift through multiple different types of sensors installed on the forklift, and the multiple different types of sensors include at least: a lidar sensor and a visual sensor; a processing module, used to perform pallet slot detection based on the environmental data, obtain a first distance between the forklift's tooth and the pallet slot and a first deviation information between the tooth and the pallet slot, perform cargo and shelf space detection based on the environmental data to obtain second deviation information of cargo and shelf space, and perform obstacle identification based on the environmental data to obtain a second distance between the forklift and the obstacle; an adjustment module, used to adjust the forklift based on the first distance, the deviation information, and the second distance.

[0023] According to another aspect of an embodiment of the present invention, there is further provided an electronic device, comprising: a memory storing an executable program; and a processor for running the program, wherein the method in each embodiment of the present invention is executed when the program is running.

[0024] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided. The computer-readable storage medium includes a stored executable program, wherein when the executable program is running, the device where the computer-readable storage medium is located is controlled to execute the methods in various embodiments of the present invention.

[0025] According to another aspect of an embodiment of the present invention, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the method in each embodiment of the present invention is implemented.

[0026] According to another aspect of an embodiment of the present invention, a computer program product is provided, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method in each embodiment of the present invention is implemented.

[0027] According to another aspect of the embodiments of the present invention, a computer program is further provided. When the computer program is executed by a processor, the methods in the embodiments of the present invention are implemented.

[0028] In an embodiment of the present invention, multiple different types of sensors installed on a forklift are used to obtain environmental data in front of the forklift, and the multiple different types of sensors include at least: a laser radar sensor and a visual sensor; pallet slot detection is performed based on the environmental data to obtain a first distance between the forklift's tooth and the pallet slot and a first deviation information between the tooth and the pallet slot; cargo and shelf vacancy detection is performed based on the environmental data to obtain second deviation information between the cargo and shelf vacancy; obstacle identification is performed based on the environmental data to obtain a second distance between the forklift and the obstacle; and the forklift is adjusted based on the first distance, the deviation information, and the second distance. It is easy to notice that multi-sensor data fusion is achieved through laser radar sensors and visual sensors, overcoming the limitations of traditional machine vision in complex environments such as low light, occlusion, and reflection, and achieving the technical effect of being able to accurately detect pallets. Furthermore, based on the first distance, the first deviation information, the second deviation information and the second distance obtained through the environmental data, the purpose of accurately adjusting the forklift is achieved. Furthermore, the purpose of improving the safety of the forklift during the operation of inserting and removing pallets is achieved, thereby achieving a technical effect of making the forklift safer during the operation of inserting and removing pallets, and thus solving the technical problem of low safety of the forklift during the operation of inserting and removing pallets due to the inability to accurately detect the pallet. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0030] Figure 1 is a flow chart of a forklift control method according to an embodiment of the present invention;

[0031] Figure 2 It is an optional forklift pallet slot detection and auxiliary operation system according to an embodiment of the present invention;

[0032] Figure 3 is a schematic diagram of a forklift control device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0033] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0034] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0035] Example 1

[0036] According to an embodiment of the present invention, a forklift control method embodiment is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0037] Figure 1 is a flow chart of a forklift control method according to an embodiment of the present invention. Figure 1 As shown, the method comprises the following steps:

[0038] Step S102, obtaining environmental data in front of the forklift through a plurality of different types of sensors installed on the forklift, where the plurality of different types of sensors at least include: a laser radar sensor and a visual sensor.

[0039] The above-mentioned laser radar sensor can be any sensor in the prior art that can perform distance measurement, three-dimensional modeling and environmental monitoring and is installed on the forklift in advance. The specific sensor type is not limited in this embodiment. In this embodiment, point cloud data can be obtained through the laser radar sensor. The above-mentioned visual sensor can be any sensor in the prior art that can perform image acquisition, feature extraction, target recognition and depth perception and is installed on the forklift in advance. The specific sensor type is not limited in this embodiment. In this embodiment, image data, depth data, etc. can be obtained through the visual sensor. The above-mentioned environmental data may include but is not limited to: point cloud data obtained by the laser radar sensor, image data and depth data obtained by the visual sensor. In particular, the point cloud data may include but is not limited to: position information of the forklift's tooth, position information of the pallet slot, position information of the shelf, position information of the goods and position information of the obstacle. Image data may include but is not limited to: image data of the forklift's tooth, image data of the pallet slot, image data of the shelf, image data of the goods and image data of the obstacle. Depth data may include but is not limited to: depth data of the forklift's tooth, depth data of the pallet slot, depth data of the shelf, depth data of the goods and depth data of the obstacle. The above-mentioned shelf is used for placing pallets, and the pallets are used for placing goods. The obstacle can be an obstacle between the insert and the goods.

[0040] In order to address the deficiencies in the prior art regarding forklift pallet detection technology, the present embodiment provides a forklift pallet slot detection and auxiliary operation system based on the fusion of lidar and machine vision, which aims to improve the accuracy of pallet detection through multi-sensor information fusion technology, increase applicable scenarios and detection objects, ensure the accuracy of fork teeth inserting pallets, and improve safety and efficiency during operation. Figure 2 is an optional forklift pallet slot detection and auxiliary operation system according to an embodiment of the present invention, such as Figure 2 As shown, the system includes: a sensing unit 21, a data processing unit 22, an output unit 23 and a vehicle-mounted display terminal 24. Figure 2It can be known that the perception unit 21 is connected in communication with the data processing unit 22, the data processing unit 22 is connected in communication with the output unit 23, and the output unit 23 is connected in communication with the vehicle-mounted display terminal 24. Among them, the perception unit 21 includes: a camera 21-1 and a laser radar 21-2, and the camera 21-1 and the laser radar 21-2 are connected in communication; the data processing unit 22 includes: a filter module 22-1, a coordinate conversion module 22-2, an integrated control module 22-3 and a safe driving module 22-4, the filter module 22-1 and the coordinate conversion module 22-2 are connected in communication, the coordinate conversion module 22-2 and the integrated control module 22-3 are connected in communication, and the integrated control module 22-3 and the safe driving module 22-4 are connected in communication; the output unit 23 includes: a display module 23-1 and a communication module 23-2, and the display module 23-1 and the communication module 23-2 are connected in communication.

[0041] The sensing unit may include but is not limited to: a camera and a laser radar installed inside the forklift's tine. The sensing unit is used to obtain real-time environmental information about the forklift, especially the relative position data between the tine and the pallet slot, as well as point cloud information. Through real-time data collection by multiple sensors such as laser radar and cameras, the sensing unit can accurately measure the distance and position between the slot and the tine. This information is transmitted to the integrated control module for processing, so that the system can more accurately determine the slot position of the pallet, thereby ensuring the alignment of the forklift's tine with the pallet. This not only improves the accuracy of pallet detection, but also effectively reduces accidents caused by operational errors and ensures safety during the operation process.

[0042] Among them, the data processing unit includes a filtering module, a coordinate conversion module and an integrated control module. The filtering module is used to remove noise and irrelevant points from the laser radar point cloud data to ensure the reliability of subsequent data processing. The coordinate conversion module is used to convert the camera and laser radar data into a unified coordinate system to achieve information fusion. The integrated control module is responsible for the comprehensive analysis of the data collected and converted by multiple sensors to achieve accurate detection of the target pallet slot, judgment of the deviation between the shelf space and the goods, and identification of obstacles. First, the module accurately detects the position of the pallet slot to ensure that the fork teeth can accurately align with the pallet slot. Next, by calculating the left and right deviations between the goods and the shelf space, ensure that the goods can be safely stored in a suitable space. Then, according to the analysis results, the system uses the servo motor to make automatic adjustments, such as automatically adjusting the position of the fork teeth according to the deviation, to ensure the accuracy and safety of the operation. At the same time, the integrated control module is also responsible for real-time monitoring of obstacles in front of the forklift, identifying potential risks, and ensuring safe operation. Finally, the integrated control module outputs the processing results to the on-board display screen, provides the driver with operation suggestions or directly performs corresponding automatic adjustment operations, thereby improving the accuracy and efficiency of the operation.

[0043] Among them, the output unit is connected to the data processing unit, including a display module and a communication module. The display module presents the processed information on the forklift operation interface to assist the driver in making decisions. The communication module is used to transmit video data of slot detection and obstacle detection. The communication module is also used to transmit video data of slot detection and obstacle detection between the decision module, the safety module and the on-board display screen to ensure that the driver can obtain information about the surrounding environment in real time and make correct driving decisions. This module ensures that high-quality video and data streams can be transmitted in a timely manner even in complex environments.

[0044] Among them, the on-board display terminal is used to display the processed information to assist the driver in controlling the forklift.

[0045] In an optional embodiment, in order to solve the problem that the forklift is not able to accurately detect the pallet in the prior art, resulting in low safety during the operation of inserting and removing the pallet, the embodiment of the present invention provides a forklift control method, which is applied to the forklift pallet slot detection and auxiliary operation system mentioned above. At this time, the environmental data in front of the forklift can be obtained through the perception unit, that is, multiple types of sensors installed on the forklift. For example, the point cloud data in front of the forklift can be obtained through the laser radar sensor, and the image data and depth data in front of the forklift can be obtained through the visual sensor.

[0046] Step S104, performing pallet slot detection based on environmental data to obtain a first distance between the forklift's tooth and the pallet slot and first deviation information between the tooth and the pallet slot, performing cargo and shelf space detection based on environmental data to obtain second deviation information between the cargo and shelf space, and performing obstacle recognition based on environmental data to obtain a second distance between the forklift and the obstacle.

[0047] The first distance is used to determine whether it is feasible for the forklift's inserting teeth to insert into the pallet slot. The first deviation information is used to determine whether the forklift's inserting teeth can accurately insert into the pallet slot, and may include but is not limited to: horizontal deviation information and vertical deviation information. The second deviation information is used to determine whether the position of the forklift's inserting teeth needs to be adjusted, and may include but is not limited to: horizontal deviation information and vertical deviation information. The second distance information is used to ensure the safety of the forklift's operation.

[0048] In an optional embodiment, when environmental data in front of the forklift is obtained, the data processing unit can perform pallet slot detection based on the environmental data to obtain a first distance between the forklift's tooth and the pallet slot and a first deviation information between the tooth and the pallet slot, perform cargo and shelf space detection based on the environmental data to obtain second deviation information between the cargo and shelf space, and perform obstacle identification based on the environmental data to obtain a second distance between the forklift and the obstacle.

[0049] For example, the data processing unit can detect the pallet slot based on the point cloud cluster coordinates and three-dimensional position of the pallet slot in the environmental data, the three-dimensional position of the forklift's tooth, and the width and thickness data of the tooth, to obtain the first distance between the forklift's tooth and the pallet slot. At the same time, the data processing unit can detect the pallet slot based on the point cloud cluster coordinates and three-dimensional position of the pallet slot in the environmental data, the three-dimensional position of the forklift's tooth, and the width and thickness data of the tooth, and the pre-set horizontal threshold and vertical threshold to obtain the first deviation information between the tooth and the pallet slot. At the same time, the data processing unit can detect the cargo and shelf vacancy based on the shelf image data, three-dimensional point cloud data, and cargo image data in the environmental data to obtain the second deviation information between the cargo and the shelf vacancy. At the same time, the data processing unit can also identify the obstacle based on the image data of the obstacle and the pre-set distance threshold between the tooth and the obstacle to obtain the second distance between the forklift and the obstacle.

[0050] Step S106: adjusting the forklift based on the first distance, the deviation information and the second distance.

[0051] In an optional embodiment, when the first distance, the deviation information, and the second distance are obtained, the forklift can be adjusted based on the first distance, the deviation information, and the second distance. For example, when it is determined that the first distance is greater than the first distance threshold value set in advance, it can be determined that the forklift's inserting teeth cannot insert the pallet slot, and the forklift can be adjusted based on the first distance. For another example, when it is determined that the first distance is less than or equal to the set distance threshold, but the horizontal deviation in the deviation information is greater than the deviation value set in advance, it can be determined that the forklift's inserting teeth cannot accurately insert the pallet slot, and the forklift can be adjusted based on the deviation information. For another example, when it is determined that the first distance is less than or equal to the first distance threshold value set in advance, and the deviation information is less than or equal to the deviation value set in advance, and the second distance is less than or equal to the second distance threshold value set in advance, it can be determined that there is no need to adjust the forklift.

[0052] Through the above steps, it is possible to obtain environmental data in front of the forklift through multiple different types of sensors installed on the forklift, and the multiple different types of sensors include at least: a laser radar sensor and a visual sensor; perform pallet slot detection based on environmental data to obtain a first distance between the forklift's tooth and the pallet slot and a first deviation information between the tooth and the pallet slot, perform cargo and shelf space detection based on environmental data to obtain a second deviation information of cargo and shelf space, and perform obstacle identification based on environmental data to obtain a second distance between the forklift and the obstacle; and adjust the forklift based on the first distance, deviation information, and second distance. It is easy to notice that multi-sensor data fusion is achieved through laser radar sensors and visual sensors, overcoming the limitations of traditional machine vision in complex environments such as low light, occlusion, and reflection, and achieving the technical effect of being able to accurately detect pallets. Furthermore, based on the first distance, the first deviation information, the second deviation information and the second distance obtained through the environmental data, the purpose of accurately adjusting the forklift is achieved. Furthermore, the purpose of improving the safety of the forklift during the operation of inserting and removing pallets is achieved, thereby achieving a technical effect of making the forklift safer during the operation of inserting and removing pallets, and thus solving the technical problem of low safety of the forklift during the operation of inserting and removing pallets due to the inability to accurately detect the pallet.

[0053] Optionally, pallet slot detection is performed based on environmental data to obtain a first distance between the forklift's tooth and the pallet slot and first deviation information between the tooth and the pallet slot, cargo and shelf space detection is performed based on environmental data to obtain second deviation information between the cargo and shelf space, and obstacle identification is performed based on environmental data to obtain a second distance between the forklift and the obstacle, including: determining the position information of the forklift's tooth based on the forklift's position information; pallet slot detection is performed based on environmental data to obtain position information of the pallet slot, cargo and shelf space detection is performed based on environmental data to obtain position information of the cargo and position information of the shelf space, and obstacle identification is performed based on environmental data to obtain position information of the obstacle; determining the first distance and first deviation information based on the position information of the tooth and the position information of the pallet slot; determining the second deviation information based on the position information of the cargo and the position information of the shelf space; determining the second distance based on the position information of the tooth and the position information of the obstacle.

[0054] In an optional embodiment, first, the position information of the forklift can be obtained through a laser radar sensor, for example, the three-dimensional coordinates of the forklift, and then a three-dimensional model can be established based on the three-dimensional coordinates of the forklift. Then, the position information of the tooth can be calculated based on the position information of the forklift and the three-dimensional model.

[0055] Optionally, first, initial point cloud data of the pallet can be acquired by a laser radar sensor installed on a forklift, and then the initial point cloud data can be processed to obtain point cloud data after removing outliers, and finally the point cloud data can be clustered to obtain at least one cluster, and then geometric features can be extracted from at least one cluster to obtain at least one geometric feature, and then based on the at least one geometric feature and a preset size range of the pallet, at least one pixel coordinate of the pallet can be obtained, and finally based on the at least one pixel coordinate and the image data of the pallet, the position information of the pallet slot can be obtained.

[0056] Optionally, point cloud data, image data and depth data of goods, shelves and obstacles can be obtained through lidar sensors and visual sensors. Secondly, the point cloud data, image data and depth data of goods and shelves can be used for goods and shelf vacancy detection through a pre-trained model to obtain the location information of the goods and the location information of the shelf vacancy. The point cloud data, image data and depth data of obstacles can also be used to identify obstacles through a pre-trained model to obtain the location information of the obstacles.

[0057] Optionally, when the position information of the tooth, the position information of the pallet slot, the position information of the cargo, the position information of the empty shelf space and the position information of the obstacle are obtained, the first distance and the first deviation information can be determined based on the position information of the tooth and the position information of the pallet slot; and the second deviation information can be determined based on the position information of the cargo and the position information of the empty shelf space.

[0058] Optionally, the second distance can be obtained by:

[0059] Step S71: Use the camera installed in front of the forklift to collect video data in real time. The video frame is I camera (t), where t is the timestamp. Video data is transmitted at a certain frame rate f ps Transmit and process.

[0060] Step S72: A pre-trained real-time target detection model (You Only Look Once version 5, YOLOv5) is loaded into the system for real-time detection of obstacles in the front video frame. The input of the YOLOv5 model is a single frame image I captured by the camera. camera (t), YOLOv5 maps image features to detection frames of multiple scales through a convolutional neural network and outputs a set of candidate frames B targ and its category label confidence C targ The output form of each candidate box is B targ ={(U min ,V min ),(U max ,Vmax )}, indicating the detection area of ​​the obstacle.

[0061] Step S73: First, set a confidence threshold C thresh =0.7, if the confidence C of a detection box targ ≥C thresh , then the detection box is considered valid, otherwise the detection box is ignored. For the detection box that passes the confidence screening, the system uses the category label C targ Further determine whether the object in the detection frame is a potential dangerous obstacle (such as "pedestrian", "vehicle" or other objects that may hinder the forklift's travel). Only when the category label meets the dangerous obstacle judgment criteria will further processing be performed. For a valid obstacle detection frame, output its upper left corner and lower right corner coordinates B targ , and is used to calculate the specific location of obstacles.

[0062] Step S74: The detection frame coordinates B output by YOLOv5 targ The coordinate conversion module is used to convert the laser radar into a three-dimensional coordinate system so that obstacles can be found in the point cloud and the position information of the obstacles can be further calculated. obstacle ={(X obs ,Y obs ,Z obs )}, which provides the three-dimensional coordinates and distance information of obstacles.

[0063] Step S75: Setting the safety threshold D safe = 200cm, calculate the second distance D from the obstacle to the gear obs :

[0064] D obs =√(X f -X obs ) 2 +(Y f -Y obs ) 2 +(Z f -Z obs ) 2 .

[0065] Optionally, pallet slot detection is performed based on environmental data to obtain position information of the pallet slot, including: acquiring initial point cloud data of the pallet using a lidar sensor installed on a forklift; processing the initial point cloud data to obtain point cloud data after removing outliers; clustering the point cloud data to obtain at least one cluster; extracting geometric features from at least one cluster to obtain at least one geometric feature; obtaining at least one pixel coordinate of the pallet based on the at least one geometric feature and a preset size range of the pallet; and obtaining position information of the pallet slot based on the at least one pixel coordinate and image data of the pallet.

[0066] In an optional embodiment, the integrated control module can extract possible tray slot areas based on the originally collected laser radar point cloud data (i.e., the initial point cloud data mentioned above), and use the depth information provided by the laser radar and the geometric features of the tray (height, width, etc.) to perform preliminary position recognition on the tray slot.

[0067] Furthermore, the position information of the tray slot can be obtained through the following steps.

[0068] Step S31, using the density-based spatial clustering algorithm (Density-Based Spatial Clustering of Applications with Noise, DBSCAN) to cluster the point cloud data acquired by the lidar, extract possible slot areas, and use the point cloud data set: P filtered ={p1,p2,…,p n}, each point p i =(X i , Y i , Z i ), where all coordinates are in the lidar coordinate system.

[0069] Optionally, the point cloud dataset P can be obtained by the following steps: filtered :

[0070] Among them, the filtering module adopts the method of statistical outlier removal, and the specific steps are as follows:

[0071] Step S11: Obtain the original point cloud data from the laser radar. The format is (x, y, z) coordinates, that is, a three-dimensional coordinate system, in which each point is represented by three coordinate values, corresponding to three mutually perpendicular axes: the x-axis (horizontal direction), the y-axis (vertical direction) and the z-axis (depth direction).

[0072] Step S12: Set the filter parameter K: Set the number of points contained in the neighborhood, which is used to calculate the average distance between points. Std: Set the multiplier threshold of the standard deviation, which is used to determine which points are outliers.

[0073] Step S13: For each point p i (X i ,Y i ,Z i ), we calculate it and all other p in the point cloud j (X j ,Y j ,Z j ) between the Euclidean distance d(p i ,p j ), from these calculated distances, find the nearest K points, called point p i The average distance d of the points in these neighborhoods is then calculated. avg And standard deviation σ:

[0074]

[0075] Step S14: If point p satisfies the following conditions: d(p i ,p j )>d avg +Std×σ, then the point is marked as an outlier.

[0076] Step S15: For all points p that meet the requirements of step S14, remove them from the original point cloud and generate a new point cloud dataset P. filtered , all points marked as outliers are removed, and valid points are retained for subsequent slot detection and other analysis processing.

[0077] Step S32, set the distance threshold ∈ of the cluster slot (∈ value is usually related to the size of the slot) and the distance threshold ε of the obstacle. Select an unprocessed point as the core point, and calculate the distance between the core point and all other points: According to the distance calculation results, all points within the threshold ∈ of the core point are found to form the neighborhood of the slot core point, and potential obstacle points within the range of (∈, ε) are removed.

[0078] Step S33, set the minimum point threshold min Pts =4, if the number of points in the neighborhood exceeds the set minimum number of points min Pts , then the core point and all the points in its neighborhood are marked as the same cluster. Next, repeat the above steps for each point in the neighborhood, check the neighborhood of these points, and continue to add points that meet the conditions until there are no new points that can be added to the current cluster.

[0079] Step S34, once the current clustering is completed, select the next unprocessed point and repeat the above steps until all points are processed. Through this process, DBSCAN can not only identify multiple point cloud clusters, each cluster represents a possible slot area, but also effectively handle noise points and deal with non-uniformly distributed data. This process ensures that the slot can be accurately identified and located under specific environmental conditions, thereby improving the operating efficiency and safety of the forklift.

[0080] Step S35, further extract geometric features from the point cloud clusters generated by clustering, and calculate the width of each cluster in the laser radar point cloud. The width feature can be obtained by finding the minimum and maximum values ​​of the point cloud cluster in the X-axis direction and calculating their difference: W slot =max(X i )-min(X i ). Similarly, the height difference of each cluster on the Z axis is calculated, and the height feature is obtained by solving the difference between the highest point and the lowest point in the point cloud cluster: H slot =max(Z i )-min(Z i ). By calculating the depth range of the point cloud cluster in the Y-axis direction, the depth characteristics of the slot are determined: D slot =max(Y i )-min(Y i ).

[0081] Optionally, when at least one collective feature is obtained, at least one pixel coordinate of the pallet can be obtained based on at least one geometric feature and a preset size range of the pallet, and then the position information of the pallet slot can be obtained based on the at least one pixel coordinate and the image data of the pallet.

[0082] Optionally, the geometric features include: the width, depth and height of the pallet, and the preset size range includes: a preset width range, a preset depth range and a preset height range of the pallet; based on the geometric features and the preset size range of the pallet, obtaining at least one pixel coordinate of the pallet includes: in response to the width of the pallet satisfying the preset width range, the depth of the pallet satisfying the preset depth range, and the height of the pallet satisfying the preset height range, determining at least one cluster corresponding to the at least one geometric feature as at least one candidate cluster; obtaining at least one geometric center coordinate of the at least one candidate cluster, performing coordinate transformation on the at least one geometric center coordinate, and obtaining at least one pixel coordinate.

[0083] In an optional embodiment, at least one pixel coordinate may be obtained by the following steps:

[0084] Step S36, compare the geometric feature values ​​calculated above with the preset tray slot size range (standard size ± 5cm). For each cluster, check whether its width, depth and height meet the known geometric parameter range of the tray slot. Assume that the predefined size range of the tray slot (i.e. the preset size range of the tray mentioned above) is:

[0085] Width range: W min ,W max ;

[0086] Height range: H min ,H max ;

[0087] Depth range: D min ,D max ;

[0088] The following conditions are applied to each cluster:

[0089] W min ≤W slot ≤W max ;

[0090] H min ≤H slot ≤H max ;

[0091] D min ≤D slot ≤D max ;

[0092] If the geometric features of a cluster meet all the conditions, it is marked as a potential tray slot. This cluster is represented as: k ={(X k1 ,Y k1 ,Z k1 ),(X k2 ,Y k2 ,Z k2 ),…,(X kn ,Y kn ,Z kn )}. Where k is the index of the cluster and n is the number of points in the cluster.

[0093] Step S37, select the cluster at the lower position, and first define a threshold range of the Z value:

[0094] Z min ≤Z i ≤Z min +1.5×H pallet ;

[0095] In the formula, Z minis the minimum Z coordinate among all potential tray slot clusters, H pallet is the height of the tray.

[0096] For clusters filtered by geometric features, calculate their average Z value: The clusters satisfying the above conditions are taken as the final potential tray slots (ie, the candidate clusters mentioned above).

[0097] Step S38: After obtaining the matching point cloud clusters, the geometric center coordinates of these clusters (i.e., at least one geometric center coordinate mentioned above) are calculated. The geometric center calculation formula of the cluster is as follows: And use the coordinate conversion module to convert it into pixel coordinates (u center ,v center ) (i.e. at least one pixel coordinate mentioned above).

[0098] Optionally, based on at least one pixel coordinate and an image of the pallet, the position information of the pallet is obtained, including: obtaining the center pixel coordinates of the image data; obtaining the distance between the center pixel coordinates and at least one pixel coordinate to obtain at least one third distance; determining the distance with the smallest absolute value in the at least one third distance as the target distance; and performing coordinate conversion on the pixel coordinates corresponding to the target distance to obtain the position information.

[0099] In an optional embodiment, after obtaining the preliminary slot position of the pallet slot, the system further refines the positioning of the pallet slot by fusing image information and laser radar data. At this time, the system can use image recognition algorithms to accurately detect the pallet slot area and align and match it with the three-dimensional point cloud data of the laser radar. By fusing the position information of the two, the center point, angle and size of the pallet slot are determined, thereby ensuring that the forklift can accurately align with the slot during operation, improving the accuracy and stability of the insertion and extraction operation.

[0100] Furthermore, after the point cloud clustering and geometric feature extraction preliminarily determine the slot area, machine vision is used to further analyze the image information of the area:

[0101] Step S41: The input image is I(U, V), where (U, V) is the pixel coordinates converted by the coordinate conversion module. The input image is passed to the YOLOv5 model for recognition, and the output detection result is obtained. The model will return the category of each detection box (such as tray, slot) and the pixel coordinates of the upper left corner and the lower right corner B image ={(U min ,V min ),(U max ,V max )} and its corresponding confidence C slot .

[0102] Step S42: Setting the confidence threshold Cthreshold = 0.7, only the detection boxes with confidence exceeding this threshold are selected as candidate slot areas. The selected candidate area is B candidate .

[0103] Step S43: Calculate the slot detection box B in the image candidate The center pixel coordinates (U center ,V center ):

[0104]

[0105] Step S44: Match the point cloud cluster and the image detection frame, and compare the slot detection frame B in the image candidate The pixel coordinates in (U center ,V center ) and the cluster center coordinates (u center ,v center ), find the cluster that best matches the point cloud clustering result:

[0106] argmin cluster ∥(U center ,V center )-(u center ,v center )∥;

[0107] The significance of this formula is to find the minimum distance (i.e., target distance) between the center point of the slot detection box in the image and the cluster center of the lidar point cloud data, thereby determining the most likely slot location.

[0108] Step S45: Based on the calculated distance, the center point of the detection frame with the smallest distance between the two is selected as the final slot position, and the pixel coordinate (U s ,V s ) mark, and then convert it into three-dimensional coordinates (X s ,Y s ,Z s ), so as to calculate the distance between the tooth and the slot.

[0109] This process not only takes into account the image information, but also combines the geometric features of the point cloud data to ensure that the confirmed slot position is accurate and reliable. After confirming the final slot position, the system will output the coordinates of the position to the on-board display screen to facilitate the driver to perform precise insertion and extraction operations.

[0110] Optionally, the position information of the pallet slot includes at least: the three-dimensional coordinates of the pallet slot and the cluster coordinates of the pallet slot, the cluster coordinates are the coordinates obtained after clustering the point cloud data of the pallet slot; based on the position information of the tooth and the position information of the pallet slot, determining the first distance includes: obtaining the width and height of the pallet slot based on the three-dimensional coordinates of the pallet slot and the cluster coordinates; judging whether the width of the pallet slot is greater than or equal to the width of the tooth, and whether the height of the pallet slot is greater than or equal to the thickness of the tooth; in response to the width of the pallet slot being greater than or equal to the width of the tooth, and the height of the pallet slot being greater than or equal to the thickness of the tooth, determining the first distance based on the three-dimensional coordinates of the tooth and the three-dimensional coordinates of the slot.

[0111] The clustering coordinates mentioned above may be the clustering coordinates of the tray slots obtained through the above step S35.

[0112] In an optional embodiment, the insertion feasibility detection and self-adjustment system analyzes the current environmental conditions, evaluates whether the conditions for successfully inserting the pallet are met, and automatically adjusts the insertion teeth to the appropriate position. This step combines the real-time position of the forklift and the relative position of the pallet slot and the insertion teeth to ensure that the forklift can smoothly enter the slot. If the system detects an offset or insufficient space, it will automatically adjust and feedback the corresponding warning information to the driver.

[0113] Furthermore, the first distance can be determined through the following steps.

[0114] Step S51, first define the system variables:

[0115] Gear position, expressed as coordinates (X f ,Y f ,Z f ) represents the three-dimensional position of the forklift tine.

[0116] The tray slot position is obtained by using the coordinates (X s ,Y s ,Z s ) indicates the three-dimensional position of the tray slot.

[0117] Slot size: The slot width W is obtained by the point cloud cluster coordinates determined in step S35. s =X max -X min and height D s =Z max -Z min .

[0118] Gear size, set the width W of the gear f and thickness D f , this data is entered manually.

[0119] Step S52: Perform feasibility judgment and match the slot size with the gear size:

[0120] Determine the slot width W s Is it greater than or equal to the gear width W? f :W s ≥W f ;

[0121] Determine the slot height D f Is it greater than or equal to the gear thickness D? s :D s ≥D f ;

[0122] When it is determined that the width of the tray slot is greater than or equal to the width of the inserting gear, and the height of the tray slot is greater than or equal to the thickness of the inserting gear, the first distance is determined based on the three-dimensional coordinates of the inserting gear and the three-dimensional coordinates of the slot. At the same time, if all the above conditions are met, the inserting gear can be safely inserted into the tray slot, and the feedback information of "insertion is feasible" is output. If any condition is not met, the warning message of "slot does not match" is output.

[0123] Optionally, the first distance can be obtained by the following formula:

[0124]

[0125] Among them, D fs is the first distance between the insert tooth and the tray slot.

[0126] It should be noted that, based on the calculation result of the first distance, the system will provide specific digital prompts on the display screen to help the driver adjust the forklift position.

[0127] Optionally, based on the position information of the insert tooth and the position information of the pallet slot, the first deviation information is determined, including: obtaining the horizontal coordinate value in the position information of the insert tooth, and the difference between the horizontal coordinate value in the position information of the pallet slot, to obtain the first deviation; obtaining the vertical coordinate value in the position information of the insert tooth, and the difference between the vertical coordinate value in the position information of the pallet slot, to obtain the second deviation; summarizing the first deviation and the second deviation to obtain the first deviation information.

[0128] In an optional embodiment, the first deviation information may be obtained by the following steps:

[0129] Step S54: Calculate and automatically adjust the directional deviation, and set the deviation limit D for width and height. Hth and D Vth Can be set to about 5% of slot width and height to ensure sufficient insertion space, automatic adjustment using PID control algorithm

[0130] Calculate the horizontal deviation: e x =X s -X f , D horizontal =|e x |;

[0131] Among them, X s The horizontal coordinate value in the tray slot position information, X f The horizontal coordinate value of the position information of the gear, D horizontal is the first deviation.

[0132] Optionally, calculate the vertical deviation: e z =Z s -Z f , D vertical =|e z |;

[0133] Among them, Z s The vertical coordinate value in the position information of the tray slot, Z f The vertical coordinate value of the gear position information, D vertical is the second deviation.

[0134] Optionally, when the first deviation and the second deviation are obtained, the first deviation and the second deviation may be aggregated to obtain the first deviation information.

[0135] Optionally, based on the location information of the goods and the location information of the empty shelf, second deviation information is determined, including: based on the location information of the goods, obtaining the left position and the right position of the goods; based on the location information of the empty shelf, obtaining the left boundary and the right boundary of the empty shelf; based on the left position of the goods and the left boundary of the empty shelf, obtaining a third deviation between the left position of the goods and the left boundary of the empty shelf; based on the right position of the goods and the right boundary of the empty shelf, obtaining a fourth deviation between the right position of the goods and the right boundary of the shelf; based on the third deviation and the fourth deviation, obtaining second deviation information.

[0136] In an optional embodiment, the second deviation information may be obtained by the decision module based on the following steps:

[0137] Step S61: Use the YOLOv5 model to detect the bounding box of the goods from the image captured by the camera. The output of YOLOv5 is the 2D bounding box coordinates of the goods in the image: B = (x cmin ,y cmin ,x cmax ,y cmax ), where (x cmin ,y cmin ) is the coordinate of the upper left corner of the bounding box, (x cmax ,ycmax ) is the coordinate of the lower right corner of the bounding box. The left position of the cargo: x carg_l =x cmin , Right side position of the goods: x carg_r =x cmax .

[0138] Step S62: Use laser radar to scan the empty shelf and generate three-dimensional point cloud data P of the shelf space ={(x i ,y i ,z i )}. Assume that the shelf height is divided into n slices on the Y axis, and the height range of each slice is Where k = 1, 2,..., n represents different slicing layers. The specific slicing formula is as follows:

[0139]

[0140] Thus, each slice Only point clouds within this height range are included. For each slice Perform left and right boundary detection respectively, extract the minimum and maximum values ​​in the X-axis direction, and represent the left and right boundaries of the slice layer: Perform a weighted average of the left and right borders of each slice, where the weights can be adjusted based on the point cloud density or height of the slice:

[0141]

[0142] where w k is the weight of the kth slice, and the weight is assigned according to the slice height.

[0143] Step S63: Calculate the deviation Δ between the left side of the goods and the left boundary of the empty shelf space left (i.e. the third deviation), and calculate the deviation Δ between the right side of the goods and the right boundary of the shelf space right (The fourth deviation):

[0144] Δ left =x carg_l -x space_l ;

[0145] Δ right =x space_r -x carg_r ;

[0146] Here, Δ left and Δ right When both are positive, it means there is enough space for the goods on both the left and right sides; if one is negative, it means that the goods on that side exceed the empty space range and need to be adjusted.

[0147] Step S64: During the automatic adjustment process, the system performs the following adjustment each time according to the deviation calculation to obtain the second deviation information Δ adjust :

[0148]

[0149] Among them, 2 is a constant.

[0150] Optionally, the forklift is adjusted based on the first distance, the deviation information and the second distance, including: adjusting the position of the forklift based on the first distance; adjusting the position of the insert tooth horizontally and vertically based on the first deviation information; adjusting the position of the insert tooth horizontally based on the second deviation information; and adjusting the speed of the forklift based on the second distance.

[0151] In an optional embodiment, when the first distance, the deviation information and the second distance are obtained, the position of the forklift can be adjusted based on the first distance, and the position of the spline can be adjusted horizontally and vertically based on the first deviation information. The position of the spline can be adjusted horizontally based on the second deviation information, and the speed of the forklift can be adjusted based on the second distance.

[0152] Optionally, the position of the forklift is adjusted based on the first distance, including: in response to the first distance being greater than a first preset distance, controlling the forklift to move toward the slot; in response to the first distance being less than the first preset distance, controlling the inserting teeth to align with the slot; in response to the first distance being less than a second preset distance, controlling the inserting teeth to perform an insertion operation, wherein the second preset distance is less than the first preset distance.

[0153] The first preset distance mentioned above can be set in advance by the user to determine whether the position of the forklift needs to be adjusted. The second preset distance mentioned above can be set in advance by the user to determine whether the position of the forklift needs to be adjusted. The specific values ​​of the first preset distance and the second preset distance are not limited in this embodiment, and the second preset distance is smaller than the first preset distance.

[0154] In an optional embodiment, the system can provide specific digital prompts on the display screen based on the calculation results to help the driver adjust the forklift position:

[0155] If D fs >50cm (i.e. the first preset distance), the system will prompt: "The insert gear is far away from the slot, please continue to move."

[0156] If D fs If the gear is less than 50cm, the system will prompt: The gear is close to the slot, please pay attention to the alignment operation.

[0157] When Dfs <5cm (i.e. the second preset distance), the display screen prompts "The distance is close, the insertion and removal operation will be precisely adjusted."

[0158] Optionally, adjusting the position of the insert tooth in the horizontal direction and the vertical direction based on the first deviation information includes: adjusting the position of the insert tooth in the horizontal direction based on the first deviation; and adjusting the position of the insert tooth in the vertical direction based on the second deviation.

[0159] In an optional embodiment, when the first deviation information is obtained, the first deviation in the first deviation information can be compared with the first preset deviation, and the position of the insert tooth can be adjusted horizontally based on the comparison result. At the same time, the second deviation in the first deviation information can be compared with the second preset deviation, and the position of the insert tooth can be adjusted vertically based on the comparison result.

[0160] Optionally, based on the first deviation, the position of the insert tooth is adjusted in the horizontal direction, including: in response to the first deviation being less than the first preset deviation, controlling the insert tooth to move rightward; in response to the first deviation being greater than the first preset deviation, controlling the insert tooth to move leftward.

[0161] In D horizontal >D Hth In the case of

[0162] If X f <X s The system output prompts "The tooth is too far to the left relative to the slot and will move right D horizontal cm. "

[0163] If X f >X s The system output prompts "The tooth is too far to the right relative to the slot and will move left by D horizontal cm. "

[0164] The control signal of the servo motor is:

[0165] In the formula, K p , K i , K d are proportional, integral, and differential gains respectively, u x (t) is the control signal, which determines the displacement adjusted by the servo motor.

[0166] After receiving the PID control signal, the servo motor moves the gear step by step according to the step control mode to ensure that the gear is accurately aligned with the tray slot. Assuming that the distance of each step is Δx, the movement amount of each step is: X f (t+1)=X f (t)+Δx, repeat this process until the horizontal deviation is Dhorizontal <D Hth , the system prompts: "Horizontal adjustment completed, please wait for vertical adjustment."

[0167] Optionally, based on the second deviation, the position of the insert tooth is adjusted in a vertical direction, including: in response to the second deviation being greater than a second preset deviation, controlling the insert tooth to move downward.

[0168] In an optional embodiment, in D vertical >D Vth In case of:

[0169] If Z f >Z s : Output prompt "The tooth is too high relative to the slot and will move down D vertical cm. "

[0170] The control signal of the servo motor is:

[0171] Assume that the distance of each step is Δz, then the movement amount of each step is Z f (t+1)=Z f (t)+Δz, repeat this process until the vertical deviation is D vertical <D Vth , the system prompts: "The height is adjusted in place, execute the insertion operation."

[0172] Step S55: If all conditions are met, a feedback message of "insertion is feasible" is output on the display screen. If any condition is not met, a corresponding warning message is output, and the above-mentioned information on the auxiliary operation is automatically adjusted and given according to the positional relationship between the inserting teeth and the slot.

[0173] Optionally, the position of the insert tooth is adjusted horizontally based on the second deviation information, including: in response to the second deviation information being greater than a preset threshold, controlling the insert tooth to move leftward; in response to the second deviation information being less than a preset threshold, controlling the insert tooth to move rightward.

[0174] When Δ adjust >0, the right deviation is greater than the left deviation, and the system will move the fork to the left |Δ adjust |cm to reduce the deviation on the right side.

[0175] When Δ adjust <0, the left deviation is greater than the right deviation, and the system will move the fork to the right |Δ adjust |cm to reduce the deviation on the left side.

[0176] Through calculation, the system will adjust the balance to the left and right to avoid the left and right sides from exceeding the deviation range alternately. This adjustment will guide the fork to make corrections to the side with larger deviation, and the correction amount is half of the difference between the deviations on both sides. In this way, the side with larger deviation can be corrected while ensuring balance without affecting the other side.

[0177] For example, if Δ left =8cm,Δ right =4cm, then the adjustment amount is:

[0178]

[0179] The system will move 2cm to the left, so that the left and right sides gradually become consistent.

[0180] If after multiple adjustments, one side's deviation value is always negative, the system will prompt the driver to cancel the storage operation and display "Position deviation is too large and cannot be stored."

[0181] Optionally, adjusting the speed of the forklift based on the second distance includes: in response to the second distance being greater than a preset safety distance, not adjusting the speed; in response to the second distance being less than or equal to the preset safety distance, controlling the forklift to decelerate.

[0182] In an optional embodiment, the system will display the obstacle detection frame in real time on the vehicle display screen, and attach the distance from the current obstacle to the tooth to each detection frame, and mark the current safety status:

[0183] If D obs >D safe (i.e. the preset safety distance), the system displays "Safe Ahead".

[0184] If D obs ≤D safe , the system will display "Obstacle ahead is approaching, distance D obs cm, please slow down".

[0185] Optionally, the method also includes: fusing the point cloud data acquired by the lidar sensor and the image data acquired by the visual sensor to obtain processed point cloud data and processed image data; and adjusting the forklift based on the processed point cloud data and the processed image data.

[0186] In an optional embodiment, the coordinate conversion module is to fuse the detection results of the laser radar and the camera after they respectively obtain the detection information of the tray slot in order to accurately obtain the complete information of the tray slot. The premise of information fusion includes two aspects: spatial alignment and temporal alignment. The point cloud scanned by the laser radar is located in a three-dimensional coordinate system centered on the laser radar. Due to different installation locations, the laser radar coordinate system needs to be aligned. l X l Y l Z l and the camera coordinate system O c X c Y c Z c The pixel coordinate system is used to describe the position of pixels in an image or texture. The position of each pixel is represented by a triple (U, V, W), where U and V represent the horizontal and vertical positions of the pixel in the image or texture, respectively, and W represents the position of the pixel in the depth direction.

[0187] Optionally, when information fusion is performed on the point cloud data acquired by the lidar sensor and the image data acquired by the visual sensor to obtain processed point cloud data and processed image data, the forklift can be adjusted based on the processed point cloud data and the processed image data.

[0188] Optionally, information fusion is performed on the point cloud data acquired by the lidar sensor and the image data acquired by the visual sensor to obtain processed point cloud data and processed image data, including: time alignment of the point cloud data and the image data to obtain first point cloud data and first image data, wherein the first point cloud data is time synchronized with the first image data; coordinate transformation of the first point cloud data and the first image data to obtain processed point cloud data and processed image data.

[0189] In an optional embodiment, due to the different operating frequencies of the camera and the laser radar, the acquisition frequency of the camera is usually much higher than that of the laser radar. In order to ensure the time synchronization of the two sensor data, the system will attach timestamp information when receiving the point cloud data and the image. This embodiment adopts a timestamp-based synchronization strategy, with the data acquisition time of the laser radar as the reference. Specifically, when the laser radar acquires new point cloud data, the system will search for a frame of image closest to the timestamp of the point cloud data from the pre-cached image frames. Subsequently, this set of point cloud data (i.e., the first point cloud data) and image data (i.e., the first image data) that are most matched in time are sent to the processing module together to achieve the fusion of the two. This method effectively solves the time synchronization problem caused by the difference in operating frequencies.

[0190] Optionally, when the first point cloud data and the first image data are obtained, the processed point cloud data and the processed image data may be obtained by the following steps:

[0191] Step S21: l X l Y l Z l Convert to O c X c Y c Z c First, a rotation operation is required, with O l X l , Y l , Z l Axis orientation and X c , Y c , Z c The axis directions correspond one to one. Then a translation operation is required to l O l Shift to O c On, at this time O l X l Y l Z l With O c X c Y c Z c Total overlap.

[0192] Step S22: Project the coordinate system from the positive direction of the Z axis to the XOY plane. Assuming that the angle is positive when the coordinate system is rotated clockwise, the original X l OY l The coordinate system rotates by angle α to reach X′ l OY′ l Position. The rotation formula is as follows:

[0193] X′ l =X l cosα-Y l sinα;

[0194] Y′ l =X l sinα+Y l cosα;

[0195] Written in matrix form:

[0196]

[0197] Similarly, assuming that the rotation angle around the Y axis is β and the rotation angle around the X axis is γ, we can get the rotation matrix of the coordinate system around the Y axis and the X axis. By multiplying the three rotation matrices in order, we can get Ol X l Y l Z l Rotate to O c X c Y c Z c The rotation matrix R when facing:

[0198]

[0199] Step S23: The rotated laser radar coordinate system is translated in the three directions of X, Y, and Z so that the two coordinate systems completely overlap. Assuming that the translation amounts of the three axes are Δx, Δy, and Δz respectively, the translation matrix T can be expressed as:

[0200]

[0201] Step S24: Finally, O is obtained l X l Y l Z l to c X c Y c Z c The transformation matrix, at this time the point cloud data can be projected into the camera coordinate system:

[0202]

[0203] Step S25: The data collected by the camera must first be c X c Y c Z c To the image coordinate system O i X i Y i , and then perform O i X i Y i The UOV conversion is finally output as a two-dimensional pixel result. c is the optical center of the camera, f is the focal length, from O c X c Y c Z c to i X i Y i The conversion formula is:

[0204]

[0205] From O i X i Y i The conversion formula to UOV is:

[0206]

[0207] In summary, we can conclude that O c X c Y c Z c The formula for conversion to UOV is:

[0208]

[0209] Where G is the camera intrinsic parameter matrix.

[0210] Step S26: Convert step S14 to 0 c X c Y c Z c The point cloud data in the image is further converted to the pixel coordinate system, so that the point cloud and the pixel are aligned, and finally the laser radar and the camera can scan the same target in the same coordinate system. The conversion formula is:

[0211]

[0212] Where [R|T] is the extrinsic parameter matrix of the camera.

[0213] Step S27: For the subsequent calculation of the distance between the tooth and the slot and the obstacle distance, it is also necessary to convert the pixel coordinates UOV to the laser radar three-dimensional coordinates O l X l Y l Z l By inverse operation of the matrix, the pixel coordinates (U, V) can be reversely converted to the camera coordinate system O c X c Y c Z c The three-dimensional coordinates (X c ,Y c ,Z c ):

[0214] Step S28: The conversion from the camera coordinate system to the lidar coordinate system can be completed by the inverse matrix of the rotation matrix R and the translation matrix T:

[0215] Through the above steps, the conversion from pixel coordinates to LiDAR three-dimensional coordinates can be realized. After the fusion of image data and point cloud data is completed, it is used to accurately detect the position and distance information of the pallet slot and obstacles.

[0216] The present invention aims at the deficiencies in the existing forklift pallet detection technology, and provides a forklift pallet slot detection method based on the fusion of laser radar and machine vision, which aims to improve the accuracy of pallet detection through multi-sensor information fusion technology, ensure the accuracy of fork teeth inserting pallets, and improve the safety and efficiency during operation. The main advantage of laser radar is that it can accurately measure the three-dimensional position and depth information of objects, and is independent of the ambient light conditions. It generates accurate point cloud data by emitting laser beams and receiving reflected signals, and can effectively identify the outline of objects in the environment, especially the position, size and shape of objects. This ability is very important for the precise positioning of forklift operations, especially when detecting pallet slots, it can provide accurate depth and space information. The advantage of machine vision (camera) is that it can identify the color, material, shape and other information of objects through image processing algorithms, and is particularly good at detecting textures and markings. Compared with laser radar, cameras can capture visual details in the environment, such as pallet labels, edges, color changes, etc. This is very helpful for forklifts to judge subtle target states during operation. By combining the depth perception of laser radar with the visual information of machine vision, the system can ensure accurate positioning of position and shape. This allows the system to efficiently and accurately detect tray slots even in complex or low-light environments.

[0217] Furthermore, the data transmission process specifically includes:

[0218] Step S81: The communication module transmits the video data from the camera to the vehicle display screen in the form of compressed streaming media. In order to ensure the smoothness and low latency of the video, H.264 / AVC encoding is used to compress the video stream. WI FI or 5G network is used as the wireless channel for data transmission to ensure high-speed and stable data transmission without adding additional cables.

[0219] Step S82: Each frame of the video stream will be accompanied by timestamp information and corresponding environmental parameters (such as the slot's three-dimensional coordinates and geometric dimensions, the latest calculated value of the distance obstacle, etc.) to ensure that timing information is not lost during data transmission. The packaging format is as follows:

[0220] Datapacket = [X s ,Y s ,Z s ,W s ,D s ,I camera ,D obs ,Timestamp];

[0221] Step S83: When transmitting data, the communication module uses CRC check to check each frame of data to ensure that no data is damaged or lost during the transmission process. If a transmission error is detected, the system will automatically request a retransmission to ensure that the driver receives complete and accurate video information.

[0222] Step S84: The communication module supports multi-channel video synchronous transmission, that is, the slot detection video and the obstacle detection video will be transmitted to the display screen at the same time and displayed on the screen in split-screen mode, ensuring that the driver can view both types of information at the same time when inserting and removing the tray and driving.

[0223] Compared with the prior art, this technical solution mainly solves the following technical problems:

[0224] 1. Inexperienced forklift drivers misjudge the slot position with their naked eyes: For example, during pallet handling, forklift drivers without a pallet detection system rely solely on their naked eyes to estimate the pallet position and drive the forklift based on experience to complete the cargo insertion and removal operations. In this operating mode, if the driver is unskilled, inexperienced, the pallet slot size is small, or the light is poor, it is easy to make operating errors, resulting in reduced operating efficiency, damage to cargo, and damage to equipment and facilities.

[0225] 2. Perception range problem: Ultrasonic radar and other sensors can only perceive objects within the horizontal range, and cannot detect height or side positions, which increases the difficulty of obstacle avoidance. This solution combines lidar and machine vision technology to expand the perception range of the forklift auxiliary operation system and improve recognition capabilities. This multi-sensor fusion method effectively solves the limitation that traditional ultrasonic radar can only perceive objects within the horizontal range, thereby improving the safety and reliability of operation.

[0226] 3. Problems with working at heights: When a forklift without a pallet detection system is performing high-altitude cargo storage and retrieval operations, the driver can only rely on the naked eye to judge the height and position of the cargo to perform storage and retrieval operations in the case of limited vision or insufficient light, which is prone to misjudgment. Such operations require experienced drivers, especially when performing storage and retrieval operations on high-rise shelves. It is easy to misjudge the position, height and distance of the operating pallet from surrounding cargo, increasing the risk of cargo falling or collision. It is also easy to cause low shelf volume ratio due to the inability to accurately position the pallet and the large spacing between cargo spaces. This technical solution combines YOLOv5's image detection and LiDAR scanning to obtain real-time three-dimensional position and height information of cargo and shelf spaces, helping drivers to accurately judge the relative position of cargo and shelves, thereby improving the safety and accuracy of high-altitude operations and avoiding dangers caused by visual blind spots or misjudgment.

[0227] 4. Data processing efficiency: The existing forklift pallet slot detection system still faces challenges in terms of computational complexity and latency when processing large amounts of data, which can affect the real-time performance and response speed of the system. This technical solution can reduce computational overhead while processing large-scale point cloud data by optimizing the clustering algorithm (DBSCAN) of point cloud data. In addition, by using multi-sensor information fusion technology, while maintaining high detection accuracy, it reduces redundant data processing and improves the overall response speed and real-time performance of the system.

[0228] 5. Detection accuracy issues: Existing forklift pallet detection systems can usually only handle relatively simple environments, and have limited ability to detect pallet slots in complex environments. This technical solution extracts possible slot areas through the DBSCAN algorithm of the integrated control module, and screens these areas in combination with geometric feature analysis. Furthermore, the system uses the YOLOv5 model to perform target detection on image data, and matches and fuses it with the point cloud data of the lidar to ultimately confirm the slot location. Through the deep fusion of multi-sensor information and the combined use of intelligent algorithms, the accuracy of pallet slot detection in complex environments can be significantly improved, ensuring the accuracy and stability of operations.

[0229] Compared with the existing technology, the beneficial effects of this technical solution are as follows:

[0230] 1. Improve detection accuracy: The invented forklift pallet slot detection system realizes multi-sensor data fusion through the combination of laser radar and machine vision technology, overcoming the limitations of traditional machine vision in complex environments such as low light, occlusion, and reflection. Laser radar provides accurate three-dimensional environmental data, which is especially suitable for working in complex storage environments, while machine vision can identify details such as the appearance and shape of the pallet slot. The combination of the two significantly improves the accuracy of pallet slot detection and ensures the accuracy of the fork tines when inserting the pallet.

[0231] 2. Enhanced environmental adaptability: Since the present invention combines laser radar and machine vision technology, it can handle more complex environments. Laser radar can provide accurate three-dimensional environmental information, while machine vision can identify and understand objects and structures in the environment. The combination of the two can achieve effective adaptation to complex environments.

[0232] 3. Improve real-time performance and response speed: The data processing unit of the present invention includes a filtering module and a coordinate conversion module, which can effectively remove noise and synchronize data, thereby improving the real-time performance and response speed of the system. In addition, the integrated control module of the present invention confirms the slot position through the DBSCAN algorithm and geometric feature analysis, and can quickly and accurately make operational suggestions.

[0233] 4. Improve operational efficiency: Based on the fusion data of multiple sensors, this system can quickly and accurately detect the position of the pallet slot, reducing the driver's manual adjustment operations. The system provides real-time adjustment suggestions (such as "move to the left by Xmm" or "drop Zmm", etc.) through the on-board display screen, and automatically adjusts the movement of the forklift's teeth in combination with the automatic control function. This greatly reduces the driver's judgment and operation time, and significantly improves the operating efficiency of the forklift. At the same time, the system can speed up the logistics operation process and improve the overall efficiency of warehousing and handling through precise detection and automated slot adjustment functions.

[0234] 5. Reduce the risk of damage: The system can accurately locate the pallet slot and provide real-time operation suggestion information. Combined with the automatic control function, it can automatically fine-tune the fork tine when inserting or removing the pallet. This can not only effectively avoid damage to the goods or pallets caused by inaccurate positioning, but also ensure the smooth and accurate operation of the tine through automatic adjustment, further reducing the damage rate caused by operational errors. The automatic control function can make adjustments quickly and accurately in complex environments, greatly improving the safety and reliability of operations, thereby reducing the risk of equipment wear and cargo damage.

[0235] 6. Improve storage space utilization: The system of the present invention can effectively improve the compactness of cargo stacking through precise detection and automatic adjustment functions. During the stacking process, the system can perform detailed toothing operations and fine-tuning according to the detailed size and position information of the cargo and the empty space to ensure that the cargo can be placed tightly and accurately on the pallet or shelf. This not only increases the volume ratio of the cargo, but also effectively reduces the waste of storage space, thereby improving the overall efficiency of storage and logistics.

[0236] Example 2

[0237] According to another aspect of an embodiment of the present invention, a device is further provided, which can execute the method provided in the above embodiment 1. The specific implementation method and preferred application scenario are the same as those of the above embodiment 1 and will not be described in detail here.

[0238] Figure 3 is a schematic diagram of a forklift control device according to an embodiment of the present invention. Figure 3 As shown, the device comprises:

[0239] An acquisition module 32, used to acquire environmental data in front of the forklift through a plurality of different types of sensors installed on the forklift, wherein the plurality of different types of sensors at least include: a laser radar sensor and a visual sensor;

[0240] The processing module 34 is used to perform pallet slot detection based on the environmental data to obtain a first distance between the forklift's inserting teeth and the pallet slot and first deviation information between the inserting teeth and the pallet slot, perform cargo and shelf vacancy detection based on the environmental data to obtain second deviation information between the cargo and the shelf vacancy, and perform obstacle recognition based on the environmental data to obtain a second distance between the forklift and the obstacle;

[0241] The adjustment module 36 is used to adjust the forklift based on the first distance, the deviation information and the second distance.

[0242] Optionally, the processing module includes: a first determination unit, used to determine the position information of the forklift's tooth based on the forklift's position information; a detection unit, used to perform pallet slot detection based on environmental data to obtain the position information of the pallet slot, perform cargo and shelf space detection based on environmental data to obtain the cargo position information and the shelf space position information, and perform obstacle identification based on environmental data to obtain the obstacle position information; a second determination unit, used to determine the first distance and the first deviation information based on the position information of the tooth and the pallet slot; a third determination unit, used to determine the second deviation information based on the cargo position information and the shelf space position information; and a fourth determination unit, used to determine the second distance based on the position information of the tooth and the obstacle position information.

[0243] Optionally, the detection unit includes: a first acquisition subunit, used to acquire initial point cloud data of the pallet through a laser radar sensor installed on a forklift; a first processing subunit, used to process the initial point cloud data to obtain point cloud data after removing outliers; a clustering subunit, used to cluster the point cloud data to obtain at least one cluster; an extraction subunit, used to extract geometric features from at least one cluster to obtain at least one geometric feature; a second processing subunit, used to obtain at least one pixel coordinate of the pallet based on at least one geometric feature and a preset size range of the pallet; and a third processing subunit, used to obtain position information of the pallet slot based on at least one pixel coordinate and image data of the pallet.

[0244] Optionally, the geometric features include: the width, depth and height of the tray, and the preset size range includes: a preset width range, a preset depth range and a preset height range of the tray; the second processing subunit is also used to: in response to the width of the tray satisfying the preset width range, the depth of the tray satisfying the preset depth range, and the height of the tray satisfying the preset height range, determine that at least one cluster corresponding to at least one geometric feature is at least one candidate cluster; obtain at least one geometric center coordinate of at least one candidate cluster, and perform coordinate transformation on at least one geometric center coordinate to obtain at least one pixel coordinate.

[0245] Optionally, the third processing subunit is also used to: obtain the center pixel coordinates of the image data; obtain the distance between the center pixel coordinates and at least one pixel coordinate to obtain at least one third distance; determine that the distance with the smallest absolute value in at least one third distance is the target distance; perform coordinate conversion on the pixel coordinates corresponding to the target distance to obtain position information.

[0246] Optionally, the position information of the pallet slot includes at least: the three-dimensional coordinates of the pallet slot and the cluster coordinates of the pallet slot, the cluster coordinates are the coordinates obtained after clustering the point cloud data of the pallet slot; the second determination unit also includes: a fourth processing subunit, used to obtain the width and height of the pallet slot based on the three-dimensional coordinates of the pallet slot and the cluster coordinates; a judgment subunit, used to judge whether the width of the pallet slot is greater than or equal to the width of the tooth, and whether the height of the pallet slot is greater than or equal to the thickness of the tooth; a determination subunit, used to determine the first distance based on the three-dimensional coordinates of the tooth and the three-dimensional coordinates of the slot in response to the width of the pallet slot being greater than or equal to the width of the tooth, and the height of the pallet slot being greater than or equal to the thickness of the tooth.

[0247] Optionally, the second determination unit also includes: a second acquisition subunit, used to obtain the horizontal coordinate value in the position information of the insert tooth, and the difference between the horizontal coordinate value in the position information of the tray slot, to obtain a first deviation; a third acquisition subunit, used to obtain the vertical coordinate value in the position information of the insert tooth, and the difference between the vertical coordinate value in the position information of the tray slot, to obtain a second deviation; a summarizing subunit, used to summarize the first deviation and the second deviation to obtain the first deviation information.

[0248] Optionally, the third determination form includes: a fifth processing subunit, for obtaining the left position and the right position of the goods based on the position information of the goods; a sixth processing subunit, for obtaining the left and right boundaries of the empty shelf space based on the position information of the empty shelf space; a seventh processing subunit, for obtaining a third deviation between the left side position of the goods and the left boundary of the empty shelf space based on the left side position of the goods and the left boundary of the empty shelf space; an eighth processing subunit, for obtaining a fourth deviation between the right side position of the goods and the right boundary of the shelf based on the right side position of the goods and the right boundary of the empty shelf space; and a ninth processing subunit, for obtaining second deviation information based on the third deviation and the fourth deviation.

[0249] Optionally, the adjustment module includes: a first adjustment unit, used to adjust the position of the forklift based on the first distance; a second adjustment unit, used to adjust the position of the insert tooth in the horizontal direction and the vertical direction based on the first deviation information; a third adjustment unit, used to adjust the position of the insert tooth in the horizontal direction based on the second deviation information; and a fourth adjustment unit, used to adjust the speed of the forklift based on the second distance.

[0250] Optionally, the first adjustment unit includes: a first control subunit, used to control the forklift to move toward the slot in response to the first distance being greater than the first preset distance; a second control subunit, used to control the insert tooth to align with the slot in response to the first distance being less than the first preset distance; and a third control subunit, used to control the insert tooth to perform an insertion operation in response to the first distance being less than the second preset distance, wherein the second preset distance is less than the first preset distance.

[0251] Optionally, the second adjustment unit includes: a first adjustment subunit, used for adjusting the position of the inserting tooth in a horizontal direction based on the first deviation; and a second adjustment subunit, used for adjusting the position of the inserting tooth in a vertical direction based on the second deviation.

[0252] Optionally, the first adjustment subunit is further used for: in response to the first deviation being smaller than the first preset deviation, controlling the insert gear to move rightward; in response to the first deviation being larger than the first preset deviation, controlling the insert gear to move leftward.

[0253] Optionally, the second adjustment subunit is further configured to: in response to the second deviation being greater than a second preset deviation, control the inserting tooth to move downward.

[0254] Optionally, the third adjustment unit includes: a fourth control subunit, used to control the insert gear to move leftward in response to the second deviation information being greater than a preset threshold; and a fifth control subunit, used to control the insert gear to move rightward in response to the second deviation information being less than a preset threshold.

[0255] Optionally, the fourth adjustment unit includes: a judgment subunit, for not adjusting the speed in response to the second distance being greater than a preset safety distance; and a sixth control subunit, for controlling the forklift to decelerate in response to the second distance being less than or equal to the preset safety distance.

[0256] Optionally, the device also includes: a fusion module, used to perform information fusion on the point cloud data obtained by the lidar sensor and the image data obtained by the visual sensor to obtain processed point cloud data and processed image data; and a forklift adjustment module, used to adjust the forklift based on the processed point cloud data and the processed image data.

[0257] Optionally, the fusion module includes: a time alignment unit, used to perform time alignment on the point cloud data and the image data to obtain first point cloud data and first image data, wherein the first point cloud data is synchronized with the first image data in time; a coordinate transformation unit, used to perform coordinate transformation on the first point cloud data and the first image data to obtain processed point cloud data and processed image data.

[0258] Example 3

[0259] An embodiment of the present application further provides an electronic device, comprising: a memory storing an executable program; and a processor for running the program, wherein the program executes the methods in various embodiments of the present invention when running.

[0260] Example 4

[0261] An embodiment of the present application further provides a computer-readable storage medium, which includes a stored executable program, wherein when the executable program is running, the device where the computer-readable storage medium is located is controlled to execute the methods in various embodiments of the present invention.

[0262] Example 5

[0263] An embodiment of the present application further provides a computer program product, including a computer program, which implements the methods in various embodiments of the present invention when executed by a processor.

[0264] Example 6

[0265] An embodiment of the present application further provides a computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium is used to store a computer program, and when the computer program is executed by a processor, the method in each embodiment of the present invention is implemented.

[0266] Example 7

[0267] The embodiments of the present application further provide a computer program, which implements the methods in the above-mentioned embodiments of the present invention when executed by a processor.

[0268] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0269] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0270] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0271] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0272] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0273] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0274] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A forklift control method, characterized in that: include: Acquiring environmental data in front of the forklift by using a plurality of different types of sensors installed on the forklift, wherein the plurality of different types of sensors at least include: a laser radar sensor and a visual sensor; Performing pallet slot detection based on the environmental data to obtain a first distance between the forklift and the pallet slot and first deviation information between the forklift and the pallet slot, performing cargo and shelf space detection based on the environmental data to obtain second deviation information between the cargo and shelf space, and performing obstacle recognition based on the environmental data to obtain a second distance between the forklift and the obstacle; The forklift is adjusted based on the first distance, the deviation information, and the second distance.

2. The method according to claim 1, characterized in that Performing pallet slot detection based on the environmental data to obtain a first distance between the forklift and the pallet slot and first deviation information between the forklift and the pallet slot, performing cargo and shelf space detection based on the environmental data to obtain second deviation information between the cargo and shelf space, and performing obstacle recognition based on the environmental data to obtain a second distance between the forklift and the obstacle, including: Based on the position information of the forklift, determining the position information of the inserting teeth of the forklift; Performing pallet slot detection based on the environmental data to obtain location information of the pallet slot, performing cargo and shelf space detection based on the environmental data to obtain location information of the cargo and location information of the shelf space, and performing obstacle recognition based on the environmental data to obtain location information of the obstacle; Determine the first distance and the first deviation information based on the position information of the inserting teeth and the position information of the tray slot; Determine the second deviation information based on the location information of the goods and the location information of the empty shelf; The second distance is determined based on the position information of the insert and the position information of the obstacle.

3. The method according to claim 2, characterized in that Performing tray slot detection based on the environmental data to obtain position information of the tray slot includes: Acquiring initial point cloud data of the pallet by using a laser radar sensor installed on the forklift; Processing the initial point cloud data to obtain point cloud data after removing outliers; Clustering the point cloud data to obtain at least one cluster; Extracting geometric features from the at least one cluster to obtain at least one geometric feature; Based on the at least one geometric feature and a preset size range of the tray, obtaining at least one pixel coordinate of the tray; Based on the at least one pixel coordinate and the image data of the tray, position information of the tray slot is obtained.

4. The method according to claim 3, characterized in that The geometric features include: the width, depth and height of the tray, and the preset size range includes: a preset width range, a preset depth range and a preset height range of the tray; based on the geometric features and the preset size range of the tray, obtaining at least one pixel coordinate of the tray includes: In response to the width of the tray satisfying the preset width range, the depth of the tray satisfying the preset depth range, and the height of the tray satisfying the preset height range, determining that the at least one cluster corresponding to the at least one geometric feature is at least one candidate cluster; At least one geometric center coordinate of the at least one candidate cluster is acquired, and coordinate transformation is performed on the at least one geometric center coordinate to obtain the at least one pixel coordinate.

5. The method according to claim 4, characterized in that Based on the at least one pixel coordinate and the image of the pallet, obtaining the position information of the pallet includes: Obtaining the center pixel coordinates of the image data; Acquire the distance between the central pixel coordinate and the at least one pixel coordinate to obtain at least one third distance; Determine, among the at least one third distance, a distance with the smallest absolute value as the target distance; The pixel coordinates corresponding to the target distance are converted into coordinates to obtain the position information.

6. The method according to claim 2, characterized in that The position information of the tray slot includes at least: the three-dimensional coordinates of the tray slot and the cluster coordinates of the tray slot, where the cluster coordinates are coordinates obtained by clustering the point cloud data of the tray slot; determining the first distance based on the position information of the insert gear and the position information of the tray slot includes: Based on the three-dimensional coordinates of the tray slot and the cluster coordinates, obtaining the width and height of the tray slot; Determine whether the width of the tray slot is greater than or equal to the width of the inserting teeth, and whether the height of the tray slot is greater than or equal to the thickness of the inserting teeth; In response to the width of the tray slot being greater than or equal to the width of the insert tooth and the height of the tray slot being greater than or equal to the thickness of the insert tooth, the first distance is determined based on the three-dimensional coordinates of the insert tooth and the three-dimensional coordinates of the slot.

7. The method according to claim 2, characterized in that Determining the first deviation information based on the position information of the inserting tooth and the position information of the tray slot includes: Obtaining a difference between a horizontal coordinate value in the position information of the inserting tooth and a horizontal coordinate value in the position information of the tray slot to obtain a first deviation; Obtaining a difference between a vertical coordinate value in the position information of the inserting tooth and a vertical coordinate value in the position information of the tray slot to obtain a second deviation; The first deviation and the second deviation are aggregated to obtain the first deviation information.

8. The method according to claim 2, characterized in that: Determining the second deviation information based on the location information of the goods and the location information of the empty shelf position includes: Based on the position information of the goods, obtaining the left position and the right position of the goods; Based on the position information of the empty shelf space, obtaining the left boundary and the right boundary of the empty shelf space; Based on the left side position of the goods and the left boundary of the empty shelf space, obtaining a third deviation between the left side position of the goods and the left boundary of the empty shelf space; Based on the right side position of the goods and the right boundary of the empty shelf, obtaining a fourth deviation between the right side position of the goods and the right boundary of the shelf; The second deviation information is obtained based on the third deviation and the fourth deviation.

9. The method according to any one of claims 1 to 8, characterized in that Adjusting the forklift based on the first distance, the deviation information and the second distance includes: adjusting the position of the forklift based on the first distance; Adjusting the position of the inserting teeth in the horizontal direction and the vertical direction based on the first deviation information; Adjusting the position of the inserting teeth in a horizontal direction based on the second deviation information; The speed of the forklift is adjusted based on the second distance.

10. The method according to claim 9, characterized in that Adjusting the position of the forklift based on the first distance includes: In response to the first distance being greater than a first preset distance, controlling the forklift to move toward the slot; In response to the first distance being smaller than the first preset distance, controlling the inserting tooth to align with the slot; In response to the first distance being smaller than a second preset distance, the inserting teeth are controlled to perform an inserting operation, wherein the second preset distance is smaller than the first preset distance.

11. The method according to claim 9, characterized in that Adjusting the position of the inserting tooth in a horizontal direction and a vertical direction based on the first deviation information includes: Based on the first deviation, adjusting the position of the inserting teeth in a horizontal direction; Based on the second deviation, the position of the insert tooth is adjusted in a vertical direction.

12. The method according to claim 9, characterized in that Based on the first deviation, adjusting the position of the inserting teeth in a horizontal direction includes: In response to the first deviation being smaller than a first preset deviation, controlling the inserting gear to move rightward; In response to the first deviation being greater than the first preset deviation, the inserting tooth is controlled to move leftward.

13. The method according to claim 9, characterized in that Based on the second deviation, the position of the inserting teeth is adjusted in a vertical direction, comprising: In response to the second deviation being greater than a second preset deviation, the inserting teeth are controlled to move downward.

14. The method according to claim 9, characterized in that Adjusting the position of the inserting tooth in a horizontal direction based on the second deviation information includes: In response to the second deviation information being greater than a preset threshold, controlling the inserting gear to move leftward; In response to the second deviation information being smaller than the preset threshold, the insert tooth is controlled to move rightward.

15. The method according to claim 9, characterized in that Adjusting the speed of the forklift based on the second distance includes the following: In response to the second distance being greater than a preset safety distance, not adjusting the speed; In response to the second distance being less than or equal to the preset safety distance, the forklift is controlled to decelerate.

16. The method according to any one of claims 1 to 8, characterized in that The method further comprises: Performing information fusion on the point cloud data acquired by the laser radar sensor and the image data acquired by the visual sensor to obtain processed point cloud data and processed image data; The forklift is adjusted based on the processed point cloud data and the processed image data.

17. The method according to claim 16, characterized in that The point cloud data acquired by the laser radar sensor and the image data acquired by the visual sensor are fused to obtain processed point cloud data and processed image data, including: Performing time alignment on the point cloud data and the image data to obtain first point cloud data and first image data, wherein the first point cloud data is synchronized with the first image data in time; Coordinate transformation is performed on the first point cloud data and the first image data to obtain the processed point cloud data and the processed image data.

18. A forklift control device, characterized in that: include: An acquisition module, used to acquire environmental data in front of the forklift through a plurality of different types of sensors installed on the forklift, wherein the plurality of different types of sensors at least include: a laser radar sensor and a visual sensor; a processing module, configured to perform pallet slot detection based on the environmental data to obtain a first distance between a forklift and the pallet slot and first deviation information between the forklift and the pallet slot, perform cargo and shelf vacancy detection based on the environmental data to obtain second deviation information between the cargo and the shelf vacancy, and perform obstacle recognition based on the environmental data to obtain a second distance between the forklift and the obstacle; An adjustment module is used to adjust the forklift based on the first distance, the deviation information and the second distance.

19. An electronic device, characterized in that: include: A memory storing an executable program; A processor, configured to run the program, wherein the program executes the method according to any one of claims 1 to 17 when running.

20. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored executable program, wherein when the executable program is executed, the device where the storage medium is located is controlled to execute the method according to any one of claims 1 to 18.

21. A computer program product, characterized in that It comprises a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 18.

Images