Stacker control method and device, electronic equipment and medium

Abstract

The invention belongs to the technical field of stacker control, and particularly discloses a stacker control method and device, electronic equipment and a medium, and the method comprises the steps: enabling a stacker to travel in a working environment in advance, calculating the pose of a binocular camera through a high-precision integrated positioning system, observing a road sign through the binocular camera, and forming a visual point cloud map; acquiring spatial position distribution of to-be-carried materials in the current environment based on the binocular camera, and projecting the spatial position distribution to the visual point cloud map by utilizing spatial position transformation to generate a semantic map; the stacker moves according to the semantic map to arrive at a target position to work; and a to-be-carried material is recognized through the binocular camera, and the to-be-carried material is carried. When the stacking machine works in a complex and huge environment, accurate positioning, accurate moving to a carrying point and accurate carrying can be achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of stacker control, and in particular to a stacker control method, device, electronic equipment and medium. Background Art

[0002] High-Precision Integrated Positioning System (HPIPS) is a system that integrates multiple high-precision sensors, and aims to provide stable and high-precision positioning and attitude information through multi-sensor data fusion technology. The system is widely used in the fields of autonomous driving, vehicle-mounted mapping, drone navigation, etc., and is one of the key technologies to ensure accurate positioning in complex environments.

[0003] In recent years, the traditional manufacturing industry and logistics industry are undergoing tremendous changes. The traditional production system is developing in the direction of networking and digitalization, and the logistics system is developing in the direction of intelligence and automation. Digital production and manufacturing refers to the digitalization of the manufacturing field. It is the result of the intersection, integration, development and application of manufacturing technology, computer technology, network technology and management science. It is also the inevitable trend for manufacturing enterprises, manufacturing systems and production processes and production systems to continuously realize digitalization. In the digital production logistics system, stackers play a vital role. Stackers can automatically complete the loading, unloading, handling, stacking and other operations of goods without manual intervention, thereby greatly reducing labor costs. In addition, the automated operation of stackers can also reduce the labor intensity of workers and improve work safety. With the widespread application of stackers, a large amount of manual handling and stacking work required in traditional warehousing operations has been effectively replaced, and enterprises can reduce personnel requirements and reduce labor costs accordingly.

[0004] Existing stackers generally use infrared sensors, ultrasonic sensors, etc. for distance measurement, and optical distance sensors, barcode positioning systems, etc. for positioning. Traditional stackers are mainly used in simple working conditions. However, in complex working conditions, the number of cargo positions per layer in the stacker warehouse aisle is usually hundreds, and the aisle length is usually more than tens of meters. The aisle environment is complex, and the following problems may occur when the traditional stacker is automatically running: 1. Increased storage, retrieval and handling time: When a stacker with poor accuracy works automatically, it takes more time to adjust its position due to inaccurate positioning, which increases the time for storage, retrieval and handling of a single item. 2. Increased error rate: Insufficient accuracy may cause errors in the storage and handling process of the stacker, such as misplacement of goods, damage to goods, etc., which require extra time to handle; 3. Collision risk: During the movement, a stacker with poor accuracy may collide with shelves, other equipment or obstacles due to inaccurate positioning, causing equipment damage or casualties; 4. Risk of cargo falling: Due to insufficient precision, the stacker may not be able to stably grasp or place cargo when transporting cargo, causing the cargo to fall and posing a safety hazard. Summary of the invention

[0005] In view of the above-mentioned shortcomings of the prior art, an object of the present invention is to provide a stacker control method for solving the problems in the prior art that the stacker cannot be accurately positioned when working in a complex and large environment, resulting in inaccurate walking routes and inability to align goods.

[0006] To achieve the above-mentioned object and other related objects, the present invention provides a control method for a stacker, wherein a binocular camera and a working camera are provided on the stacker, wherein the binocular camera includes a left camera and a right camera, and a high-precision integrated positioning system is provided in the stacker. The stacker moves and works in a working area, and a plurality of landmarks are provided in the working area, including: The stacker travels in advance in the working environment, calculates the position and posture of the binocular camera using the high-precision integrated positioning system, observes the road signs through the binocular camera, and forms a visual point cloud map; When the stacker is working, the binocular camera is used to obtain the spatial position distribution of the materials to be transported in the current environment, and the spatial position distribution is projected onto the visual point cloud map by spatial position transformation to generate a semantic map containing information of the materials to be transported; The stacker performs path planning according to the semantic map and autonomously reaches the target position to work; the materials to be transported are identified by the binocular camera and the materials to be transported are transported.

[0007] Optionally, calculating the position and posture of the camera using the binocular camera and the high-precision integrated positioning system specifically includes: The high-precision integrated positioning system is used to calculate the position and posture of the inertial navigation system in the world coordinate system, calibrate the spatial relationship between the inertial navigation system and the monocular camera, and calculate the position and posture of the monocular camera in the world coordinate system through the mapping relationship among the world coordinate system, the camera coordinate system and the pixel coordinate system. The mapping relationship among the three is as follows: In the formula, the world coordinate system of a certain spatial point is , the corresponding camera coordinate system is , the pixel coordinate system is ; Calculated by dividing the focal length by the pixel length in the respective direction; , The coordinates of the principal point, in pixels, The matrix is ​​called the camera intrinsic parameter matrix, is the rotation matrix from the earth coordinate system to the camera coordinate system, is the translation amount; Perform binocular calibration on the binocular camera and add depth information based on the position and posture of the monocular camera in the world coordinate system to obtain the position and posture of the binocular camera in the world coordinate system: in, is the parallax of the left and right cameras in the binocular camera, is the baseline length of the binocular camera; The optical center of the binocular camera is zero in the camera coordinate system, and the rotation matrix and translation , the position and posture of the binocular camera in the world coordinate system are obtained. The mapping relationship between the optical center of the binocular camera in the camera coordinate system and its position and posture in the world coordinate system is as follows: That is, the position of the binocular camera in the world coordinate system is .

[0008] Optionally, the binocular camera is used to observe road signs to form a visual point cloud map, and the specific steps include: The binocular camera collects travel data, uses a deep learning method to perform semantic segmentation and target recognition on the travel data, sets the pixel values ​​of the identified staff and work equipment as masks to remove them, and leaves a static scene; The SIFT operator is used to extract feature points of the static scene image, and feature point matching is performed through feature point coordinates and their descriptors. For each of the road signs, after feature matching, a plurality of pixel observation values ​​are obtained, and the pixel observation values ​​are pixel coordinates and disparity of a binocular camera. The observed normal feature points are associated with the corresponding multiple feature descriptors to generate a visual point cloud map.

[0009] Optionally, the feature point matching includes matching of left and right images of the same frame and multi-frame matching of the left image, wherein: For the same-name points in the left and right images of the same frame, if the row coordinates of the left and right same-name points differ by more than 1 pixel, they are not matched; For multi-frame matching of the left image, the points with the same name in adjacent frames need to satisfy the epipolar constraint condition to be matching points, that is, satisfy the following formula: In the formula, is the pixel coordinate, is the basic matrix, through the camera intrinsic matrix And the spatial relationship between two adjacent frames Get, through the position and posture of the binocular camera in the world coordinate system Calculate the pose of adjacent images , The poses of the adjacent images The specific formula is as follows: .

[0010] Optionally, the stacker moves according to the semantic map to reach a target location to work, specifically including: Determine the target location of the handling position based on the semantic map; Integrate the transportable information of all current positions to be transported by the stacker, combine the current position of the stacker, and plan the transport route based on the semantic map; The movement is performed according to the transport route.

[0011] Optionally, identifying the material to be transported by the binocular camera and transporting the material to be transported specifically includes: The working camera is arranged at the end of the lifting mechanism of the stacker, and the optimal relative posture relationship between the working camera and the material to be transported is calculated in advance according to the position of the material to be transported in the semantic map; After reaching the target position, the lifting mechanism is controlled to adjust the posture according to the optimal point-to-posture relationship to transport the material.

[0012] Optionally, when the lifting mechanism is transporting materials, the binocular camera collects image data in real time, identifies the materials in the image data based on a deep learning method, and fine-tunes the lifting mechanism to ensure that the materials are located in the central area of ​​the image.

[0013] The second aspect of the present invention further provides a stacker control device, wherein a binocular camera and a working camera are provided on the stacker, wherein the binocular camera includes a left camera and a right camera, wherein a high-precision integrated positioning system is provided in the stacker, wherein the stacker moves and works in a working area, wherein a plurality of road signs are provided in the working area, and wherein the device comprises: Visual point cloud map generation module: used for the stacker to travel in advance in the working environment, calculate the position and posture of the binocular camera using the high-precision integrated positioning system, observe the road signs through the binocular camera, and form a visual point cloud map; Semantic map generation module: used for obtaining the spatial position distribution of the materials to be transported in the current environment by using the binocular camera when the stacker is working, and projecting the spatial position distribution onto the visual point cloud map by using spatial position transformation to generate a semantic map containing information of the materials to be transported; Stacker working module: used for the stacker to plan the path according to the semantic map, and autonomously reach the target position to work; identify the materials to be transported through the binocular camera, and transport the materials to be transported.

[0014] The third aspect of the present invention also provides an electronic device, comprising: a processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium through the bus, and the processor executes the machine-readable instructions to perform the steps of the stacker control method as described in any one of claims 1 to 8.

[0015] A fourth aspect of the present invention further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program executes the steps of the stacker control method according to any one of claims 1 to 8 when executed by a processor.

[0016] As described above, a stacker control method, device, electronic device and medium proposed in the present invention have the following beneficial effects: the present invention obtains a visual point cloud map by performing data fusion through a high-precision integrated positioning system and a binocular camera, which provides high-precision background field data for generating semantic maps to assist multi-source fusion positioning in complex environments, and obtain accurate stacker posture; the semantic map generated based on the visual point cloud map provides accurate navigation data for the stacker during movement and transportation, thereby improving the practicality of the stacker. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 Shown is a flow chart of a stacker control method according to an embodiment of the present invention; Figure 2 Shown is a schematic structural diagram of a stacker control device according to an embodiment of the present invention; Figure 3 Shown is a schematic structural diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0018] The following describes the embodiments of the present invention through specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention.

[0019] It should be noted that the diagram provided in the present embodiment only illustrates the basic concept of the present invention in a schematic manner, so the diagram only shows the components related to the present invention rather than drawing according to the number, shape and size of the components during actual implementation. The type, quantity and ratio of each component during actual implementation can be a random change, and the component layout type may also be more complicated. The structure, ratio, size, etc. illustrated in the drawings of the present specification are only used to match the content disclosed in the specification for people familiar with this technology to understand and read, and are not used to limit the limiting conditions that the present invention can implement, so they have no technical substantive significance. Any modification of the structure, change of the proportional relationship or adjustment of the size should still fall within the scope of the technical content disclosed by the present invention without affecting the effect that the present invention can produce and the purpose that can be achieved. At the same time, the terms such as "upper", "lower", "left", "right", "middle" and "one" quoted in this specification are only for the convenience of narration, and are not used to limit the scope of the present invention. The change or adjustment of its relative relationship should also be regarded as the scope of the present invention without substantially changing the technical content.

[0020] Embodiment 1 See also Figure 1 , Figure 1 A flowchart of a stacker control method provided in an embodiment of the present application. Figure 1 As shown, this embodiment discloses a stacker control method, including: a binocular camera and a working camera are arranged on the stacker, the binocular camera includes a left camera and a right camera, a high-precision integrated positioning system is arranged in the stacker, the stacker moves and works in a working area, a plurality of road signs are arranged in the working area, and the stacker control method includes: S10: The stacker pre-travels in the working environment, uses a high-precision integrated positioning system to calculate the position and posture of the binocular camera, and observes the road signs through the binocular camera to form a visual point cloud map; Among them, the binocular camera and high-precision integrated positioning system are used to calculate the camera's position and posture, including: The position and posture of the inertial navigation system in the world coordinate system is calculated through a high-precision integrated positioning system, the spatial relationship between the inertial navigation system and the monocular camera is calibrated, and the position and posture of the monocular camera in the world coordinate system is calculated through the mapping relationship between the world coordinate system, the camera coordinate system and the pixel coordinate system. The mapping relationship between the three is as follows: In the formula, the world coordinate system of a certain spatial point is , the corresponding camera coordinate system is , the pixel coordinate system is ; Calculated by dividing the focal length by the pixel length in the respective direction; , The coordinates of the principal point, in pixels, The matrix is ​​called the camera intrinsic parameter matrix, is the rotation matrix from the earth coordinate system to the camera coordinate system, is the translation amount; the pixel coordinates in the above formula have been corrected for distortion, and the intrinsic parameter matrix and distortion parameters can be obtained by Zhang Zhengyou calibration method; and in the above formula, the left camera and the right camera are aligned through binocular correction, that is, the optical axes of the two cameras are parallel and forward, the left and right image planes are coplanar, and the epipolar lines are parallel. The left and right cameras can be directly corrected for binocular correction using Matlab tools.

[0021] Perform binocular calibration on the binocular camera and add depth information based on the position and posture of the monocular camera in the world coordinate system to obtain the position and posture of the binocular camera in the world coordinate system: in, is the parallax of the left and right cameras in the binocular camera, is the baseline length of the binocular camera; The optical center of the binocular camera is zero in the camera coordinate system, through the rotation matrix and translation , the position and posture of the binocular camera in the world coordinate system are obtained. The mapping relationship between the optical center of the binocular camera in the camera coordinate system and its position and posture in the world coordinate system is as follows: That is, the position of the binocular camera in the world coordinate system is .

[0022] Among them, when the stacker travels in the working environment, the binocular camera observes the road signs to form a visual point cloud map. The specific steps include: S11: The binocular camera collects travel data, and uses deep learning methods to perform semantic segmentation and target recognition on the travel data. The pixel values ​​of the identified staff and work equipment are set as masks to remove them, leaving a static scene. Specifically, the collected image or video data is preprocessed by scaling, cropping, normalizing, frame extraction, etc.; Input the preprocessed image into the trained PSPNet or YOLOv3 model, and the model outputs the category label of each pixel; According to the category label output by the model, the pixel values ​​belonging to dynamic objects (such as workers, work equipment) are set to masks (usually 0 or a specific value) so that these areas can be removed in subsequent processing.

[0023] S12: The SIFT operator is used to extract feature points of the static scene image. Feature point matching is performed through the feature point coordinates and their descriptors. For each landmark, after feature matching, multiple pixel observation values ​​are obtained. The pixel observation values ​​are the pixel coordinates and disparity of the binocular camera. The SIFT operator is used to extract feature points and their descriptors from static scene images that have been processed for semantic segmentation and target recognition. Since the binocular camera has been calibrated, the same-name points of the left and right cameras should be on the same line, and the depth is calculated by disparity. For the left camera, the current frame is matched with the previous frames to find all possible same-name points. The epipolar constraint and disparity constraint are used to remove incorrect matching points.

[0024] Among them, feature point matching includes matching of left and right images of the same frame and multi-frame matching of the left image, among which, For the same-name points in the left and right images of the same frame, since both the left and right cameras have undergone binocular calibration, the pixel coordinates of the same-name points should be in the same row. Therefore, if the row coordinates of the same-name points on the left and right differ by more than 1 pixel, they are not matched; For multi-frame matching of the left image, the points with the same name in adjacent frames need to satisfy the epipolar constraint condition to be matching points, that is, satisfy the following formula: In the formula, is the pixel coordinate, is the basic matrix, through the camera intrinsic matrix And the spatial relationship between two adjacent frames The geometric meaning of the formula is: correspond The epipolar lines on the image are , Point on the polar line So the point to the extreme line The distance is 0. Through the position of the binocular camera in the world coordinate system Calculate the pose of adjacent images , The poses of adjacent images The specific formula is as follows: Through the above two types of constraints, feature points with incorrect matching can be removed; in addition, if a feature point does not match on the left and right images, it will be directly eliminated and will not participate in the matching between multiple frames.

[0025] S13: Associating the observed normal feature points with the corresponding multiple feature descriptors to generate a visual point cloud map.

[0026] The generated visual point cloud map is entered into the database for subsequent visual positioning.

[0027] S20: When the stacker is working, the binocular camera is used to obtain the spatial position distribution of the materials to be transported in the current environment, and the spatial position distribution is projected onto the visual point cloud map by spatial position transformation to generate a semantic map containing information of the materials to be transported; Specifically, the binocular camera observes the material to be transported to obtain its spatial position distribution, which is similar to the binocular camera observing the road sign. The difference is that after the binocular camera collects image data and matches features, it needs 3D reconstruction and coordinate conversion: Using the matched feature points, the three-dimensional coordinates of the material to be transported in the camera coordinate system are reconstructed through triangulation calculation method; Using a high-precision integrated positioning system and a binocular camera, the three-dimensional coordinates of the material to be transported in the camera coordinate system are converted into world coordinates to obtain its position in the global space.

[0028] S30: The stacker plans the path according to the semantic map and reaches the target position autonomously to work; the stacker identifies the materials to be transported through the binocular camera and transports the materials.

[0029] The stacker moves according to the semantic map and reaches the target location to work, including: Determine the target location of the handling position based on the semantic map; Integrate the transportable information of all the current positions to be transported by the stacker, combine it with the current position of the stacker, and plan the transport route based on the semantic map; Move according to the transport route.

[0030] The stacker that moves to the target area stops moving and starts the lifting mechanism to carry out the transport work, that is, the binocular camera is used to identify the materials to be transported and the materials to be transported are transported, which specifically includes: The working camera is set at the end of the stacker's lifting mechanism. The optimal relative posture relationship between the working camera and the material to be transported is calculated in advance according to the position of the material to be transported in the semantic map. After reaching the target position, the lifting mechanism is controlled to adjust the posture according to the optimal point-to-posture relationship to transport the materials.

[0031] During the handling process, the target spatial position of the lifting mechanism is determined, and then the relative transformation between the target spatial position of the lifting mechanism and the position of the material being handled is calculated, so that the lifting mechanism is always aligned with the material to be handled.

[0032] Furthermore, when the lifting mechanism is transporting materials, the binocular camera collects image data in real time, identifies the materials in the image data based on the deep learning method, and fine-tunes the lifting mechanism to ensure that the materials are located in the central area of ​​the image.

[0033] This embodiment provides a stacker control method, which obtains a visual point map of the stacker's working area in advance, determines the work location, and obtains an accurate semantic map that can be used as a navigation, so that the stacker can automatically reach the destination to operate. The high-precision integrated positioning system and binocular camera used can not only accurately position the stacker, but also accurately position and transport materials during operation.

[0034] Embodiment 2 The embodiment of the present application also provides a stacker control device corresponding to the stacker control method. Since the principle of solving the problem by the device in the embodiment of the present application is similar to the above-mentioned stacker control method in the embodiment of the present application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0035] See also Figure 2 , Figure 2 This is a schematic diagram of the structure of a stacker control device provided in this embodiment. As shown in the figure, a binocular camera and a working camera are provided on the stacker, the binocular camera includes a left camera and a right camera, a high-precision integrated positioning system is provided in the stacker, the stacker moves and works in a working area, a plurality of road signs are provided in the working area, and the stacker control device 200 includes: Visual point cloud map generation module 201: used for the stacker to travel in advance in the working environment, calculate the position and posture of the binocular camera using a high-precision integrated positioning system, observe the landmarks through the binocular camera, and form a visual point cloud map; Semantic map generation module 202: used to obtain the spatial position distribution of the materials to be transported in the current environment by using a binocular camera when the stacker is working, and project the spatial position distribution onto a visual point cloud map by using spatial position transformation to generate a semantic map containing information of the materials to be transported; Stacker working module 203: used for the stacker to plan the path according to the semantic map, and autonomously reach the target position to work; identify the materials to be transported through the binocular camera, and transport the materials to be transported.

[0036] See also Figure 3 , Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 3 As shown in , the electronic device 300 includes a processor 310 , a memory 320 and a bus 330 .

[0037] The memory 320 stores machine-readable instructions executable by the processor 310. When the electronic device 300 is running, the processor 310 communicates with the memory 320 via the bus 330. When the machine-readable instructions are executed by the processor 310, the above-mentioned Figure 1 The steps of a stacker control method in the method embodiment shown are specifically implemented in accordance with the method embodiment and will not be described in detail here.

[0038] The present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the computer program can execute the above-mentioned Figure 1 The steps of a stacker control method in the method embodiment shown are specifically implemented in accordance with the method embodiment and will not be described in detail here.

[0039] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0040] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0041] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0042] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0043] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application can essentially be embodied in the form of a software product, or in other words, the part that contributes to the prior art or the part of the technical solution. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0044] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by a person of ordinary skill in the art without departing from the spirit and technical concept disclosed by the present invention shall still be covered by the claims of the present invention.

Claims

1. A stacker control method, characterized in that: The stacker is provided with a binocular camera and a working camera, the binocular camera includes a left camera and a right camera, and a high-precision integrated positioning system is provided in the stacker. The stacker moves and works in a working area, and a plurality of landmarks are provided in the working area, including: The stacker travels in advance in the working environment, calculates the position and posture of the binocular camera using the high-precision integrated positioning system, observes the road signs through the binocular camera, and forms a visual point cloud map; When the stacker is working, the binocular camera is used to obtain the spatial position distribution of the materials to be transported in the current environment, and the spatial position distribution is projected onto the visual point cloud map by spatial position transformation to generate a semantic map containing information of the materials to be transported; The stacker performs path planning according to the semantic map and autonomously reaches the target position to work; the materials to be transported are identified by the binocular camera and the materials to be transported are transported.

2. A stacker control method according to claim 1, characterized in that: Calculating the camera's position and posture using the binocular camera and the high-precision integrated positioning system specifically includes: The high-precision integrated positioning system is used to calculate the position and posture of the inertial navigation system in the world coordinate system, calibrate the spatial relationship between the inertial navigation system and the monocular camera, and calculate the position and posture of the monocular camera in the world coordinate system through the mapping relationship among the world coordinate system, the camera coordinate system and the pixel coordinate system. The mapping relationship among the three is as follows: In the formula, the world coordinate system of a certain spatial point is , the corresponding camera coordinate system is , the pixel coordinate system is ; Calculated by dividing the focal length by the pixel length in the respective direction; , The coordinates of the principal point, in pixels, The matrix is ​​called the camera intrinsic parameter matrix, is the rotation matrix from the earth coordinate system to the camera coordinate system, is the translation amount; Perform binocular calibration on the binocular camera and add depth information based on the position and posture of the monocular camera in the world coordinate system to obtain the position and posture of the binocular camera in the world coordinate system: in, is the parallax of the left and right cameras in the binocular camera, is the baseline length of the binocular camera; The optical center of the binocular camera is zero in the camera coordinate system, and the rotation matrix and translation , the position and posture of the binocular camera in the world coordinate system are obtained. The mapping relationship between the optical center of the binocular camera in the camera coordinate system and its position and posture in the world coordinate system is as follows: That is, the position of the binocular camera in the world coordinate system is .

3. A stacker control method according to claim 2, characterized in that: The binocular camera is used to observe the road signs to form a visual point cloud map, and the specific steps include: The binocular camera collects travel data, uses a deep learning method to perform semantic segmentation and target recognition on the travel data, sets the pixel values ​​of the identified staff and work equipment as masks to remove them, and leaves a static scene; The SIFT operator is used to extract feature points of the static scene image, and feature point matching is performed through feature point coordinates and their descriptors. For each of the road signs, after feature matching, a plurality of pixel observation values ​​are obtained, and the pixel observation values ​​are pixel coordinates and disparity of a binocular camera. The observed normal feature points are associated with the corresponding multiple feature descriptors to generate a visual point cloud map.

4. A stacker control method according to claim 3, characterized in that: The feature point matching includes matching of left and right images of the same frame and multi-frame matching of the left image, wherein: For the same-name points in the left and right images of the same frame, if the row coordinates of the left and right same-name points differ by more than 1 pixel, they are not matched; For multi-frame matching of the left image, the points with the same name in adjacent frames need to satisfy the epipolar constraint condition to be matching points, that is, satisfy the following formula: In the formula, is the pixel coordinate, is the basic matrix, through the camera intrinsic matrix And the spatial relationship between two adjacent frames Get, through the position and posture of the binocular camera in the world coordinate system Calculate the pose of adjacent images , The poses of the adjacent images The specific formula is as follows: 。 5. A stacker control method according to claim 4, characterized in that: The stacker moves according to the semantic map and reaches the target position to work, specifically including: Determine the target location of the handling position based on the semantic map; Integrate the transportable information of all current positions to be transported by the stacker, combine the current position of the stacker, and plan the transport route based on the semantic map; The movement is performed according to the transport route.

6. A stacker control method according to claim 5, characterized in that: The binocular camera is used to identify the material to be transported, and the material to be transported is transported, specifically including: The working camera is arranged at the end of the lifting mechanism of the stacker, and the optimal relative posture relationship between the working camera and the material to be transported is calculated in advance according to the position of the material to be transported in the semantic map; After reaching the target position, the lifting mechanism is controlled to adjust the posture according to the optimal point-to-posture relationship to transport the material.

7. A stacker control method according to claim 6, characterized in that: When the lifting mechanism is transporting materials, the binocular camera collects image data in real time, identifies the materials in the image data based on a deep learning method, and fine-tunes the lifting mechanism to ensure that the materials are located in the central area of ​​the image.

8. The control device of the stacker control method according to any one of claims 1 to 7, characterized in that: The stacker is provided with a binocular camera and a working camera, the binocular camera includes a left camera and a right camera, the stacker is provided with a high-precision integrated positioning system, the stacker moves and works in a working area, a plurality of road signs are provided in the working area, and the device includes: Visual point cloud map generation module: used for the stacker to travel in advance in the working environment, calculate the position and posture of the binocular camera using the high-precision integrated positioning system, observe the road signs through the binocular camera, and form a visual point cloud map; Semantic map generation module: used for obtaining the spatial position distribution of the materials to be transported in the current environment by using the binocular camera when the stacker is working, and projecting the spatial position distribution onto the visual point cloud map by using spatial position transformation to generate a semantic map containing information of the materials to be transported; Stacker working module: used for the stacker to plan the path according to the semantic map, and autonomously reach the target position to work; identify the materials to be transported through the binocular camera, and transport the materials to be transported.

9. An electronic device, characterized in that: include: A processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the storage medium communicate via the bus, and the processor executes the machine-readable instructions to perform the steps of the stacker control method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the stacker control method according to any one of claims 1 to 7 are executed.

Images