Method and apparatus for relative positioning of a spreader

Abstract

An apparatus is provided that includes means for receiving a first image of a first feature of a load; receiving a second image of a second feature of the load; determining image plane coordinates of the features of the load based on the first image and the second image; determining one or more action candidates based on the image plane coordinates; evaluating the one or more action candidates using an intermediary embodying historical experience information over a limited time frame; selecting a control action based on the evaluation, wherein the control action causes a spreader to move relative to the load.

Description

Technical Field

[0001] Various example embodiments involve the positioning of the lifting device. Background Technology

[0002] The heavy-duty transportation industry involves handling heavy loads, such as when loading and unloading vehicles at ports and on ships. For example, in container logistics, spreaders are used in crane systems to lift containers. Spreaders are typically controlled by trained operators who require extensive training to become familiar with the spreader control system. Such spreader control systems are susceptible to human error. Summary of the Invention

[0003] The subject matter of the independent claims is provided according to several aspects. Several exemplary embodiments are defined in the dependent claims. The independent claims state the scope of protection sought with respect to the various exemplary embodiments. Exemplary embodiments and features (if any) described in this specification that are not within the scope of the independent claims are to be interpreted as examples useful for understanding the various exemplary embodiments.

[0004] According to a first aspect, an apparatus is provided, comprising means for: receiving a first image of a first feature of a load; receiving a second image of a second feature of the load; determining image plane coordinates of the load features based on the first and second images; determining one or more action candidates based on the image plane coordinates; evaluating the one or more action candidates using an intermediate medium embodying historical experience information over a finite time range; and selecting a control action based on the evaluation, wherein the control action causes a spreader to move relative to the load.

[0005] According to one embodiment, the device includes means for: determining a pairwise operation between image plane coordinates of a first feature and image plane coordinates of a second feature; determining one or more action candidates based on the pairwise operation; and determining a control action based on the cost and / or reward based on the action candidates.

[0006] According to one embodiment, the reward reaches its maximum value when the spreader is substantially aligned with or achieves substantially aligned with the load within a finite future timeframe.

[0007] According to one embodiment, based on action candidates and their effects on the movement of the spreader at the present moment or within a finite future timeframe, the cost is proportional to the force or energy or pressure or voltage or current or placement or placement consumption; and / or reflects the risk of losing features in the camera's field of view at the present moment or within a finite future timeframe.

[0008] According to one embodiment, the device includes means for: sending control actions directly or indirectly to one or more actuators to move the spreader relative to a load.

[0009] According to one embodiment, a first image is received from a first camera positioned on a first corner of the spreader, and a second image is received from a second camera positioned on a second corner of the spreader, wherein the first corner and the second corner are different corners, and wherein a first feature of the load is a first corner of the container, and a second feature of the load is a second corner of the container, wherein the first corner of the spreader and the first corner of the container are corresponding corners, and the second corner of the spreader and the second corner of the container are corresponding corners.

[0010] According to one embodiment, the device includes: at least one processor; and at least one memory including computer program code, wherein the at least one memory and the computer program code are configured together with the at least one processor to cause performance of the device.

[0011] According to a second aspect, a method is provided, the method comprising: receiving a first image of a first feature of a load; receiving a second image of a second feature of the load; determining image plane coordinates of the features of the load based on the first and second images; determining one or more action candidates based on the image plane coordinates; evaluating the one or more action candidates using an intermediate medium that embodies historical experience information within a finite time range; and selecting a control action based on the evaluation, wherein the control action causes the spreader to move relative to the load.

[0012] According to one embodiment, the method includes: determining a pairwise operation between image plane coordinates of a first feature and image plane coordinates of a second feature; determining one or more action candidates based on the pairwise operation; and determining a control action based on the cost and / or reward based on the action candidates.

[0013] According to one embodiment, the reward reaches its maximum value when the spreader is substantially aligned with or achieves substantially aligned with the load within a finite future timeframe.

[0014] According to one embodiment, based on action candidates and their effects on the movement of the spreader at the present moment or within a finite future timeframe, the cost is proportional to the force or energy or pressure or voltage or current or placement or placement consumption; and / or reflects the risk of losing features in the camera's field of view at the present moment or within a finite future timeframe.

[0015] According to one embodiment, the method includes sending control actions directly or indirectly to one or more actuators to move the spreader relative to the load.

[0016] According to one embodiment, a first image is received from a first camera positioned on a first corner of the spreader, and a second image is received from a second camera positioned on a second corner of the spreader, wherein the first corner and the second corner are different corners, and wherein a first feature of the load is a first corner of the container, and a second feature of the load is a second corner of the container, wherein the first corner of the spreader and the first corner of the container are corresponding corners, and the second corner of the spreader and the second corner of the container are corresponding corners.

[0017] According to a third aspect, a computer-readable medium including program instructions, which, when executed by at least one processor, cause a device to at least: receive a first image of a first feature of a load; receive a second image of a second feature of the load; determine image plane coordinates of the features of the load based on the first and second images; determine one or more action candidates based on the image plane coordinates; evaluate the one or more action candidates using an intermediate medium that embodies historical experience information over a finite time range; and select a control action based on the evaluation, wherein the control action causes a spreader to move relative to the load.

[0018] According to other embodiments, the computer-readable medium includes program instructions that, when executed by at least one processor, cause the device to perform at least the method of any embodiment of the second aspect.

[0019] According to another aspect, a computer program is provided, which is configured to cause the execution of a method according to any embodiment of the second aspect. Attached Figure Description

[0020] Figure 1 The spreader alignment task is illustrated by way of example;

[0021] Figure 2 Motion control action visualization is illustrated with examples;

[0022] Figure 3 A flowchart illustrating a method for aligning spreaders is shown as an example.

[0023] Figure 4 The spreader and container, as well as the state design based on the image plane, are illustrated as examples.

[0024] Figure 5 The image plane-based state design is illustrated as an example.

[0025] Figure 6 The system architecture is illustrated with examples;

[0026] Figure 7 A block diagram of the device is shown as an example;

[0027] Figure 8 An example diagram of the error measurement for the alignment test is shown.

[0028] Figure 9 An example diagram of the error measurement for the alignment test is shown.

[0029] Figure 10 An example diagram of the error measurement for the alignment test is shown. Detailed Implementation

[0030] For example, load handling arrangements are used in ports, docks, ships, distribution centers, and various industries. The following examples are described in the context of crane systems, but the methods disclosed herein can be used in any environment where lifting loads and precise positioning of spreaders for handling loads (e.g., containers) is required. Load handling includes, for example, lifting, moving, and placing loads. A crane system can be considered any system or device with spreaders.

[0031] In container logistics, spreaders are used in crane systems to lift containers. Spreaders have torsion locking mechanisms at each corner, which precisely position them to the corner castings of the container. To lift a container, the spreader needs to be aligned with it. For example, the process of lifting a container can be divided into three stages: a search stage, an alignment stage, and a lowering stage.

[0032] During the search phase, the spreader can be moved above the container, reaching the so-called gap area. A rough estimate of the container's position can be received, for example, from the terminal operating system. Moving the spreader above the container can be executed using motion commands.

[0033] During the alignment phase, the position of the spreader can be fine-tuned (e.g., oriented and / or translated) relative to the location of the container or the area available for container placement. This fine-tuning movement can be performed by using motion control commands to operate actuators capable of executing those commands.

[0034] During the descent phase, the spreader can be lowered to the desired position determined during the alignment phase. The container can then be lifted if the spreader's twist locks are engaged and locked to the corner castings of the container.

[0035] A controller is provided for the alignment stage to adjust the position of the spreader so that it lands on the load (e.g., a container) with high precision.

[0036] Figure 1 The spreader alignment task is illustrated as an example. Figure 1The figure above shows a top view of a situation where the spreader 102 is not aligned with the container 104. The x-axis and y-axis are the axes of an imaginary common coordinate system, where spreader 102 and container 104 can be represented.

[0037] The center offset of the spreader 102 and the container 104 in the xy plane is 110. For (d) x ,d y The height between spreader 102 and container 104 is h. Line 112 has been drawn through the center point 122 of spreader 102. Line 114 has been drawn through the center point 124 of container 104. The angle 130 between lines 112 and 114 is γ, which represents the tilt angle between the spreader and the container. During this spreader alignment phase, the strategy generated by the controller aims to... Minimize γ and make And γ = 0.

[0038] Figure 1 The figure below shows a view in the xyz coordinate system, where the spreader is aligned with the container. For example, the center points of the spreader and the container (i.e., points 124 and 122) are aligned, such that... And γ = 0.

[0039] The controller can be represented as:

[0040] a = π(s)

[0041] where a=[a x ,a y ,a skew ] is a three-dimensional vector representing motion control actions (e.g. Figure 2 (As shown), s represents the spreader observation state, and π represents the control strategy. The observation state can be calculated based on the spreader coordinates and container coordinates represented in a common coordinate system or an image coordinate system. When obtaining coordinates from sensors, the state based on the common coordinate system is used, i.e., the coordinates of the container and spreader are accurately measured in the common coordinate system. The state design based on the image coordinate system is described below.

[0042] Figure 2 Motion control action visualization is illustrated with an example. Action a x This indicates the movement of the lifting device along the x-axis; action a y This indicates the movement of the lifting device along the y-axis, and the action a. skew This indicates the rotational movement of the lifting device in the xy plane.

[0043] A method for aligning spreader is provided. This method allows for the selection of motion control actions, enabling high-precision spreader alignment.

[0044] Figure 3 A flowchart illustrating a method for aligning a spreader is shown as an example. The method can be, for example, through... Figure 7 The method is performed using a device. Method 300 includes: receiving 310 a first image of a first feature of the load. Method 300 includes: receiving 320 a second image of a second feature of the load. Method 300 includes: determining 330 image plane coordinates of the features of the load based on the first and second images. Method 300 includes: determining 340 one or more action candidates based on the image plane coordinates. Method 300 includes: evaluating 350 one or more action candidates using an intermediate medium that embodies historical experience information within a finite time range. Method 300 includes: selecting 360 a control action based on the evaluation, wherein the control action causes the spreader to move relative to the load.

[0045] The method disclosed herein provides for determining control actions based on image information from spreader sensors (e.g., based on spreader camera streams). Additional sensor information is not necessarily required. However, various sensor data can be used, for example, to create images, as described below. The method disclosed herein enables precise positioning of the spreader to lift loads, such as containers, without manual intervention. The method is robust to changes in the cameras and their alignment with each other, and to singularities in the aligned camera views. Furthermore, the method relies on multi-point evaluation of the images, which can significantly increase sensitivity to measurement noise and accuracy of prior information compared to, for example, single-point evaluation. The method is time-independent and based on system geometry. Time-independent geometric operations make the system well-suited for variable delay control. This is advantageous compared to, for example, pure trajectory control, which enforces highly synchronized actuator control and is time-critical.

[0046] Figure 4 The spreader 102, container 104, and image-plane-based state design are illustrated by way of example. Cameras 400, 401, 402, and 403 are attached to the spreader. For example, the cameras can be positioned at the corners of the spreader, such as one camera at each of the four corners. However, the camera positions do not need to be precisely at the corners, but rather approximately at them. The number of cameras and the choice of their viewpoints can depend on the field of view and the visibility of the container. The camera positions and orientations can be such that they point downwards to capture the space below the spreader. The camera positions are known in a common coordinate system.

[0047] For example, two cameras (e.g., a first camera and a second camera) can be attached to the rigging. If two cameras are used, the cameras can be wide-angle cameras. The first camera and the second camera are attached to different corners of the rigging. For example, the first camera can be positioned on a first corner of the rigging, and the second camera can be positioned on a second corner of the rigging. The first corner and the second corner are different corners. The first corner can be opposite the second corner, such that the first corner and the second corner can be connected by a diagonal line passing through the center point of the rigging. Alternatively, the first corner and the second corner can be adjacent corners.

[0048] As another example, a bird's-eye view camera can be used.

[0049] The camera may be a video camera. The camera includes a digital image sensor, such as a charge-coupled device (CCD) and / or an active pixel sensor, such as a complementary metal-oxide-semiconductor (CMOS) sensor. Images are received from one or more cameras. For example, a first image may be received from a first camera, and a second image may be received from a second camera. Alternatively, images may be received from three or four cameras or from a bird's-eye view camera. In the case of a bird's-eye view camera, for example, the first and second images are cropped from a wider image. The first image includes or shows an image of a first feature of the container. The first feature may be a first corner of the container. Alternatively, the first feature may be a twist lock hole, a mark, a marker, or any feature that can be detected in the image and can be associated with the first corner of the container. In some cases, a feature that can be geometrically defined as a rectangle may be an alternative to a corner. The first corner of the container corresponds to the first corner of the spreader. This correspondence means, for example, that camera 400 attempts to capture the corner 410 of the container or some other feature. For example, corner 410 corresponds to the corner where camera 400 is located; corner 411 corresponds to the corner where camera 401 is located; corner 412 corresponds to the corner where camera 402 is located; and corner 413 corresponds to the corner where camera 403 is located.

[0050] Instead of receiving images from a camera, the images can be received from a memory (where the images are already stored). In some cases, images including features (e.g., corners) can be created based on range sensor data or distance sensor data. For example, a time-of-flight camera or lidar can be used for feature detection, such as corner detection.

[0051] The second image includes or shows an image of a second feature of the container. The second feature may be a second corner of the container. Alternatively, the second feature may be a toggle lock hole, a mark, a marker, or any feature that can be detected from the image and can be associated with the second corner of the container. The second corner of the container corresponds to the second corner of the spreader. Features, such as corners, can be detected from the image via image processing techniques for object detection. The corner detection function is denoted as F. For example, edge detection methods (e.g., edge approximation (EA) detection methods) and hue, saturation, value (HSV) algorithms can be used to detect corners. The HSV algorithm can filter and segment containers based on color, and the EA method can calculate the rotation angle of the container. Neural networks (NNs) provide a robust approach to object detection. For example, deep learning can be used for feature extraction, such as corner casting detection. Images received from a camera stream can be fed into a neural network to detect features, such as the corners of the container. The NN may consist of, for example, two modules: a convolutional neural network (CNN) part and a long short-term memory (LSTM) module. The received images can be combined and fed into a CNN to extract high-level features, while an LSTM can repeatedly predict the corners of the container.

[0052] Image plane coordinates of container features can be determined based on received images. For example, the image plane coordinates of a container corner can be determined based on a first image and a second image. The image plane-based state is based on the container corner projected onto the image plane from a common coordinate system. By determining or measuring the feature positions in the image plane, measurement errors associated with physical coordinate measurements of sensors are avoided. Using physical coordinate measurements makes the system sensitive to any changes in configuration and requires very precise knowledge of system dimensions. Furthermore, camera calibration is not required, as in model-based methods where small errors in the estimation of the camera's external and / or internal parameters can ultimately lead to large physical estimation errors proportional to the camera's focal length and the size of the spreader. Errors accumulate when multiple cameras are used in model-based methods. As disclosed herein, directly finding the mapping from image coordinates (rather than physical coordinates) to the target pose avoids the need for camera calibration.

[0053] In this example, let's consider four image planes 450, 451, 452, and 453. There can be four cameras 400, 401, 402, and 403 positioned at the corners of the spreader. Cameras 400, 401, 402, and 403 can be represented as cam0, cam1, cam2, and cam3, respectively. Image 450 can be received from camera 400, image 451 from camera 401, image 452 from camera 402, and image 453 from camera 403. Let's represent the four corners of the container 410, 411, 412, and 413 as points p in a common coordinate system. c0 p c1 p c2 and p c3 Let us denote the corners (or other features) 460, 461, 462, and 463 of the containers on the projected camera image planes 450, 451, 452, and 453 as points p0, p1, p2, and p3, respectively.

[0054] Let us introduce a homogeneous 4-vector (x) in relative coordinates. offsetc ,y offset ,z offset ,1) The symbol X represents the world point j Let's represent the camera's position in the common coordinate system as... And the position of the corner of the container in the common coordinate system is represented as but:

[0055]

[0056] For each corner p on each image plane j,j＝0,1,2,3 Its projection is based on the projection equation:

[0057] p j =PX j T

[0058] Where p is the projection matrix:

[0059] P = K[R|T]

[0060] R and T are the camera's extrinsic parameters, which relate the orientation and position of image frames to a common coordinate system. K is the intrinsic parameter matrix of the finite projection camera.

[0061]

[0062] If the number of pixels per unit distance in the image coordinates is m in both the x and y directions x and m y And if the focal length is expressed as f, then it is applied to

[0063] α x =f·m x ,α y =f·m y

[0064] Where, α x and α y The camera's focal length is represented by pixel dimensions in the x and y directions, respectively. The parameter 's' is called the tilt parameter. For most ordinary cameras, the tilt parameter will be zero.

[0065] Figure 5 The image plane-based state design is illustrated as an example. The state can be defined on a new Cartesian coordinate system composed of four image planes. The vector 560 from p0 460 to the origin 0 is... The vector 561 from p1 461 to the origin is The vector 562 from p2 462 to the origin is The vector 563 from p3 463 to the origin is

[0066] The angle between vectors 560 and 562 can be defined as

[0067]

[0068] The angle between vectors 561 and 563 can be defined as

[0069]

[0070] In addition, θ′=π-θ, θ′∈[-π,π] and α′=π-α, α′∈[-π,π] can be defined.

[0071] The state can be defined by four vectors and two angles between them. The state can be defined as follows:

[0072]

[0073] In the case of two images, the state can be defined by two vectors and the angle between them. As another example, the state can be the image itself.

[0074] Another option is to use symmetry features to match the position between the spreader and the container. In other words, a pairwise operation can be determined between the image plane coordinates of the first corner (or feature) and the image plane coordinates of the second corner (feature). The state S can be defined based on the image plane symmetry coordinates. IPS .

[0075] In pairwise operations, images are compared with images, or image features are compared with image features, without mapping them to physical distances (e.g., metric distances). For example, this minimizes the impact of calibration errors, camera misalignment, and tilted containers or floors on crane operation. The effect of camera optical calibration or internal camera parameters is minimized as long as the cameras are nearly similar to each other. When using pairwise operations, decisions are made based on the current view and what will happen to the compared pair in the near future. Therefore, the system updates its recent expectations based on the differences in the camera views, rather than relying on past points and their positions in the physical system. The features being compared can be planar, simple features, such as mathematical points at the corners of containers, without requiring size sensing or previous template views of the objects. This simplifies image processing because complex object recognition and / or object pose estimation are not required.

[0076] A pairwise operator is an operator with monotonic or piecewise monotonic behavior that is related to the alignment error of the spreader and container or the decrease or increase of the available position of the container. An example of a pairwise operator is the norm of the dot product or cross product of the error vectors in any pair of camera images. Another example of a pairwise operator is the norm of the dot product or cross product of the feature position vectors in any pair of camera images. In other words, for any selected pair of camera images, the system compares a feature on the camera image either with its error or with its position.

[0077] Pairwise operations (e.g., pairwise symmetric operations) are beneficial for learning algorithms such as reinforcement learning (RL) algorithms. By defining the symmetric or pairwise properties of an artificial intelligence (AI) controller, it can learn available control strategies without any fundamental knowledge of truth from uncalibrated sensors or cameras or from precise physical coordinate measurements or human expert guidance. A spreader has a rectangular geometry similar to a shipping container. Therefore, changes in the view become comparable to each other or from one camera to another when there is no offset or minimal offset in x, y, and tilt. Pairwise operations can be generalized to rectangular geometries that exhibit any symmetric visual properties.

[0078] f = Flip(p) represents a symmetric operation, for example, Flip tl→br (p0) means the flip point p0 460 from the top left (tl) to the bottom right (br). The top left (tl) refers to image 450, the top right (tr) refers to image 451, the bottom right (br) refers to image 452, and the bottom left (bl) refers to image 453.

[0079]

[0080]

[0081]

[0082]

[0083]

[0084]

[0085] Symmetry features can be used to match the position between the spreader and the container. When the spreader and container are aligned, the coordinates of the corners on the image plane have the following characteristics:

[0086]

[0087]

[0088]

[0089]

[0090]

[0091]

[0092] Make

[0093] This refers to the target offset when the spreader and container are aligned. The target offset is defined based on the camera's pose and the image plane. symmetric It can be non-zero.

[0094] The state in this case can be defined as:

[0095]

[0096] Action candidates (e.g., motion control action candidates) can be determined. This determination can be based on determined image plane coordinates of features of the containers in the images, such as the determined image plane coordinates of the corners of the containers in a first image and the corners of the containers in a second image relative to each other. Alternatively or further, the determination can be based on historical information derived from the images.

[0097] Action candidates determine different control commands for moving the spreader. As mentioned above, for a controller, control commands can be represented as vectors defining movement and rotation (i.e., skew) in the x and y directions. Actions or control commands can be determined, for example, via energy, force, power, voltage, current, and / or displacement. For example, the system requires energy to move the spreader, and energy can be transferred in the system, for example, via pressure changes, electricity, etc. Actions can be, for example, discrete or continuous. Reinforcement learning (RL) algorithms can be used to learn spreader alignment tasks. Reinforcement learning (RL) is a machine learning technique that enables an agent to learn in an interactive environment using feedback from its own actions and experiences. In RL, in a specific state of the environment, the agent or optimization algorithm performs actions according to its policy (e.g., a neural network) that changes the state of the environment and receives the new state and a reward for that action. The agent's policy is then updated based on the reward for the state-action pair. RL learns through self-exploration, which can or may not be performed without human intervention.

[0098] For discrete actions, there exists a set of action candidates, where the actions have fixed values. For example, an action candidate can be defined as a = [a...]. x ∈{-1,1},a y ∈{-1,1},a skew [∈{-1,1}], where -1 (negative) and 1 (positive) refer to different directions. For example, -1 can point to a displacement along the negative x-axis or y-axis or a counterclockwise rotation, and +1 can point to a displacement along the positive x-axis or y-axis or a clockwise rotation. A policy can be learned in this case to determine the probability of what action should be taken based on the current state. This action can be given as π(a|s), where a is the action, s is the state, and π is the policy. In at least some embodiments, the action candidate is independent of the sampling time, which is beneficial for systems with variable delays. The sampling time, or cycle time, is the rate at which a discrete system samples its input or state.

[0099] The outcome of a strategy indicates the likelihood of different actions, for example [a x_positive =0.5,a y_positive =0.35,a skew_positive =0.8,a x_negative =0.35,a y_negative =0.35,a skew_negative =0.18], and can choose the action a with the highest probability. skew_positive RL algorithms for discrete actions can be, for example, deep Q-learning networks (DQN).

[0100] For continuous actions, there exists a set of action candidates, where the values ​​of the actions are not fixed. For example, an action candidate can be defined as a = [a...].x ∈[-1,1],a y ∈[-1,1],a skew ∈[-1,1]]. In this case, the policy can be a deterministic policy, which is learned to give a specific value for each action. The action can be given as a = π(s).

[0101] The outcome of the strategy can be, for example, [a] x =-0.3,a y =0.8,a skew =1], and all actions can be performed in one step. RL algorithms for continuous actions can be, for example, Deep Deterministic Policy Gradient (DDPG).

[0102] Action candidates can be evaluated using an intermediate medium that reflects historical experience within a finite time frame. For example, RL algorithms can be used to evaluate action candidates. In RL, the goal of the RL agent is to maximize the cumulative reward. In the context of scenarios, this reward can be represented as the sum of all reward signals received during a scenario. The term scenario refers to a series of actions used to position a spreader above a container for lifting.

[0103] The reward can be defined based on mathematical operations, such as image plane coordinates or image plane symmetric coordinates.

[0104] When coordinates can be obtained from sensors, that is, when the coordinates of containers and spreaders in a common coordinate system are accurately measured, a state based on common coordinates can be used.

[0105] The environment can respond by transitioning to another state and generating a reward signal. The reward signal can be considered a fundamental estimate of the agent's performance. The reward signal can be calculated based on a reward function, which can be stochastic and dependent on action a:

[0106] reward(r t |s t ) is the reward function, which is based on the current state s at time instant t. t Calculate instantaneous reward r t .

[0107] This process is repeated continuously, with the agent making action choices based on observations and the environment in response to the next state and reward signal. The agent's goal is to maximize the cumulative reward R.

[0108] The instantaneous reward r of a trajectory t The sum of .

[0109] Reward functions can be designed to guide RL agents in optimizing their policies. For example, a reward function based on a common coordinate system can be defined as follows:

[0110]

[0111] As shown in the reward function, a reward is given when the spreader reaches the target position. common Increase. At the target position, it maintains d. x =0, d y =0 and γ=0. (See) Figure 1 The reward reaches its maximum value when the spreader is substantially aligned with the load (e.g., when the spreader is perfectly aligned with the load, such as a container). Alternatively, the reward reaches its maximum value when the spreader achieves substantial alignment within a finite future timeframe. This may mean that the alignment is within a predefined threshold. The threshold can be predetermined for either substantial or perfect alignment.

[0112] The reward based on image coordinates is defined based on the symmetry of the corners or other predetermined features of the received image. The reward can be the L2 norm of the state.

[0113] reward symmetric =-1*||state symmetric -target symmetric ||2

[0114] If the reward exceeds the predetermined range, the task is successful.

[0115] The primary objective is to maximize the reward. Where possible, this can occur alongside minimizing the cost. Based on action candidates and their impact on the spreader's movement at the present moment or within a finite future timeframe, the cost can be proportional to force, energy, pressure, voltage, current, or placement / positioning consumption. The cost can also reflect the risk of losing features within the camera's field of view at the present moment or within a finite future timeframe.

[0116] You can select action candidates that will lead to mission success as control actions. This control action moves the spreader relative to the container.

[0117] Deep Deterministic Policy Gradient (DDPG) is a non-policy model-free RL algorithm for continuous control. DDPG's actor-critic structure allows it to leverage the advantages of policy gradient methods (actors) and d-value approximation methods (critics). For a trajectory, the state s and action a at time step t are represented as s t and a t Action-value function approximator (i.e., critic Q(s)) t ,a t )) indicates based on s t Action at The expected cumulative return afterwards. Q(s) t ,a t ) Optimization is performed by minimizing the Bellman error, so that

[0118]

[0119] in, It is the expected value of its argument. The action strategy part (the actor) is the function a. t =π(s) t ), and optimizes it by directly maximizing the estimated action-value function of the actor relative to the policy parameters. Specifically, DDPG maintains a parameter θ π The actor function π(s) has the parameter θ Q The critical function Q(s,a) and the experience buffer B are given as a set of tuples t. i =(s t ,a t ,r,s t+1 This tuple is used for each transition after an action. It is time-independent.

[0120] DDPG alternates between running a policy to collect trajectories and updating parameters. During the policy running phase, DDPG executes actions generated by the current policy with added noise, e.g., a = π(s) + noise, and stores the RL transformations in an experience buffer B. After storing the sampled trajectories, during the training phase of non-policy, model-free RL, mini-batches of N tuples are randomly sampled from the experience buffer B to update the actor and critic networks by minimizing the following loss:

[0121] Among them, target y i It is the expected future cumulative return from step i:

[0122]

[0123] As shown in this equation, the target term y i It also depends on the parameters φ and θ. This can make training unstable. To address this issue, a target network is introduced. and Use and and π θ The target network is initialized with the same parameters. During training, and θ target Polyak performs a soft update on average once per main network update:

[0124]

[0125] θ target ←τθ+(1-τ)θ target

[0126] -Where, τ < 1.

[0127] - Current term y i become:

[0128]

[0129] For an actor network, the goal is to learn a policy with parameters θ, and the solution is:

[0130] in, It is the expected value of its argument.

[0131] For a batch sampling of N transformations, the policy gradient can be calculated as:

[0132]

[0133] Non-policy reinforcement learning is more sample-efficient than policy reinforcement learning because it can repeatedly optimize the policy from historical trajectories. However, it will fail when the policy is initialized incorrectly. In this case, RL needs to try and collect a large number of samples to approximate the correct Q-function, and therefore, sample efficiency may still be an issue.

[0134] To further improve model-free RL from a practical perspective, the policy network can be trained first using expert demonstrations. One reason for the sample efficiency problem is that most generated trajectories are failure cases at the beginning of the training phase. Expert demonstrations can also be stored in an experience buffer. During training, in addition to the policy gradient loss, an auxiliary supervised learning loss can also be computed as the behavior cloning (BC) loss:

[0135] Where a demo It's a demonstration action, N D This indicates a transformation based on samples taken from human demonstration trajectories.

[0136] To prevent the policy from falling into suboptimal solutions when learning from demonstrations, a q-filter can be applied. However, criticized by critics of networks, the behavior cloning loss is only applied when the demonstration action performs better. The final behavior cloning loss can be expressed as...

[0137]

[0138] The gradients applied to the policy network are as follows:

[0139] Here, λ1 and λ2 are hyperparameters that define the weights of each loss, λ1+λ2=1, λ1>0, λ2>0.

[0140] This expert demonstration shortens the exploration phase of RL.

[0141] The goal of the training phase of an RL model is to optimize the policy function so that it can complete the alignment task.

[0142] In this training example, the following parameters are given: 1) with parameters 1) The critique function Q; 2) The policy function π with parameter θ; 3) The policy function π with parameter θ. Target critic function Q target ;4) with parameter θ target The policy function π target ;5) Experience playback buffer B;6) Corner detection function F;7) Image set I captured by the camera =<im1,im2,im3,im4> During the training phase, multiple trial training tracks may be required:

[0143] At the start of the trial training trajectory, the position of the spreader is random: the height distance between the spreader and the container is random, ranging from 1 to 3 meters. (xy displacement d) x and d y The values ​​are random between -25cm and 25cm. The angular displacement γ (skew) is random between -5 degrees and 5 degrees.

[0144] During the training phase, for each step in the trajectory:

[0145] - Detect corners:<p1,p2,p3,p4> =F(I)

[0146] - Calculate the symmetry coordinates of the image plane

[0147] - Example action a using the policy function π t :

[0148] a t =π(s) t )

[0149] - Perform action a t .

[0150] During training, action 'a' can be determined, for example, based on event-based control or real-time control. In event-based control, 'a' indicates the displacement of the xy-axis movement and rotation angle (e.g., 'a = [0.2, -0.2, 1] means: move the spreader 20 cm to the right, move it 20 cm down, and rotate it 1 degree clockwise). In real-time control, 'a' indicates the direction of the xy-axis movement and rotation, as well as the corresponding duration (e.g., 'a = [-10, 0, 20] means: move the spreader to the left for 1 second and rotate it clockwise for 2 seconds).

[0151] - Wait for a certain time interval or until action a is completed and the next state is obtained.

[0152] - Calculate the reward r based on image symmetry.

[0153] - Transform the tuple t = t ,a t ,r t ,s t+1 Stored in experience buffer B.

[0154] At the end of each step:

[0155] -N transformations t from B i = i ,a i ,r i ,S i+1 > Random sample small batch experience.

[0156] -Optimize Q by minimizing the loss:

[0157]

[0158] - Calculate the policy gradient

[0159]

[0160] If we transform tuple t from the demonstration i After sampling, the actor policy is updated using gradients:

[0161]

[0162] - Otherwise, update the actor network using only one-step gradient ascent in the following steps:

[0163] - At the end of each step, use a soft update to update the parameters of the target network:

[0164]

[0165] ​​

[0166] During the testing phase, for each step in the trajectory:

[0167] - Detect corners:<p1,p2,p3,p4> =F(I)

[0168] - Calculate the symmetry coordinates of the image plane

[0169] - Use the policy function π to sample or evaluate action a:

[0170]

[0171] - Perform action a.

[0172] - Action 'a' can be determined, for example, based on event-based control or real-time control. In event-based control, 'a' indicates the displacement of the xy-axis movement and rotation angle (e.g., 'a = [0.2, -0.2, 1] means: move the spreader 20 cm to the right, move it 20 cm down, and rotate it 1 degree clockwise). In real-time control, 'a' indicates the direction of the xy-axis movement and rotation, as well as the corresponding duration (e.g., 'a = [-10, 0, 20] means: move the spreader to the left for 1 second and rotate it clockwise for 2 seconds).

[0173] - Wait for a certain time interval or until action a is completed and the next state is obtained.

[0174] - Repeat this step.

[0175] Figure 6 System architecture 600 is illustrated by way of example. The system architecture includes a real-time processing unit 602 and an artificial intelligence (AI) processing unit 604. The AI ​​processing unit may be a processor with parallel processing capabilities that performs logical reasoning (RL) and detects features, such as the corner of a container. The role of the real-time processor 602 is to maintain real-time communication by receiving time-based signals and transmitting data for time-critical processing needs. This processing unit may be, for example, a reduced instruction set computer (RISC) processing unit. The communication channel between real-time hardware components (e.g., sensor 606 and actuator 608) and the onboard processing unit 630 may be, for example, a real-time fieldbus 610. Alternatively, the low-latency AI processing unit 604 may include the real-time processing unit 602 for receiving time-based signals and transmitting data for time-critical processing needs. Sensors may include, for example, range sensors, distance sensors, lidar, etc.

[0176] AI processing unit 604 is the processing unit that runs RL algorithms. Running RL algorithms does not necessarily need to be performed in hard real-time, but rather as an online module with a sufficiently fast response. The AI ​​processing unit receives input from two or more cameras (e.g., from four cameras 620, 633, 624, 626 connected to a multicast camera network 616) via a communication interface 614 through a communication channel (e.g., a local area network 612). Computation is performed on a processing unit capable of providing fast response based on its memory hardware resources, the processing power of a CPU, or parallel processing capabilities (e.g., in a graphics processing unit (GPU)). Here, "sufficiently fast" means that the RL results should be ready at a frequency higher than the inherent frequency of the crane machinery. For example, the results should be ready, for example, at least two to ten times higher than the inherent frequency of the crane machinery. Therefore, the specific processing power specifications can depend on the requirements of the system mechanism and the availability of the processor. Depending on the limitations of the application environment, the processing unit can be placed in the electrical room of the mobile platform or in the camera unit. Figure 6 As shown, the AI ​​processing unit 604 can access the platform's runtime information through the real-time processing unit 602. The AI ​​processing unit and the real-time processing unit can be physically housed in the same enclosure and / or partially share resources.

[0177] Figure 7 A block diagram of an apparatus 700 capable of performing the methods disclosed herein is shown by way of example. An apparatus configured to perform the methods disclosed herein includes means for performing the methods. The means includes: at least one processor 710; and at least one memory 720 including computer program code, the at least one memory and the computer program code being configured, together with the at least one processor, to cause performance of the apparatus. The memory 720 may include computer instructions, and the processor 710 is configured to execute the computer instructions. The memory 720 may be at least partially included in the processor 710. The memory 720 may be at least partially external to the apparatus 700, but accessible by the apparatus 700.

[0178] The device may be the AI ​​processing unit 604 or another device connected to the AI ​​processing unit. The device can directly or indirectly send action commands to the actuator 608, such as control action commands, to move the spreader according to the commands. The user interface (UI) 730 may be, for example, the onboard processing unit 630. The UI may include, for example, a display, keyboard, touchscreen, and / or mouse. Users can operate the device via the UI.

[0179] The device may include a communication device 740. The communication device may include, for example, a transmitter and a receiver, configured to transmit and receive information via wired or wireless communication, respectively.

[0180] Figure 8 , Figure 9 and Figure 10 Figures 800, 900, and 1000 illustrate the error measurements of alignment tests. The x-axis of each figure, 802, 902, and 1002, represents the number of actions the controller has created that caused the spreader to move. The different lines in the figures represent different alignment tests of the system or equipment used to position the spreader.

[0181] Figure 8 The y-axis 804 represents the x-offset in normalized distance units. The x-offset indicates (for example, the difference in normalized distance units in the x-direction on the xy-plane measured between the center point of the spreader and the container). (See...) Figure 1 (Offset 110)

[0182] Figure 9 The y-axis 904 represents the y-offset in normalized distance units. The y-offset indicates, for example, the difference in normalized distance units in the y-direction on the xy-plane measured between the center point of the spreader and the container. (See...) Figure 1 (Offset 110)

[0183] Figure 10 The y-axis at 1004 represents the skew-offset in normalized distance units. The skew-offset indicates the tilt angle between the spreader and the container. (See...) Figure 1 (Angle 130)

[0184] During the spreader alignment phase, the strategy generated by the controller aims to minimize the x-offset, y-offset, and tilt angle, making them equal to zero.

[0185] Figure 8 , Figure 9 and Figure 10 This demonstrates that the method disclosed herein enables accurate positioning of the spreader relative to the container, within acceptable tolerances of the mechanical system. Figure 8 and Figure 9 In the example, the error was reduced to about 1 / 4 of the maximum offset. Figure 10 In the example, the error was reduced to about 1 / 3 of the maximum offset.

Claims

1. A device for relative positioning of a lifting device, comprising means for: The first image of the first characteristic of the received load; Receive a second image of the second feature of the load; Determine the image plane coordinates of the features of the load based on the first image and the second image; Determine the pairwise operations between the image plane coordinates of the first feature and the image plane coordinates of the second feature, wherein the pairwise operators of the pairwise operations have monotonic or piecewise monotonic behavior; One or more action candidates are determined based on the pairwise operations; The one or more action candidates are evaluated using an intermediate medium that reflects historical experience information within a limited time frame to determine the cost and / or reward of the one or more action candidates; and A control action is determined based on the cost and / or reward of the action candidates, wherein the control action causes the spreader to move relative to the load.

2. The device as claimed in claim 1, wherein, A self-exploration algorithm is used when evaluating the one or more action candidates.

3. The device as described in claim 1 or 2, wherein, The one or more action candidates are independent of the sampling time.

4. The device as claimed in claim 1 or 2, wherein, The one or more action candidates and control actions are defined based on displacement in the x-direction, displacement in the y-direction, and rotation.

5. The device as claimed in claim 1 or 2, wherein, The paired operators have piecewise monotonic behavior related to the reduction or increase of errors in the alignment of the spreader and the load; and Wherein, the pairwise operation is a pairwise symmetric operation; or The pairwise operator is the norm of the dot product or cross product of the error vectors in the first image and the second image; or The pairwise operator is the norm of the dot product or cross product of the feature location vectors in the first image and the second image.

6. The device as claimed in claim 1 or 2, wherein, The reward reaches its maximum value when the spreader is substantially aligned with or achieves substantially aligned with the load within a finite future timeframe.

7. The device as claimed in claim 1 or 2, wherein, The cost is proportional to the force, energy, pressure, voltage, current, placement, or placement consumption of the action candidate and its effect on the movement of the rigging at the present moment or within a finite future timeframe; and / or reflects the risk of losing features in the camera's field of view at the present moment or within a finite future timeframe.

8. The apparatus of claim 1 or 2, further comprising means for... The control action is sent to one or more actuators to move the spreader relative to the load.

9. The device as claimed in claim 1 or 2, wherein, The first image is received from a first camera positioned at a first corner of the spreader, and the second image is received from a second camera positioned at a second corner of the spreader, wherein the first corner and the second corner are different corners, and wherein a first feature of the load is a first corner of the container, and a second feature of the load is a second corner of the container, wherein the first corner of the spreader and the first corner of the container are corresponding corners, and the second corner of the spreader and the second corner of the container are corresponding corners.

10. The apparatus of claim 9, further comprising means for: Receive a third image of the third feature of the load, wherein, The third image is received from a third camera positioned on the triangular portion of the lifting device; A fourth image of a fourth feature of the load is received, wherein the fourth image is received from a fourth camera positioned at the fourth corner of the spreader; wherein the third corner and the fourth corner are corners different from the first corner and the second corner; and wherein the third feature of the load is the third triangular portion of the container, and the fourth feature of the load is the fourth corner of the container, wherein the third triangular portion of the spreader and the third triangular portion of the container are corresponding corners, and the fourth corner of the spreader and the fourth corner of the container are corresponding corners; and The device also includes means for... The image plane coordinates of the third and fourth features of the load are determined based on the third and fourth images; Determine another pairwise operation between the image plane coordinates of the third feature and the image plane coordinates of the fourth feature, wherein the pairwise operators of the pairwise operation have monotonic or piecewise monotonic behavior; One or more action candidates are determined based on the pairwise operations.

11. The device as claimed in claim 1 or 2, wherein, The device includes: at least one processor; and at least one memory including computer program code, the at least one memory and the computer program code being configured together with the at least one processor to cause performance of the device.

12. A method for relative positioning of a lifting device, comprising: The first image of the first characteristic of the received load; Receive a second image of the second feature of the load; Determine the image plane coordinates of the features of the load based on the first image and the second image; Determine the pairwise operations between the image plane coordinates of the first feature and the image plane coordinates of the second feature, wherein the pairwise operators of the pairwise operations have monotonic or piecewise monotonic behavior; One or more action candidates are determined based on the pairwise operations; The one or more action candidates are evaluated using an intermediate medium that reflects historical experience information within a limited time frame to determine the cost and / or reward of the one or more action candidates; and A control action is determined based on the cost and / or reward of the action candidates, wherein the control action causes the spreader to move relative to the load.

13. The method of claim 12, wherein, A self-exploration algorithm is used when evaluating the one or more action candidates.

14. The method of claim 12 or 13, wherein, The one or more action candidates are independent of the sampling time.

15. The method of claim 12 or 13, wherein, The one or more action candidates and control actions are defined based on displacement in the x-direction, displacement in the y-direction, and rotation.

16. The method of claim 12 or 13, wherein, The paired operators have piecewise monotonic behavior related to the reduction or increase of errors in the alignment of the spreader and the load; and Wherein, the pairwise operation is a pairwise symmetric operation; or The pairwise operator is the norm of the dot product or cross product of the error vectors in the first image and the second image; or The pairwise operator is the norm of the dot product or cross product of the feature location vectors in the first image and the second image.

17. The method of claim 12 or 13, wherein, The reward reaches its maximum value when the spreader is substantially aligned with or achieves substantially aligned with the load within a finite future timeframe.

18. The method of claim 12 or 13, wherein, The cost is proportional to the force, energy, pressure, voltage, current, placement, or placement consumption of the action candidate and its effect on the movement of the rigging at the present moment or within a finite future timeframe; and / or reflects the risk of losing features in the camera's field of view at the present moment or within a finite future timeframe.

19. The method of claim 12 or 13, further comprising: The control action is sent directly or indirectly to one or more actuators to move the spreader relative to the load.

20. The method of claim 12 or 13, wherein, The first image is received from a first camera positioned at a first corner of the spreader, and the second image is received from a second camera positioned at a second corner of the spreader, wherein the first corner and the second corner are different corners, and wherein a first feature of the load is a first corner of the container, and a second feature of the load is a second corner of the container, wherein the first corner of the spreader and the first corner of the container are corresponding corners, and the second corner of the spreader and the second corner of the container are corresponding corners.

21. The method of claim 20, further comprising: Receive a third image of the third feature of the load, wherein, The third image is received from a third camera positioned on the triangular portion of the lifting device; A fourth image of a fourth feature of the load is received, wherein the fourth image is received from a fourth camera positioned at the fourth corner of the spreader; wherein the third corner and the fourth corner are corners different from the first corner and the second corner; and wherein the third feature of the load is the third triangular portion of the container, and the fourth feature of the load is the fourth corner of the container, wherein the third triangular portion of the spreader and the third triangular portion of the container are corresponding corners, and the fourth corner of the spreader and the fourth corner of the container are corresponding corners; and The equipment for relative positioning of the spreading device also includes a device for... The image plane coordinates of the third and fourth features of the load are determined based on the third and fourth images; Determine another pairwise operation between the image plane coordinates of the third feature and the image plane coordinates of the fourth feature, wherein the pairwise operators of the pairwise operation have monotonic or piecewise monotonic behavior; One or more action candidates are determined based on the pairwise operations.

22. A computer-readable medium comprising program instructions that, when executed by at least one processor, cause a device to perform at least the following: The first image of the first characteristic of the received load; Receive a second image of the second feature of the load; Determine the image plane coordinates of the features of the load based on the first image and the second image; Determine the pairwise operations between the image plane coordinates of the first feature and the image plane coordinates of the second feature, wherein the pairwise operators of the pairwise operations have monotonic or piecewise monotonic behavior; One or more action candidates are determined based on the pairwise operations; The one or more action candidates are evaluated using an intermediate medium that reflects historical experience information within a limited time frame to obtain the cost and / or reward of the one or more action candidates. A control action is determined based on the cost and / or reward of the action candidates, wherein the control action causes the spreader to move relative to the load.

23. The computer-readable medium of claim 22, comprising program instructions that, when executed by at least one processor, cause the device to perform at least the method of any one of claims 13 to 21.

Images