Vehicle control method, apparatus and vehicle based on detection of port vertical transport target

Abstract

The present application provides a vehicle control method, apparatus and vehicle based on detection of a port vertical transport target. The method is applied to a vehicle. First point cloud data of a region above the vehicle is acquired, then target position information and target grabbing state of a spreader in a lifting device are determined from a pre-determined point cloud mapping relationship according to the first point cloud data, where the point cloud mapping relationship records point cloud data corresponding to different position information and different grabbing states of the spreader, and then a moving and stopping state of the vehicle is controlled according to the target position information and the target grabbing state. In this technical solution, the point cloud data of the region above is acquired by the vehicle, and then compared with the point cloud data corresponding to pre-determined possible spreader positions and grabbing states, to determine the current spreader position and grabbing state of a region above the vehicle, thereby controlling the starting and stopping of the vehicle, to avoid dangerous operations of the vehicle caused by manual intervention.

Description

Vehicle control method, device and vehicle based on port vertical transportation target detection

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on March 15, 2024, with application number 202410310963.5 and application name “Vehicle control method, device and vehicle based on port vertical transportation target detection”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of port loading and unloading technology, and in particular to a vehicle control method, device and vehicle based on port vertical transportation target detection. Background Art

[0003] For vehicles operating in ports, ensuring vehicle dynamic stopping in various loading and unloading scenarios is of great technical significance, protecting the cargo from damage or loss. During the loading and unloading of boxes, precise dynamic stopping ensures stable transportation of the cargo, protecting it from vibration or collision.

[0004] In the prior art, during the loading and unloading process, the driver or corresponding supervisor generally determines whether the vehicle should start or stop. Generally, the loading and unloading controller tells the driver to control the vehicle to start or continue to stop according to the loading and unloading progress.

[0005] However, the above implementation relies too much on manual intervention. Long-term loading and unloading operations may cause fatigue of drivers or supervisors and lead to inattention, which may easily lead to misjudgment of vehicle starting and stopping, increasing the risk of accidents. Summary of the Invention

[0006] The present application provides a vehicle control method, device and vehicle based on port vertical transportation target detection to solve the problem of dangerous vehicle operation caused by manual intervention in the prior art.

[0007] In a first aspect, an embodiment of the present application provides a method, applied to a vehicle, comprising:

[0008] Acquiring first point cloud data of the area above the vehicle;

[0009] Determining, based on the first point cloud data, target position information and target gripping state of a spreader in the lifting equipment in a predetermined point cloud mapping relationship, wherein the point cloud mapping relationship records point cloud data corresponding to different position information and different gripping states of the spreader;

[0010] The vehicle's moving and stopping states are controlled according to the target position information and the target grasping state.

[0011] In one or more embodiments, determining target position information and target grasping state of a sling in the lifting equipment in a predetermined point cloud mapping relationship based on the first point cloud data includes:

[0012] Based on the first point cloud data, point cloud data with a difference value less than a first preset threshold is matched in the point cloud mapping relationship, and the position information and grasping status corresponding to the point cloud data with a difference value less than the first preset threshold are used as the target position information and target grasping status of the sling.

[0013] In one or more embodiments, the lifting equipment is a beam-type structured equipment;

[0014] Accordingly, before determining the target position information and the target grasping state of the sling in the lifting equipment in a predetermined point cloud mapping relationship based on the first point cloud data, the method further includes:

[0015] Acquire second point cloud data of an area of ​​interest of a sling in the lifting equipment from the first point cloud data, and use the second point cloud data as new first point cloud data; the area of ​​interest of the sling is determined based on third point cloud data of the beam in the lifting equipment and lane information of the vehicle.

[0016] In one or more embodiments, before obtaining second point cloud data of an interest area located in the spreader of the lifting equipment in the first point cloud data, the method further includes:

[0017] Acquiring third point cloud data of the beam in the lifting equipment;

[0018] Performing fitting processing on a point cloud corresponding to a projection of a normal vector of a point in the third point cloud data onto a vertical plane of the beam to obtain a vertical plane equation of the beam;

[0019] An area of ​​interest of a sling in the lifting equipment is determined according to lane information of the vehicle and a vertical plane equation of the beam.

[0020] In one or more embodiments, after obtaining second point cloud data of an interest area located in the spreader of the lifting equipment in the first point cloud data, the method further includes:

[0021] performing clustering processing on the point clouds in the second point cloud data to obtain different types of point clouds;

[0022] The point cloud in the second point cloud data is updated to a point cloud of a non-noise type among all types.

[0023] In one or more embodiments, before determining the target position information and target grasping state of the sling in the lifting equipment in a predetermined point cloud mapping relationship based on the first point cloud data, the method further includes:

[0024] performing clustering processing on the first point cloud data according to positional relationships of the point clouds to obtain different types of point clouds;

[0025] According to the lane information of the vehicle, the point cloud in the first point cloud data is updated to a point cloud of a non-clutter type among all types.

[0026] In one or more embodiments, the method further comprises:

[0027] The motion state of the spreader is determined according to the position information and position information of a previous frame of the position information.

[0028] In a second aspect, an embodiment of the present application provides a device, applied to a vehicle, comprising:

[0029] An acquisition module, configured to acquire first point cloud data of an area above the vehicle;

[0030] a determination module configured to determine, based on the first point cloud data, target position information and target gripping state of a spreader in the lifting equipment in a predetermined point cloud mapping relationship, wherein the point cloud mapping relationship records point cloud data corresponding to different position information and gripping states of the spreader;

[0031] The control module is used to control the moving and stopping state of the vehicle according to the target position information and the target grasping state.

[0032] In one or more embodiments, the determining module is specifically configured to:

[0033] Based on the first point cloud data, point cloud data with a difference value less than a first preset threshold is matched in the point cloud mapping relationship, and the position information and grasping status corresponding to the point cloud data with a difference value less than the first preset threshold are used as the target position information and target grasping status of the sling.

[0034] In one or more embodiments, the lifting equipment is a beam-type structured equipment;

[0035] Accordingly, before determining the target position information and the target grasping state of the spreader in the lifting equipment in the predetermined point cloud mapping relationship based on the first point cloud data, the determination module is further configured to:

[0036] Acquire second point cloud data of an area of ​​interest of a sling in the lifting equipment from the first point cloud data, and use the second point cloud data as new first point cloud data; the area of ​​interest of the sling is determined based on third point cloud data of the beam in the lifting equipment and lane information of the vehicle.

[0037] In one or more embodiments, before obtaining second point cloud data of an area of ​​interest located in the spreader of the lifting equipment in the first point cloud data, the determining module is further configured to:

[0038] Acquiring third point cloud data of the beam in the lifting equipment;

[0039] Performing fitting processing on a point cloud corresponding to a projection of a normal vector of a point in the third point cloud data onto a vertical plane of the beam to obtain a vertical plane equation of the beam;

[0040] An area of ​​interest of a sling in the lifting equipment is determined according to lane information of the vehicle and a vertical plane equation of the beam.

[0041] In one or more embodiments, after obtaining second point cloud data of an area of ​​interest located in a spreader of the lifting equipment in the first point cloud data, the determining module is further configured to:

[0042] performing clustering processing on the point clouds in the second point cloud data to obtain different types of point clouds;

[0043] The point cloud in the second point cloud data is updated to a point cloud of a non-noise type among all types.

[0044] In one or more embodiments, before determining the target position information and target grasping state of the spreader in the lifting equipment in a predetermined point cloud mapping relationship based on the first point cloud data, the determining module is further configured to:

[0045] performing clustering processing on the first point cloud data according to positional relationships of the point clouds to obtain different types of point clouds;

[0046] According to the lane information of the vehicle, the point cloud in the first point cloud data is updated to a point cloud of a non-clutter type among all types.

[0047] In one or more embodiments, the determining module is further configured to:

[0048] The motion state of the spreader is determined according to the position information and position information of a previous frame of the position information.

[0049] In a third aspect, the present application provides a vehicle, comprising: a processor, and a memory communicatively connected to the processor;

[0050] The memory stores computer-executable instructions;

[0051] The processor executes the computer-executable instructions stored in the memory to implement the method as described in the first aspect or any one of the above methods.

[0052] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the method described in the first aspect or any one of the above methods.

[0053] The present application provides a vehicle control method, device, and vehicle based on port vertical transportation target detection. The method is applied to a vehicle by acquiring first point cloud data of the area above the vehicle; based on the first point cloud data, the target position information and target gripping state of the sling in the lifting equipment are determined in a predetermined point cloud mapping relationship. The point cloud mapping relationship records point cloud data corresponding to different position information and different gripping states of the sling, and then the vehicle's dynamic and stop states are controlled based on the target position information and target gripping state. In this technical solution, the point cloud data of the area above is acquired by the vehicle, and then compared with the point cloud data corresponding to the possible sling positions and gripping states that have been determined in advance to determine the sling position and gripping state of the area above the current vehicle, thereby controlling the start and stop of the vehicle to avoid manual intervention that may cause dangerous operation of the vehicle. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0055] FIG1 is a flow chart of a vehicle control method based on port vertical transportation target detection according to an embodiment of the present application;

[0056] FIG2 is a second flow chart of a vehicle control method based on port vertical transportation target detection according to an embodiment of the present application;

[0057] FIG3 is a third flow chart of a vehicle control method based on port vertical transportation target detection provided by an embodiment of the present application;

[0058] FIG4 is a schematic diagram of the detection process of the Qiaolong large machine provided in an embodiment of the present application;

[0059] FIG5 is a schematic diagram of a detection process of a forklift provided in an embodiment of the present application;

[0060] FIG6 is a schematic structural diagram of a vehicle control device based on port vertical transportation target detection provided by an embodiment of the present application;

[0061] FIG7 is a schematic structural diagram of a vehicle provided in an embodiment of the present application.

[0062] The above drawings illustrate specific embodiments of the present disclosure, which will be described in more detail below. These drawings and textual descriptions are not intended to limit the scope of the present disclosure in any way, but rather to illustrate the concepts of the present disclosure to those skilled in the art by reference to specific embodiments. DETAILED DESCRIPTION

[0063] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0064] Before introducing the embodiments of the present application, the technical terms and application background of the embodiments of the present application are first explained:

[0065] Technical terms (involving port operations):

[0066] Re-closing: The spreader is holding the container and is in the air above and in front of the vehicle in the lane (or adjacent lane). This state is the re-closing scene;

[0067] Road closed: When there is no container on the spreader and the spreader is at a certain height above the lane in front of the vehicle (or adjacent lane), this state is the road closed scene;

[0068] Shelf: The container is placed on the trailer's baffle due to a discrepancy between the container and the trailer's expected loading position. This is called shelving.

[0069] Lifting: The container, spreader and trailer locks are all open, and the spreader has a tendency to rise. This state is the lifting state;

[0070] Start: When the spreader height is below the threshold height or the spreader is laterally within the working lane, this state is the working state and the vehicle needs to stop and wait for the work to be completed.

[0071] Technical background:

[0072] For vehicles operating in ports, ensuring vehicle dynamic stopping in various loading and unloading scenarios is of great technical significance, protecting the cargo from damage or loss. During the loading and unloading of boxes, precise dynamic stopping ensures stable transportation of the cargo, protecting it from vibration or collision.

[0073] In the prior art, during the loading and unloading process, the driver or corresponding supervisor generally determines whether the vehicle should start or stop. Generally, the loading and unloading controller tells the driver to control the vehicle to start or continue to stop according to the loading and unloading progress.

[0074] The problems existing in the prior art that need to be solved by the embodiments of the present application are: the above implementation relies too much on manual intervention, and long-term loading and unloading operations may cause fatigue of the driver or supervisor, and lack of concentration, which may easily lead to misjudgment of vehicle starting and stopping, increasing the risk of accidents.

[0075] In response to the technical problems existing in the prior art, the inventor of this application has the following idea: during actual port operations, the start and stop status of the vehicle are related to the position of the spreader and the gripping status of the spreader. If the possible positions of the spreader within its activity range and the corresponding point cloud information of the gripping status can be obtained first, in the actual scenario, the point cloud information within the activity range of the spreader can be collected by the sensor in real time, and then matched, the current position of the spreader, gripping status and other information can be accurately determined. Then, based on the matched information, the vehicle can be informed whether it should continue to wait, start and leave, or stop, etc., so as to avoid possible vehicle risks.

[0076] The technical solution of the present application is described in detail below through specific embodiments. It should be noted that the following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0077] It is worth noting that the application fields of the methods, devices, vehicles and storage media involved in the present disclosure are not limited, and can be applied to various scenarios such as spreader loading and unloading, not limited to port scenarios.

[0078] Among them, the executing entity of this application is the vehicle, specifically the control unit on the vehicle, etc.

[0079] FIG1 is a flow chart of a vehicle control method based on port vertical transport target detection according to an embodiment of the present application. As shown in FIG1 , the method may include the following steps:

[0080] Step 11: Acquire first point cloud data of the area above the vehicle;

[0081] In this step, when the vehicle is performing loading and unloading operations on the spreader, the vehicle obtains first point cloud data of the upper area in real time. The first point cloud data may include point clouds of various possible objects.

[0082] Among them, the upper area can include the front, upper, rear, left, and right sides of the vehicle, etc. The actual implementation is to adjust the corresponding acquisition angle based on needs.

[0083] Optionally, the first point cloud data of the area above the vehicle can be obtained by using sensors such as cameras, radars, etc. installed in the area above the vehicle. The pixel data obtained by some sensors can also be converted into point cloud data and converted into a certain coordinate system of the fixed position of the vehicle through coordinate transformation.

[0084] For example, the coordinate system may be based on the vehicle head position as the origin, with the front as the positive direction of the x-axis, the left as the positive direction of the y-axis, and the top as the positive direction of the z-axis.

[0085] It should be understood that the vertical transportation targets involved in the embodiments of the present application include but are not limited to: lifting equipment, slings in the lifting equipment, beams, tires, and side door grabs (boxes, containers, boxes, packages).

[0086] Among them, lifting equipment is not limited to gantry cranes, gantry cranes, tower cranes, construction elevators, forklifts, etc.

[0087] Step 12: Determine the target position information and target grasping state of the spreader in the lifting equipment in a predetermined point cloud mapping relationship based on the first point cloud data;

[0088] The point cloud mapping relationship records different position information of the sling and point cloud data corresponding to different grasping states.

[0089] In this step, the corresponding point cloud data for different positions and different grasping states are inconsistent.

[0090] Optionally, the grabbing status includes but is not limited to: whether the spreader grabs the object, whether the locks of the object, the spreader and the trailer are all open; the position information includes but is not limited to: the height of the spreader, the horizontal position of the spreader (i.e. the lane condition, etc.).

[0091] Optionally, the acquisition of point cloud data in the point cloud mapping relationship may be achieved by using the above-mentioned sensor to obtain the point cloud data, and filtering the obtained point cloud data using the region of interest (RoI) involved in the sling, and retaining only the point cloud data within the region of interest as the point cloud data in the point cloud mapping relationship.

[0092] The method for obtaining the region of interest is given in the following embodiment and will not be repeated here.

[0093] It should be understood that when the point cloud data in the point cloud mapping relationship is actually collected, the position information and the grasping status of the corresponding spreader are recorded.

[0094] For example, the grasped object may be a box, a container, a package, etc.; the spreader may be a clamp, a shovel, or other device that has the function of grasping the grasped object.

[0095] It should be understood that the acquisition of point cloud data is achieved based on a fixed coordinate system of the vehicle, and the point cloud mapping relationship is consistent with the acquisition of the first point cloud data.

[0096] Optionally, one implementation of step 12 is: based on the first point cloud data, match the point cloud data whose difference value is less than the first preset threshold in the point cloud mapping relationship, and use the position information and grasping status corresponding to the point cloud data whose difference value is less than the first preset threshold as the target position information and target grasping status of the sling.

[0097] Under this implementation, one possible implementation could be:

[0098] First, the first point cloud data and the point cloud data in the point cloud mapping relationship are initially aligned. Some initial estimation methods (such as feature matching and nearest neighbor search) are usually used to obtain a rough transformation matrix. Second, the iterative closest point (ICP) algorithm is used for iterative optimization to further improve the point cloud alignment effect. The ICP algorithm gradually adjusts the relative position and posture of the point clouds by finding the minimum square error between corresponding point pairs until the optimal alignment effect is achieved. Then, the difference value is calculated. After each ICP iteration, the difference value between the first point cloud data and the point cloud data in the mapping relationship is calculated. Common difference value calculation methods include average distance, least squares method, or Hausdorff distance, which are used to evaluate the similarity between the two sets of point clouds. Finally, according to application requirements, an appropriate difference value threshold (for example, a first preset threshold) is set to judge the matching result. By setting the threshold, the difference value can be compared with the threshold, and the point cloud data with a difference value within the threshold is considered to be successfully matched.

[0099] In other implementations, since point cloud data needs to be collected in real time, that is, different frames correspond to different first point cloud data, in order to improve efficiency, the matching result of the previous frame can be used as the starting matching point, and the front or back point cloud data corresponding to the matching result of the previous frame (the specific selection can be arbitrary or determined based on the previous operating status of the sling) can be used as the matching target for matching the first point cloud data, so that the target position information and target grasping status of the sling in the lifting equipment can be quickly determined.

[0100] In this implementation, the point cloud mapping relationship can be stored in the form of a tree-type point cloud set. For example, a certain range of spreader height is set, with spreaders at 1-3m as one branch and 3-5m as another branch. Each branch is equipped with point cloud data corresponding to different spreader heights within 1-3m. In addition, the distance between the lifting equipment and the vehicle (such as the horizontal distance) can be added to improve the matching efficiency.

[0101] Step 13: Control the vehicle's movement and stop status based on the target position information and the target grasping state.

[0102] In this step, the position of the hoist in the area above the vehicle and the state of gripping objects are obtained. At this time, based on the position of the hoist and the state of gripping objects, the vehicle's dynamic or stopped state can be confirmed based on the judgment conditions of scenes such as heavy closing, road closing, box storage, lifting, and abnormal starting.

[0103] Among them, the dynamic and stop states include: starting, stopping, running, braking, etc.

[0104] In one possible implementation, taking heavy-closing as an example, the height of the spreader is detected, and if the spreader is holding the container and is in the air in front of the vehicle's lane (or adjacent lane), the vehicle needs to stop and wait; taking road closing as an example, if it is detected that the spreader is not holding the container and is at a certain height above the vehicle's lane (or adjacent lane), the vehicle needs to stop and wait; taking container placement as an example, if it is detected that there is a difference between the expected loading position of the container and the vehicle's trailer, and the container is placed on the trailer baffle, the vehicle needs to stop and wait and issue a warning; taking lifting as an example, if it is detected that the container, spreader and trailer locks are all open, and the spreader has a tendency to rise, the vehicle needs to stop and wait and issue a warning; taking abnormal starting as an example, when the spreader height is below the threshold height or the spreader is laterally within the working lane range, the vehicle needs to stop and wait for the operation to be completed.

[0105] Optionally, controlling the vehicle's stopping state may also include issuing a prompt message to enable the driver or dispatcher to control the vehicle, or issuing corresponding instructions to automatically control the vehicle.

[0106] For example, if the current traffic jam is critical and the vehicle needs to stop and wait, the driver or dispatcher can be reminded by lighting a light, speaking, or displaying the information to control the vehicle to stop.

[0107] Furthermore, the motion state of the spreader may be determined based on the position information and the position information of the previous frame of the position information.

[0108] The movement state of the spreader includes: the movement direction, speed, etc. of the spreader; the movement direction may be determined based on the change direction of two frames of position information, and the speed may be determined based on the time interval.

[0109] The vehicle can then be controlled to stop or move based on the spreader's motion status. If the spreader is still moving, the vehicle must stop and wait.

[0110] In addition, the height of the spreader in the next frame can be estimated by using the least square method based on the heights of the spreaders in the two frames.

[0111] The vehicle control method based on port vertical transportation target detection provided in the embodiment of the present application is applied to the vehicle, by acquiring the first point cloud data of the area above the vehicle; based on the first point cloud data, the target position information and target grasping state of the sling in the lifting equipment are determined in a predetermined point cloud mapping relationship, and the point cloud mapping relationship records the different position information of the sling and the point cloud data corresponding to the different grasping states, and then the dynamic and stopping state of the vehicle is controlled according to the target position information and the target grasping state. In this technical solution, the point cloud data of the upper area is acquired by the vehicle, and then compared with the point cloud data corresponding to the possible sling positions and grasping states determined in advance to determine the sling position and grasping state of the current area above the vehicle, thereby controlling the start and stop of the vehicle to avoid manual intervention that may cause dangerous operation of the vehicle.

[0112] On the basis of the above embodiment, the lifting equipment is a beam-type structured equipment. FIG2 is a second flow chart of the vehicle control method based on port vertical transportation target detection provided by the embodiment of the present application. As shown in FIG2 , the following steps may be further included before step 12:

[0113] Step 21: Acquire second point cloud data of an interest area located in a sling in the lifting equipment from the first point cloud data, and use the second point cloud data as new first point cloud data;

[0114] The region of interest of the spreader is determined based on the third point cloud data of the beam in the lifting equipment and the lane information of the vehicle.

[0115] In this step, in order to ensure the above matching efficiency and accuracy, the first point cloud data can be updated, that is, the point cloud data (i.e., the second point cloud data) located in the active area of ​​the spreader (i.e., the area of ​​interest of the spreader) in the first point cloud data is retained.

[0116] Optionally, the region of interest of a spreader in a lifting device is determined as follows:

[0117] Step 1: Obtain the third point cloud data of the beam in the lifting equipment;

[0118] In this implementation, the vehicle obtains point cloud data of beams (eg, two beams) in the lifting equipment, which is recorded as third point cloud data.

[0119] One implementation is: determine the region of interest of the beam based on information such as the position of the beam in the lifting equipment, the size of the beam, and the approximate position of the crane determined by other methods (with meter-level accuracy), so that each beam has a region of interest; use these two regions of interest to filter the point cloud to obtain point clouds falling in different beam areas, i.e., the third point cloud data.

[0120] Step 2: Fit the point cloud corresponding to the projection of the normal vector of the midpoint of the third point cloud data onto the vertical plane of the beam to obtain the vertical plane equation of the beam;

[0121] In this step, the normal vector (n x ,n y ,n z ),

[0122] The z direction is the extension direction of the vehicle's lane, the y direction is the direction parallel to the beam, and the z direction is the direction perpendicular to the vehicle / lifting equipment.

[0123] Project the normal vector onto the xy plane (parallel to the ground) to obtain a two-dimensional vector (n x ,n y ), using relationships Filter the points in the point cloud whose normal vectors are approximately parallel to the xy plane, that is, the points that fall on the vertical surface of the beam. Where T is the set threshold, such as 0.99.

[0124] The RANSAC (random sampling consensus) method is then used to perform plane fitting on the points falling on the vertical plane of the beam, which can effectively eliminate the influence of noise points on the fitting. The vertical plane equation of the beam obtained by fitting is ax+by+cz+d=0.

[0125] It should be understood that the vertical plane equation of each beam corresponds to a corresponding vertical plane equation.

[0126] Step 3: Determine the region of interest of the hoisting equipment based on the lane information of the vehicle and the vertical plane equation of the beam.

[0127] In this implementation, the vertical plane equation of the beam obtains the precise position of the crane (for example, within 10 cm accuracy), and the region of interest of the lifting equipment's spreader is determined in combination with the lane information of the vehicle.

[0128] Afterwards, based on the region of interest, second point cloud data corresponding to the first point cloud data is determined.

[0129] Optionally, in step 21, after obtaining the second point cloud data of the area of ​​interest located in the sling in the lifting equipment in the first point cloud data, you can also: cluster the point clouds in the second point cloud data to obtain different types of point clouds, and then update the point clouds in the second point cloud data to non-noise type point clouds among all types.

[0130] In order to ensure the authenticity and validity of the second point cloud data of the area of ​​interest, the second point cloud data can be clustered (for example, distance can be used as the judgment condition for clustering), and the point clouds with a number less than a certain threshold after clustering (for example, and these point clouds are far away) are regarded as point clouds of the miscellaneous type. These point clouds are then removed and the other point clouds are retained, thereby obtaining the point clouds in the updated second point cloud data.

[0131] For example, these noise points may be rain in the air collected by sensors, etc. In order not to affect the matching effect, the above method can be used; wherein, the setting of the lower threshold is related to the number of point clouds collected for the spreaders and grasping objects. Generally speaking, the number of point clouds for spreaders and grasping objects is much larger than the number of point clouds corresponding to rain.

[0132] It should be understood that the thresholds involved in the embodiments of the present application can be set by the user based on experience, as well as adjusted by the size, density, etc. of the object.

[0133] The vehicle control method based on port vertical transport target detection provided in an embodiment of the present application obtains second point cloud data from a first point cloud data region located within a spreader of a lifting equipment and uses the second point cloud data as new first point cloud data. The spreader region of interest is determined based on third point cloud data of the lifting equipment's crossbeam and information about the vehicle's lane. This technical solution filters point cloud data outside the spreader region of interest to increase the accuracy and efficiency of matching using point cloud mapping relationships.

[0134] Based on the above embodiment, FIG3 is a flow chart of a vehicle control method based on port vertical transportation target detection according to an embodiment of the present application. As shown in FIG3 , the following steps may be included before step 12 (this implementation can be applied to any lifting equipment):

[0135] Step 31: clustering the first point cloud data according to the positional relationship of the point clouds to obtain different types of point clouds;

[0136] In this step, in order to ensure the authenticity and validity of the first point cloud data, the first point cloud data can be clustered by clustering, and the positional relationship of the point cloud (such as distance measurement, density clustering, etc.) can be used as the judgment condition for clustering to obtain different types of point clouds.

[0137] Step 32: Update the point cloud in the first point cloud data to a non-clutter point type among all types of point clouds according to the lane information of the vehicle.

[0138] In this step, when determining the target position information and target grasping state of the spreader, it can be considered that only the point cloud information within the relevant lane involved in the vehicle needs to be determined. Based on the lane information, the point cloud of non-noise type within the expected position range is retained.

[0139] Among them, the number of point clouds of the miscellaneous point type is relatively small. At this time, these point clouds are removed and the other point clouds are retained, that is, the point clouds in the updated first point cloud data are obtained.

[0140] For example, these noise points may be rain in the air collected by sensors, etc. In order not to affect the matching effect, the above method can be used; wherein, the setting of the lower threshold is related to the number of point clouds collected for the spreaders and grasping objects. Generally speaking, the number of point clouds for spreaders and grasping objects is much larger than the number of point clouds corresponding to rain.

[0141] The vehicle control method based on port vertical transport target detection provided in this embodiment of the application clusters first point cloud data based on the positional relationships of the point clouds to obtain different types of point clouds. Based on the lane information of the vehicle, the point clouds in the first point cloud data are updated to include non-cluttered point clouds among all types. This technical solution filters out non-cluttered point clouds to increase the accuracy and efficiency of matching using point cloud mapping relationships.

[0142] The following are two examples of specific implementation methods provided in the embodiments of the present application, taking a Qiaolong crane and a forklift as examples respectively:

[0143] FIG4 is a schematic diagram of the detection process of the Qiaolong large machine provided in an embodiment of the present application. As shown in FIG4 , the schematic diagram includes:

[0144] Step 41: Install sensors, typically cameras and / or lidars, at positions and angles that cover the area above the vehicle.

[0145] Step 42: Coordinate conversion, converting the sensor data to a coordinate system fixed to the vehicle, where the direction directly in front of the vehicle is the positive x-axis direction of the coordinate system, the direction directly to the left of the vehicle is the positive y-axis direction, and the direction directly above the vehicle is the positive z-axis direction;

[0146] Step 43: Collect point cloud data of each position (x, y, z) of the spreader relative to the vehicle and whether the spreader is holding a box, and record the spreader position value and spreader holding state value corresponding to each point cloud set; store the point cloud set in a tree topology according to the true value as a reference point cloud set (i.e., a point cloud mapping relationship);

[0147] Step 44: Determine the region of interest (RoI) for the crane beam based on the crane's position, beam dimensions, and the crane's approximate position determined by other methods (with meter-level accuracy). Each of the two beams has an RoI. Use these two RoIs to filter the point cloud to obtain point clouds that fall within different beam regions.

[0148] Step 45: Estimate the normal vector (n x ,n y ,n z ), Project the normal vector onto the xy plane (parallel to the ground) to obtain a two-dimensional vector (n x ,n y ), using relationships Filter the points in the point cloud whose normal vectors are approximately parallel to the xy plane, that is, the points that fall on the vertical surface of the beam. T is a set threshold, such as 0.99.

[0149] Step 46: Use the Random Sample Consensus (RANSAC) method to perform plane fitting on the points on the vertical plane of the beam obtained in step 44. This effectively eliminates the influence of noise points on the fitting. The vertical plane equation of the beam obtained by fitting is ax + by + cz + d = 0. At the same time, the interior points in the fitting process are obtained, that is, the points used to determine the vertical plane equation in the fitting process.

[0150] Step 47: Obtain the precise crane position (within 10 cm accuracy) using the vertical plane equations of the two beams in step 45. Combined with the vehicle's lane information, determine the spreader's region of interest (RoI) and obtain a spreader's RoI point cloud.

[0151] Step 48: cluster the point cloud of the spreader area of ​​interest using the k-means method, and retain the non-clutter point cloud class within the expected position range based on the lane where the vehicle is located and the number of each type of point cloud;

[0152] Step 49: Perform point cloud matching on the point cloud in step 48 and the reference point cloud set in step 43. The matching result of the previous frame is used as the starting matching point. The reference point cloud group is extracted from the tree-type reference point cloud set according to the set threshold. The ICP method is used to obtain the difference values ​​in the detection point cloud and the reference point cloud group. The matching result is obtained according to the preset threshold of the difference value set. The detection position (x, y, z) and the box grabbing status of the spreader are obtained based on the matching result.

[0153] Step 50: Based on the currently calculated spreader height and the n frames of historically stored spreader heights, the spreader height of the next frame is estimated using the least squares method to obtain the current spreader speed and direction.

[0154] FIG5 is a schematic diagram of a detection process of a forklift provided in an embodiment of the present application. As shown in FIG5 , the schematic diagram includes:

[0155] Step 51: Install the sensor, usually a camera or lidar, at a location and angle that covers the area above the vehicle.

[0156] Step 52: Coordinate conversion, converting the sensor data to a coordinate system fixed to the vehicle, where the direction directly in front of the vehicle is the positive x-axis direction of the coordinate system, the direction directly to the left of the vehicle is the positive y-axis direction, and the direction directly above the vehicle is the positive z-axis direction;

[0157] Step 53: Collect point cloud data of the spreader's relative position (x, y, z) relative to the vehicle and whether the spreader is holding a container. Record the spreader position value and spreader holding state value for each point cloud. Store the point cloud set in a tree topology based on the true value as a reference point cloud set.

[0158] Step 54: Based on the positional relationship of the point clouds, the k-means method is used to cluster the point clouds, and the non-clutter point cloud class within the expected position range is retained according to the lane where the vehicle is located and the number of each type of point cloud;

[0159] Step 55: Perform point cloud matching on the point cloud in step 54 and the reference point cloud set in step 53. The matching result of the previous frame is used as the starting matching point. A reference point cloud group is extracted from the tree-type reference point cloud set according to a set threshold. The ICP method is used to obtain the difference values ​​between the detection point cloud and the reference point cloud group. The matching result is obtained based on the preset threshold of the difference value set. The detection position (x, y, z) of the spreader and the box grabbing status are obtained based on the matching result.

[0160] Step 56 : Based on the currently calculated spreader height and the historically stored spreader heights of n frames, the spreader height of the next frame is estimated using the least squares method to obtain the current spreader speed and direction.

[0161] The technical solutions and technical effects of the embodiments shown in FIG4 and FIG5 are similar to those of the above-mentioned embodiments and will not be described in detail here.

[0162] The following are device embodiments of the present application, which can be used to implement the method embodiments of the present application. For details not disclosed in the device embodiments of the present application, please refer to the method embodiments of the present application.

[0163] FIG6 is a schematic diagram of the structure of a vehicle control device based on port vertical transportation target detection provided by an embodiment of the present application. As shown in FIG6 , the device is applied to a vehicle and includes:

[0164] An acquisition module 61 is configured to acquire first point cloud data of an area above the vehicle;

[0165] a determination module 62 for determining, based on the first point cloud data, target position information and target gripping state of a spreader in the lifting equipment in a predetermined point cloud mapping relationship, wherein the point cloud mapping relationship records point cloud data corresponding to different position information and gripping states of the spreader;

[0166] The control module 63 is used to control the vehicle's movement and stop status according to the target position information and the target grasping state.

[0167] In one or more embodiments, the determination module 62 is specifically configured to:

[0168] According to the first point cloud data, point cloud data with a difference value less than a first preset threshold is matched in the point cloud mapping relationship, and the position information and grasping status corresponding to the point cloud data with a difference value less than the first preset threshold are used as the target position information and target grasping status of the sling.

[0169] In one or more embodiments, the lifting device is a device with a beam structure;

[0170] Accordingly, before determining the target position information and the target grasping state of the spreader in the lifting equipment in the predetermined point cloud mapping relationship based on the first point cloud data, the determination module 62 is further configured to:

[0171] Second point cloud data of an area of ​​interest located on a sling in the lifting equipment in the first point cloud data is obtained, and the second point cloud data is used as new first point cloud data; the area of ​​interest of the sling is determined based on third point cloud data of the beam in the lifting equipment and lane information of the vehicle.

[0172] In one or more embodiments, before obtaining second point cloud data of the area of ​​interest located in the spreader of the lifting equipment in the first point cloud data, the determination module 62 is further configured to:

[0173] Obtain the third point cloud data of the beam in the lifting equipment;

[0174] Fitting the point cloud corresponding to the projection of the normal vector of the midpoint of the third point cloud data onto the vertical plane of the beam to obtain the vertical plane equation of the beam;

[0175] The region of interest of the spreader in the lifting equipment is determined based on the lane information of the vehicle and the vertical plane equation of the beam.

[0176] In one or more embodiments, after obtaining second point cloud data of an area of ​​interest located in a spreader of the lifting equipment in the first point cloud data, the determination module 62 is further configured to:

[0177] performing clustering processing on the point clouds in the second point cloud data to obtain different types of point clouds;

[0178] The point cloud in the second point cloud data is updated to a point cloud of a non-noise type among all types.

[0179] In one or more embodiments, before determining the target position information and target grasping state of the spreader in the lifting equipment in a predetermined point cloud mapping relationship based on the first point cloud data, the determination module 62 is further configured to:

[0180] Clustering the first point cloud data according to the positional relationship of the point clouds to obtain different types of point clouds;

[0181] According to the lane information of the vehicle, the point cloud in the first point cloud data is updated to a point cloud of a non-clutter type among all types.

[0182] In one or more embodiments, the determination module 62 is further configured to:

[0183] The motion state of the spreader is determined based on the position information and the position information of the previous frame of the position information.

[0184] The device provided in the embodiments of the present application can be used to execute the method in any of the above embodiments. Its implementation principles and technical effects are similar and will not be repeated here.

[0185] It should be noted that it should be understood that the division of the various modules of the above device is merely a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. Moreover, these modules can all be implemented in the form of software called by processing elements; they can also all be implemented in the form of hardware; some modules can also be implemented in the form of software called by processing elements, and some modules can be implemented in the form of hardware. In addition, these modules can be fully or partially integrated together or implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each of the above modules can be completed by the hardware integrated logic circuit in the processor element or by instructions in the form of software.

[0186] Figure 7 is a structural schematic diagram of a vehicle provided in an embodiment of the present application. As shown in Figure 7, the vehicle may include: a processor 71, a memory 72, and computer program instructions stored in the memory 72 and executable on the processor 71. When the processor 71 executes the computer program instructions, the method provided in any of the aforementioned embodiments is implemented.

[0187] Optionally, the above-mentioned components of the vehicle can be connected via a system bus.

[0188] The memory 72 may be a separate storage unit or a storage unit integrated in the processor 71. The number of the processor 71 may be one or more.

[0189] It should be understood that the processor 71 can be a central processing unit (CPU), or other general-purpose processors 71, digital signal processors 71 (DSP), application-specific integrated circuits (ASIC), etc. The general-purpose processor 71 can be a microprocessor 71 or any conventional processor 71. The steps of the method disclosed in this application can be directly implemented by the hardware processor 71 or implemented by a combination of hardware and software modules in the processor 71.

[0190] The system bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, among others. A system bus can be divided into an address bus, a data bus, a control bus, and so on. For ease of illustration, the figure shows only one thick line, but this does not imply that there is only one bus or only one type of bus. Memory 72 may include random access memory 72 (RAM) and may also include non-volatile memory 72 (NVM), such as at least one disk storage 72.

[0191] All or part of the steps of the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a readable memory 72. When the program is executed, it performs the steps of the above-mentioned method embodiments; and the aforementioned memory 72 (storage medium) includes: read-only memory 72 (ROM), RAM, flash memory 72, hard disk, solid-state drive, magnetic tape, floppy disk, optical disc, and any combination thereof.

[0192] The vehicle provided in the embodiments of the present application can be used to execute the method provided in any of the above-mentioned method embodiments. The implementation principles and technical effects are similar and will not be repeated here.

[0193] An embodiment of the present application provides a computer-readable storage medium, in which computer instructions are stored. When the computer instructions are executed on a computer, the computer executes the above method.

[0194] The computer-readable storage medium mentioned above may be implemented by any type of volatile or non-volatile memory device, or a combination thereof, such as static random access memory, electrically erasable programmable read-only memory, erasable programmable read-only memory, programmable read-only memory, read-only memory, magnetic storage, flash memory, magnetic disk, or optical disk. The computer-readable storage medium may be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0195] Optionally, a readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be an integral part of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0196] An embodiment of the present application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor can read the computer program from the computer-readable storage medium, and the at least one processor can implement the above method when executing the computer program.

[0197] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A vehicle control method based on port vertical transportation target detection, characterized in that: Applied to a vehicle, the method comprises: Acquiring first point cloud data of the area above the vehicle; Determining, based on the first point cloud data, target position information and target gripping state of a spreader in the lifting equipment in a predetermined point cloud mapping relationship, wherein the point cloud mapping relationship records point cloud data corresponding to different position information and different gripping states of the spreader; The vehicle's moving and stopping states are controlled according to the target position information and the target grasping state.

2. The method according to claim 1, characterized in that Determining target position information and target grasping state of a sling in the lifting equipment in a predetermined point cloud mapping relationship based on the first point cloud data includes: Based on the first point cloud data, point cloud data with a difference value less than a first preset threshold is matched in the point cloud mapping relationship, and the position information and grasping status corresponding to the point cloud data with a difference value less than the first preset threshold are used as the target position information and target grasping status of the sling.

3. The method according to claim 1 or 2, characterized in that The lifting equipment is a beam-type structured equipment; Accordingly, before determining the target position information and the target grasping state of the sling in the lifting equipment in a predetermined point cloud mapping relationship based on the first point cloud data, the method further includes: Obtain second point cloud data of an area of ​​interest of a sling in the lifting equipment from the first point cloud data, and use the second point cloud data as new first point cloud data, wherein the area of ​​interest of the sling is determined based on third point cloud data of the beam in the lifting equipment and lane information of the vehicle.

4. The method according to claim 3, characterized in that Before acquiring second point cloud data of an interest area located in the spreader of the lifting equipment in the first point cloud data, the method further includes: Acquiring third point cloud data of the beam in the lifting equipment; Performing fitting processing on a point cloud corresponding to a projection of a normal vector of a point in the third point cloud data onto a vertical plane of the beam to obtain a vertical plane equation of the beam; An area of ​​interest of a sling in the lifting equipment is determined according to lane information of the vehicle and a vertical plane equation of the beam.

5. The method according to claim 3, characterized in that After obtaining second point cloud data of an interest area located in a spreader of the lifting equipment from the first point cloud data, the method further includes: performing clustering processing on the point clouds in the second point cloud data to obtain different types of point clouds; The point cloud in the second point cloud data is updated to a point cloud of a non-noise type among all types.

6. The method according to claim 1 or 2, characterized in that Before determining the target position information and target grasping state of the sling in the lifting equipment in a predetermined point cloud mapping relationship based on the first point cloud data, the method further includes: performing clustering processing on the first point cloud data according to positional relationships of the point clouds to obtain different types of point clouds; According to the lane information of the vehicle, the point cloud in the first point cloud data is updated to a point cloud of a non-clutter type among all types.

7. The method according to claim 1 or 2, characterized in that The method further comprises: The motion state of the spreader is determined according to the position information and position information of a previous frame of the position information.

8. A vehicle control device based on port vertical transportation target detection, characterized in that: Applied to a vehicle, the device comprises: An acquisition module, configured to acquire first point cloud data of an area above the vehicle; a determination module configured to determine, based on the first point cloud data, target position information and target gripping state of a spreader in the lifting equipment in a predetermined point cloud mapping relationship, wherein the point cloud mapping relationship records point cloud data corresponding to different position information and gripping states of the spreader; The control module is used to control the moving and stopping state of the vehicle according to the target position information and the target grasping state.

9. A vehicle, characterized in that: include: a processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 7 when executed by a processor.

Images