Vehicle control method, apparatus and vehicle based on detection of port vertical transport target

Abstract

The present application provides a vehicle control method, apparatus and vehicle based on detection of a port vertical transport target. The method is applied to a vehicle. First point cloud data of a region above the vehicle is acquired, then second point cloud data located in the region of interest of the spreader or the grabbed object is determined from the first point cloud data according to the first point cloud data and a pre-determined region of interest of a spreader or a grabbed object in a lifting device, and position information and grabbing state of the spreader corresponding to the second point cloud data are determined according to feature information of the second point cloud data, and the moving and stopping state of the vehicle is controlled according to the position information and the grabbing state. In this technical solution, the acquired point cloud data is filtered from the region of interest of the spreader or the grabbed object, to improve the efficiency of subsequent processing. Then, after feature extraction is performed based on the filtered point cloud data, corresponding processing is performed to obtain the position information and the grabbing state of the spreader, thereby more accurately realizing the determination of the starting and stopping state of the vehicle.

Description

Vehicle control method, device and vehicle based on port vertical transportation target detection

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on March 15, 2024, with application number 202410304020.1 and application name “Vehicle control method, device and vehicle based on port vertical transportation target detection”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of port loading and unloading technology, and in particular to a vehicle control method, device and vehicle based on port vertical transportation target detection. Background Art

[0003] For transport vehicles operating in ports, ensuring their precise stopping and starting in various loading and unloading scenarios is of great technical significance. This can effectively reduce the incidence of accidents and ensure the safety of drivers, staff, and the surrounding environment. Especially in busy port environments, accurate control of transport vehicle stopping and starting can prevent collisions and injuries.

[0004] In the prior art, during the loading and unloading process, the driver or corresponding supervisor generally determines whether the transport vehicle should start or stop. Generally, the loading and unloading controller tells the driver to control the transport vehicle to start or continue to stop according to the loading and unloading progress.

[0005] However, in the above implementation, there may be errors in the subjective judgment of the driver or supervisor, especially in a complex loading and unloading operation environment, which is easily affected by external factors, resulting in inaccurate judgment and causing damage to the transport vehicle. Summary of the Invention

[0006] The present application provides a vehicle control method, device and vehicle based on port vertical transportation target detection to solve the problem of dangerous vehicle operation caused by manual intervention in the prior art.

[0007] In a first aspect, an embodiment of the present application provides a method, a vehicle control method based on port vertical transportation target detection, characterized in that, when applied to a vehicle, the method includes:

[0008] Acquiring first point cloud data of the area above the vehicle;

[0009] Determining, based on the first point cloud data and a predetermined region of interest of a spreader or a grasped object in the lifting equipment, second point cloud data located in the region of interest of the spreader or grasped object in the first point cloud data;

[0010] determining, based on the characteristic information of the second point cloud data, position information and a grasping state of the spreader corresponding to the second point cloud data;

[0011] The vehicle's moving and stopping states are controlled according to the position information and the grasping state.

[0012] In one or more embodiments, the lifting equipment is a beam-type structured equipment;

[0013] Accordingly, determining the position information and the grasping state of the spreader corresponding to the second point cloud data according to the feature information of the second point cloud data includes:

[0014] dividing the point clouds in the second point cloud data into plane point clouds of different dimensions based on normal vectors of points in the second point cloud data, and if the number of point clouds in the plane is greater than a first threshold value, determining the vertical mean of the point clouds on the plane as the height position of the spreader; or / and, vertically slicing the second point cloud data, and obtaining the vertical mean of the point clouds in the cut area having the largest number of point clouds after slicing and a number of point clouds after slicing greater than a second threshold value as the height position of the spreader;

[0015] determining, based on a point cloud density of a plane in a horizontal dimension and a first density threshold, or a vertical value of a point cloud of a plane in another dimension and a first value threshold, whether the spreader grasps the grasped object;

[0016] The position information includes the height of the sling; and the object-grabbing state includes the situation in which the sling grasps the object.

[0017] In one or more embodiments, the region of interest of the spreader is the area formed by the beam of the lifting equipment and the lane in which the vehicle is located;

[0018] Accordingly, before determining second point cloud data located in the region of interest of the spreader in the first point cloud data based on the first point cloud data and the predetermined region of interest of the spreader in the lifting equipment, the method further includes:

[0019] Acquiring third point cloud data of the beam;

[0020] Performing fitting processing on a point cloud corresponding to a projection of a normal vector of a point in the third point cloud data onto a vertical plane of the beam to obtain a vertical plane equation of the beam;

[0021] An area of ​​interest of a sling in the lifting equipment is determined according to lane information of the vehicle and a vertical plane equation of the beam.

[0022] In one or more embodiments, before performing fitting processing on the point cloud corresponding to the projection of the normal vector of the point in the third point cloud data onto the vertical plane of the beam to obtain the vertical plane equation of the beam, the method further includes:

[0023] Projecting the normal vector of the midpoint of the third point cloud data onto the horizontal plane to obtain a two-dimensional vector;

[0024] The two-dimensional vector is filtered using a preset strategy to obtain a point cloud corresponding to the vertical plane projected onto the beam. The preset strategy is to filter the points corresponding to the two-dimensional vector whose sum of the squares of the data in the two-dimensional vector is greater than a preset threshold.

[0025] In one or more embodiments, the lifting equipment is a lifting structure equipment;

[0026] Accordingly, determining the position information and the grasping state of the spreader corresponding to the second point cloud data according to the feature information of the second point cloud data includes:

[0027] performing clustering and filtering processing on the second point cloud data to obtain fourth point cloud data;

[0028] dividing the point cloud in the fourth point cloud data into plane point clouds of different dimensions based on the normal vectors of the points in the fourth point cloud data, performing plane fitting if the number of the point clouds in the plane is greater than a third number threshold, and taking the vertical mean of the point clouds on the fitted plane as the height position of the spreader;

[0029] determining whether the spreader grasps the object based on whether the fitted plane exists;

[0030] The height position includes the height position of the sling; and the object grasping state includes the condition of the sling grasping the object.

[0031] In one or more embodiments, the region of interest for the grasped object is the region formed by the lifting of the object by the sling belt of the lifting equipment;

[0032] Accordingly, before determining second point cloud data located in the region of interest of the grasped object in the first point cloud data based on the first point cloud data and the predetermined region of interest of the grasped object in the lifting equipment, the method further includes:

[0033] acquiring fifth point cloud data of the lifting equipment within an expected range;

[0034] Determining the center position of the lifting equipment based on the side door plane area and tire area of ​​the lifting equipment determined after clustering the point clouds of multiple part types of the fifth point cloud data;

[0035] The region of interest of the grasped object is determined according to the center position.

[0036] In one or more embodiments, the method further comprises:

[0037] The motion state of the spreader is determined according to the position information and position information of a previous frame of the position information.

[0038] In a second aspect, an embodiment of the present application provides a vehicle control device based on port vertical transport target detection, characterized in that the device is applied to a vehicle and includes:

[0039] An acquisition module, configured to acquire first point cloud data of an area above the vehicle;

[0040] a first determining module configured to determine, based on the first point cloud data and a predetermined region of interest of a spreader or a grasped object in the lifting equipment, second point cloud data located in the region of interest of the spreader or grasped object in the first point cloud data;

[0041] a second determining module, configured to determine position information and a grasping state of a spreader corresponding to the second point cloud data based on characteristic information of the second point cloud data;

[0042] The control module is used to control the moving and stopping state of the vehicle according to the position information and the grasping state.

[0043] In one or more embodiments, the lifting equipment is a beam-type structured equipment;

[0044] Accordingly, the second determining module is specifically configured to:

[0045] dividing the point clouds in the second point cloud data into plane point clouds of different dimensions based on normal vectors of points in the second point cloud data, and if the number of point clouds in the plane is greater than a first threshold value, determining the vertical mean of the point clouds on the plane as the height position of the spreader; or / and, vertically slicing the second point cloud data, and obtaining the vertical mean of the point clouds in the cut area having the largest number of point clouds after slicing and a number of point clouds after slicing greater than a second threshold value as the height position of the spreader;

[0046] determining, based on a point cloud density of a plane in a horizontal dimension and a first density threshold, or a vertical value of a point cloud of a plane in another dimension and a first value threshold, whether the spreader grasps the grasped object;

[0047] The position information includes the height of the sling; and the object-grabbing state includes the situation in which the sling grasps the object.

[0048] In one or more embodiments, the region of interest of the spreader is the area formed by the beam of the lifting equipment and the lane in which the vehicle is located;

[0049] Accordingly, before determining second point cloud data located in the region of interest of the spreader in the first point cloud data based on the first point cloud data and the predetermined region of interest of the spreader in the lifting equipment, the second determining module is further configured to:

[0050] Acquiring third point cloud data of the beam;

[0051] Performing fitting processing on a point cloud corresponding to a projection of a normal vector of a point in the third point cloud data onto a vertical plane of the beam to obtain a vertical plane equation of the beam;

[0052] An area of ​​interest of a sling in the lifting equipment is determined according to lane information of the vehicle and a vertical plane equation of the beam.

[0053] In one or more embodiments, before performing fitting processing on the point cloud corresponding to the projection of the normal vector of the point in the third point cloud data onto the vertical plane of the beam to obtain the vertical plane equation of the beam, the second determining module is further configured to:

[0054] Projecting the normal vector of the midpoint of the third point cloud data onto the horizontal plane to obtain a two-dimensional vector;

[0055] The two-dimensional vector is filtered using a preset strategy to obtain a point cloud corresponding to the vertical plane projected onto the beam. The preset strategy is to filter the points corresponding to the two-dimensional vector whose sum of the squares of the data in the two-dimensional vector is greater than a preset threshold.

[0056] In one or more embodiments, the lifting equipment is a lifting structure equipment;

[0057] Accordingly, the second determining module is specifically configured to:

[0058] performing clustering and filtering processing on the second point cloud data to obtain fourth point cloud data;

[0059] dividing the point cloud in the fourth point cloud data into plane point clouds of different dimensions based on the normal vectors of the points in the fourth point cloud data, performing plane fitting if the number of the point clouds in the plane is greater than a third number threshold, and taking the vertical mean of the point clouds on the fitted plane as the height position of the spreader;

[0060] determining whether the spreader grasps the object based on whether the fitted plane exists;

[0061] The height position includes the height position of the sling; and the object grasping state includes the condition of the sling grasping the object.

[0062] In one or more embodiments, the region of interest for the grasped object is the region formed by the lifting of the object by the sling belt of the lifting equipment;

[0063] Accordingly, before determining second point cloud data located in the region of interest of the grasped object in the first point cloud data based on the first point cloud data and the predetermined region of interest of the grasped object in the lifting equipment, the second determining module is further configured to:

[0064] acquiring fifth point cloud data of the lifting equipment within an expected range;

[0065] Determining the center position of the lifting equipment based on the side door plane area and tire area of ​​the lifting equipment determined after clustering the point clouds of multiple part types of the fifth point cloud data;

[0066] The region of interest of the grasped object is determined according to the center position.

[0067] In one or more embodiments, the second determining module is further configured to:

[0068] The motion state of the spreader is determined according to the position information and position information of a previous frame of the position information.

[0069] In a third aspect, the present application provides a vehicle, comprising: a processor, and a memory and a transceiver communicatively connected to the processor;

[0070] The memory stores computer-executable instructions; the transceiver is used to transmit and receive data;

[0071] The processor executes the computer-executable instructions stored in the memory to implement the method as described in the first aspect or any one of the above methods.

[0072] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the method described in the first aspect or any one of the above methods.

[0073] The present application provides a vehicle control method, device, and vehicle based on port vertical transportation target detection. The method is applied to a vehicle by acquiring first point cloud data of the area above the vehicle. Then, based on the first point cloud data and a predetermined area of ​​interest of a sling or a gripping object in the lifting equipment, second point cloud data located in the area of ​​interest of the sling or gripping object in the first point cloud data is determined. Based on the feature information of the second point cloud data, the position information and gripping status of the sling corresponding to the second point cloud data are determined. Based on the position information and gripping status, the vehicle's start and stop status is controlled. In this technical solution, the acquired point cloud data from the area of ​​interest of the sling or gripping object is filtered to improve the efficiency of subsequent processing. After that, feature extraction is performed based on the filtered point cloud data, and corresponding processing is performed to obtain the position information and gripping status of the sling, thereby more accurately realizing the judgment of the start and stop status of the vehicle. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0075] FIG1 is a flow chart of a vehicle control method based on port vertical transportation target detection according to an embodiment of the present application;

[0076] FIG2 is a second flow chart of a vehicle control method based on port vertical transportation target detection according to an embodiment of the present application;

[0077] FIG3 is a third flow chart of a vehicle control method based on port vertical transportation target detection provided by an embodiment of the present application;

[0078] FIG4 is a fourth flow chart of a vehicle control method based on port vertical transportation target detection according to an embodiment of the present application;

[0079] FIG5 is a fifth flow chart of a vehicle control method based on port vertical transportation target detection according to an embodiment of the present application;

[0080] FIG6 is a schematic diagram of the detection process of the Qiaolong large machine provided in an embodiment of the present application;

[0081] FIG7 is a schematic diagram of a detection process of a forklift provided in an embodiment of the present application;

[0082] FIG8 is a schematic structural diagram of a vehicle control device based on port vertical transportation target detection provided by an embodiment of the present application;

[0083] FIG9 is a schematic structural diagram of a vehicle provided in an embodiment of the present application.

[0084] The above drawings illustrate specific embodiments of the present disclosure, which will be described in more detail below. These drawings and textual descriptions are not intended to limit the scope of the present disclosure in any way, but rather to illustrate the concepts of the present disclosure to those skilled in the art by reference to specific embodiments. DETAILED DESCRIPTION

[0085] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0086] Before introducing the embodiments of the present application, the application background of the embodiments of the present application is first explained:

[0087] Technical terms (involving port operations):

[0088] Re-closing: The spreader is holding the container and is in the air above and in front of the vehicle in the lane (or adjacent lane). This state is the re-closing scene;

[0089] Road closed: When there is no container on the spreader and the spreader is at a certain height above the lane in front of the vehicle (or adjacent lane), this state is the road closed scene;

[0090] Shelf: The container is placed on the trailer's baffle due to a discrepancy between the container and the trailer's expected loading position. This is called shelving.

[0091] Lifting: The container, spreader and trailer locks are all open, and the spreader has a tendency to rise. This state is the lifting state;

[0092] Start: When the spreader height is below the threshold height or the spreader is laterally within the working lane, this state is the working state and the vehicle needs to stop and wait for the work to be completed.

[0093] Technical background:

[0094] For vehicles operating in ports, ensuring vehicle dynamic stopping in various loading and unloading scenarios is of great technical significance, protecting the cargo from damage or loss. During the loading and unloading of boxes, precise dynamic stopping ensures stable transportation of the cargo, protecting it from vibration or collision.

[0095] In the prior art, during the loading and unloading process, the driver or corresponding supervisor generally determines whether the transport vehicle should start or stop. Generally, the loading and unloading controller tells the driver to control the transport vehicle to start or continue to stop according to the loading and unloading progress.

[0096] The problems existing in the prior art that need to be solved by the embodiments of the present application are: the subjective judgment of the driver or supervisor in the above implementation may have errors, especially in a complex loading and unloading operation environment, which is easily affected by external factors, resulting in inaccurate judgment, thereby causing damage to the transport vehicle, etc.

[0097] In response to the technical problems existing in the prior art, the inventors of this application have the following idea: during actual port operations, the start and stop status of the vehicle are related to the position of the spreader and the grasping status of the spreader. If the possible position of the spreader within its range of activity can be obtained first, in the actual scenario, point cloud information can be collected by sensors in real time and filtered based on the area of ​​the above position. The characteristic information of the point cloud data can correspond to the position and status of the box and spreader. For example, if there is a box being grasped, the density of the captured point cloud is larger or the number of point clouds is larger. The corresponding position information can also be determined based on the position on the coordinate axis, thereby avoiding possible vehicle risks.

[0098] The technical solution of the present application is described in detail below through specific embodiments. It should be noted that the following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0099] It is worth noting that the application fields of the methods, devices, vehicles and storage media involved in the present disclosure are not limited.

[0100] Among them, the executing entity of this application is the vehicle, specifically the control unit on the vehicle, etc.

[0101] FIG1 is a flow chart of a vehicle control method based on port vertical transport target detection according to an embodiment of the present application. As shown in FIG1 , the method may include the following steps:

[0102] Step 11: Acquire first point cloud data of the area above the vehicle;

[0103] In this step, when the vehicle is performing loading and unloading operations on the spreader, the vehicle obtains first point cloud data of the upper area in real time. The first point cloud data may include point clouds of various possible objects.

[0104] Among them, the upper area can include the front, upper, rear, left, and right sides of the vehicle, etc. The actual implementation is to adjust the corresponding acquisition angle based on needs.

[0105] Optionally, the first point cloud data of the area above the vehicle can be obtained by using sensors such as cameras, radars, etc. installed in the area above the vehicle. The pixel data obtained by some devices can also be converted into point cloud data and converted into a certain coordinate system of the fixed position of the vehicle through coordinate transformation.

[0106] For example, the coordinate system may be based on the vehicle head position as the origin, with the front as the positive direction of the x-axis, the left as the positive direction of the y-axis, and the top as the positive direction of the z-axis.

[0107] It should be understood that the vertical transportation targets involved in the embodiments of the present application include but are not limited to: lifting equipment, slings in the lifting equipment, beams, tires, and side door grabs (boxes, containers, boxes, packages).

[0108] Among them, lifting equipment is not limited to gantry cranes, gantry cranes, tower cranes, construction elevators, forklifts, etc.

[0109] Step 12: Determine, based on the first point cloud data and a predetermined region of interest of a spreader or a grasping object in the lifting equipment, second point cloud data located in a region of interest (RoI) of the spreader or grasping object in the first point cloud data;

[0110] In this step, for equipment with a beam structure, such as a large crane, the area of ​​interest of the spreader is the area where the spreader moves between the beams; for equipment with a lifting structure, such as a forklift, the area of ​​interest of the grasped object is the area corresponding to the possible lifting and lowering of the grasped object. The following embodiments describe the two areas of interest in detail.

[0111] The grasped objects may be boxes, containers, packages, etc.; the slings may be clamps, shovels, and other devices with the function of grasping the grasped objects.

[0112] Optionally, after the first point cloud data of the area above the vehicle is obtained, the first point cloud data may be filtered using the region of interest to obtain point cloud data within the region of interest, which is recorded as second point cloud data.

[0113] It should be understood that for different lifting equipment, the vehicle can make judgments based on point clouds, images, etc. obtained by sensors, or it can be determined manually.

[0114] Step 13: Determine the position information and gripping state of the spreader corresponding to the second point cloud data based on the feature information of the second point cloud data;

[0115] In this step, after the second point cloud data is acquired, feature extraction is performed on the second point cloud data to obtain feature information.

[0116] The feature information of point cloud data can describe the number, density, and z-value of the point cloud projected onto the z-axis; in some implementations, it can also distinguish different types of objects, etc.

[0117] The number and density of the point cloud, as well as the z value and density projected to the z-axis, can also reflect the grasping status, position information, etc.

[0118] Optionally, the grabbing status includes but is not limited to: whether the spreader grabs the object, whether the locks of the object, the spreader and the trailer are all open; the position information includes but is not limited to: the height of the spreader, the horizontal position of the spreader (i.e. the lane condition, etc.).

[0119] Step 14: Control the vehicle's movement and stop status based on the position information and the grasping state.

[0120] In this step, the position of the hoist in the area above the vehicle and the state of gripping objects are obtained. At this time, based on the position of the hoist and the state of gripping objects, the vehicle's dynamic or stopped state can be confirmed based on the judgment conditions of scenes such as heavy closing, road closing, box storage, lifting, and abnormal starting.

[0121] Among them, the dynamic and stop states include: starting, stopping, running, braking, etc.

[0122] In one possible implementation, taking heavy-closing as an example, the height of the spreader is detected, and if the spreader is holding the container and is in the air in front of the vehicle's lane (or adjacent lane), the vehicle needs to stop and wait; taking road closing as an example, if it is detected that the spreader is not holding the container and is at a certain height above the vehicle's lane (or adjacent lane), the vehicle needs to stop and wait; taking container placement as an example, if it is detected that there is a difference between the expected loading position of the container and the vehicle's trailer, and the container is placed on the trailer baffle, the vehicle needs to stop and wait and issue a warning; taking lifting as an example, if it is detected that the container, spreader and trailer locks are all open, and the spreader has a tendency to rise, the vehicle needs to stop and wait and issue a warning; taking abnormal starting as an example, when the spreader height is below the threshold height or the spreader is laterally within the working lane range, the vehicle needs to stop and wait for the operation to be completed.

[0123] Optionally, controlling the vehicle's stopping state may also include issuing a prompt message to enable the driver or dispatcher to control the vehicle, or issuing corresponding instructions to automatically control the vehicle.

[0124] For example, if the current traffic jam is critical and the vehicle needs to stop and wait, the driver or dispatcher can be reminded by lighting a light, speaking, or displaying the information to control the vehicle to stop.

[0125] Furthermore, the motion state of the spreader may be determined based on the position information and the position information of the previous frame of the position information.

[0126] The movement state of the spreader includes: the movement direction, speed, etc. of the spreader; the movement direction may be determined based on the change direction of two frames of position information, and the speed may be determined based on the time interval.

[0127] The vehicle can then be controlled to stop or move based on the spreader's motion status. If the spreader is still moving, the vehicle must stop and wait.

[0128] In addition, the height of the spreader in the next frame can be estimated by using the least square method based on the heights of the spreaders in the two frames.

[0129] The vehicle control method based on port vertical transportation target detection provided in the embodiment of the present application is applied to a vehicle, and obtains first point cloud data of the area above the vehicle, and then determines second point cloud data located in the area of ​​interest of the spreader or grip in the first point cloud data based on the first point cloud data and a predetermined area of ​​interest of the spreader or grip in the lifting equipment, and determines the position information and grip state of the spreader corresponding to the second point cloud data based on the feature information of the second point cloud data, and controls the vehicle's start and stop state based on the position information and grip state. In this technical solution, the point cloud data obtained from the area of ​​interest of the spreader or grip is filtered to improve the efficiency of subsequent processing, and then feature extraction is performed based on the filtered point cloud data, and corresponding processing is performed to obtain the position information and grip state of the spreader, thereby more accurately realizing the judgment of the start and stop state of the vehicle.

[0130] 1. For lifting equipment with a beam structure (such as a gantry crane):

[0131] Based on the above embodiment, FIG2 is a second flow chart of a vehicle control method based on port vertical transportation target detection provided by an embodiment of the present application. As shown in FIG2 , step 13 may be:

[0132] Step 21: Based on the normal vectors of the midpoints in the second point cloud data, the point cloud in the second point cloud data is divided into plane point clouds of different dimensions. If the number of point clouds in the plane is greater than a first threshold, the vertical mean of the point clouds on the plane is determined as the height position of the spreader. Alternatively, / or, the second point cloud data is vertically sliced, and the vertical mean of the point clouds in the sliced ​​area having the largest number of point clouds and a number of point clouds greater than the second threshold is determined as the height position of the spreader.

[0133] In this step, after the second point cloud data is obtained as described above, the point cloud in the second point cloud data is divided into plane point clouds of different dimensions according to the normal vectors in the second point cloud data.

[0134] For example, it can be the xy plane (horizontal plane), the yz plane (vertical plane of the beam), and the xz plane. That is, the z direction is the extension direction of the vehicle's road, the y direction is the direction parallel to the beam, and the z direction is the direction perpendicular to the vehicle / lifting equipment.

[0135] The first method is to determine the vertical mean of the point cloud on the plane as the height position of the spreader if the number of point clouds on the plane is greater than the first threshold value:

[0136] The number of point clouds of various planes is checked. When the first number threshold is reached, the plane fitting is performed using the random sampling consistency method. The z-mean of the point cloud of each plane can represent the spreader height calculated by the feature.

[0137] The second method is to cut the second point cloud data in the vertical direction, and obtain the vertical average value of the point cloud of the cutting plane where the cut point cloud is the largest and the number of cut point clouds is greater than the second number threshold as the height position of the spreader:

[0138] The second point cloud data in the region of interest of the spreader is evenly segmented in the z direction to obtain a segmented region with the most point clouds. When the number of point clouds in this region exceeds a second quantity threshold, the z mean of the point clouds in this region represents the spreader height calculated by this feature.

[0139] In summary, in order to increase the accuracy of the height position of the spreader, the results of the first method and the second method can be compared and voted on to obtain the spreader detection height z value.

[0140] In addition, the position of the spreader can be determined based on the position x of the lifting equipment and the lane information y of the vehicle to obtain the detected position (x, y) value of the spreader.

[0141] It should be understood that the thresholds involved in the embodiments of the present application can be set by the user based on experience, as well as adjusted by the size, density, etc. of the object.

[0142] Step 22: Determine whether the spreader grasps the object based on the point cloud density of the horizontal plane and the first density threshold, or the vertical value of the point cloud of the other plane and the first value threshold;

[0143] The position information includes the height of the spreader; the object grasping status includes the situation of the spreader grasping the object.

[0144] In this step, since the point cloud density and number of the sling are relatively small compared to the grasped object, after obtaining each feature plane (i.e., planes of different dimensions), when the point cloud density of the xy plane is greater than the first density threshold, it is considered that the sling grasps the box; when the z-range of the point cloud of the xz plane is greater than the first numerical threshold, it is considered that the sling grasps the box; when the z-range of the point cloud of the yz plane is greater than the first numerical threshold, it is considered that the sling grasps the box.

[0145] The vehicle control method based on port vertical transport target detection provided by the embodiment of the present application is to divide the point cloud in the second point cloud data into plane point clouds of different dimensions according to the normal vectors of the points in the second point cloud data. If the number of point clouds in the plane is greater than a first threshold, the vertical mean of the point clouds on the plane is determined as the height position of the spreader; or / and, the second point cloud data is cut in the vertical direction, and the vertical mean of the point clouds in the cut area with the largest number of cut point clouds and a number of cut point clouds greater than a second threshold is obtained as the height position of the spreader. The position information includes the height position of the spreader; the grasping state includes the grasping state of the spreader. In this technical solution, since the number of point clouds, point cloud density, and vertical value of the point clouds of the spreader and the grasping object are different in planes of different dimensions, the spreader height and the grasping state of the spreader can be accurately determined based on the numerical value of the point cloud number and point cloud density.

[0146] Based on the above embodiment, FIG3 is a flow chart of a vehicle control method based on port vertical transportation target detection according to an embodiment of the present application. As shown in FIG3 , the method for determining the region of interest of the spreader can be:

[0147] Step 31: Obtain the third point cloud data of the beam;

[0148] In this step, the vehicle obtains point cloud data of beams (for example, two beams) in the lifting equipment, which is recorded as third point cloud data.

[0149] One implementation is: determine the region of interest of the beam based on information such as the position of the beam in the lifting equipment, the size of the beam, and the approximate position of the crane determined by other methods (with meter-level accuracy), so that each beam has a region of interest; use these two regions of interest to filter the point cloud to obtain point clouds falling in different beam areas, i.e., the third point cloud data.

[0150] Before the following steps, the normal vector of the point in the third point cloud data can also be projected onto the horizontal plane to obtain a two-dimensional vector; the two-dimensional vector is filtered using a preset strategy to obtain a point cloud corresponding to the vertical plane projected onto the beam. The preset strategy is to filter the points corresponding to the two-dimensional vectors whose sum of the squares of the data in the two-dimensional vectors is greater than a preset threshold.

[0151] That is, the preset strategy can be: filtering point.

[0152] In this implementation, the normal vector (n x ,n y ,n z ), Then project the normal vector onto the xy plane (horizontal plane) to obtain a two-dimensional vector (n x ,n y ), using relationships The points in the point cloud whose normal vectors are approximately parallel to the xy plane are obtained by filtering, that is, the points falling on the vertical surface of the beam, that is, the points on the yz plane.

[0153] Where T is a preset threshold, such as 0.99.

[0154] Step 32: Fitting the point cloud corresponding to the projection of the normal vector of the midpoint of the third point cloud data onto the vertical plane of the beam to obtain the vertical plane equation of the beam;

[0155] The random sampling consensus (RANSAC) method is used to perform plane fitting on the points falling on the vertical plane of the beam, which can effectively eliminate the influence of the noise points on the fitting. The vertical plane equation of the beam obtained by fitting is ax+by+cz+d=0.

[0156] It should be understood that the vertical plane equation of each beam corresponds to a corresponding vertical plane equation.

[0157] Step 33: Determine the region of interest of the lifting device in the lifting equipment according to the lane information of the vehicle and the vertical plane equation of the beam.

[0158] In this implementation, the vertical plane equation of the beam obtains the precise position of the crane (for example, within 10 cm accuracy), and the region of interest of the lifting equipment's spreader is determined in combination with the lane information of the vehicle.

[0159] The vehicle control method based on port vertical transport target detection provided in an embodiment of the present application obtains third point cloud data of a beam and performs fitting processing on the point cloud corresponding to the projection of the normal vector of the point in the third point cloud data onto the vertical plane of the beam to obtain the vertical plane equation of the beam. Based on the vehicle's lane information and the vertical plane equation of the beam, the region of interest of the spreader in the lifting equipment is determined. In this technical solution, the lane and beam are used to determine the area where the spreader may be located, thereby filtering the subsequently collected point cloud to improve the efficiency and accuracy of subsequent processing.

[0160] 2. Lifting equipment is a lifting structure equipment (such as forklift):

[0161] Based on the above embodiment, FIG4 is a fourth flow chart of a vehicle control method based on port vertical transportation target detection provided by an embodiment of the present application. As shown in FIG4 , step 13 may be:

[0162] Step 41: clustering and filtering the second point cloud data to obtain fourth point cloud data;

[0163] In this step, after obtaining the second point cloud data, the point cloud is first clustered using the k-means method, and then the non-clutter point cloud class is retained according to the lane information of the vehicle and the number of each type of point cloud, thereby obtaining the fourth point cloud data.

[0164] Step 42: Based on the normal vectors of the midpoints of the fourth point cloud data, the point cloud in the fourth point cloud data is divided into plane point clouds of different dimensions. If the number of plane point clouds is greater than a third threshold, plane fitting is performed, and the vertical mean of the point clouds on the fitted plane is taken as the height position of the spreader.

[0165] In this step, the point cloud in the fourth point cloud data is divided into three types of plane point clouds (different dimensions) according to the direction of the normal vector: xy plane, xz plane, and yz plane.

[0166] Then, the number of point clouds of each dimensional plane is checked respectively. When the third quantity threshold is reached, the random sampling consistency method is used to perform plane fitting. The z-mean of the point cloud of the obtained plane can represent the spreader height calculated by the feature.

[0167] In one implementation, the spreader height is calculated for each plane, and then compared and voted with each other to obtain the grasped object detection height z value.

[0168] Step 43: Determine whether the spreader is grabbing the object based on whether the fitted plane exists;

[0169] If any of the fitted planes is detected, the spreader grabs the object; if any of the fitted planes is not detected, the spreader grabs the unclawed object.

[0170] The height position includes the height position of the spreader; and the object grasping state includes the state of the spreader grasping the object.

[0171] The vehicle control method based on port vertical transportation target detection provided by the embodiment of the present application obtains fourth point cloud data by clustering and filtering the second point cloud data. The point cloud in the fourth point cloud data is divided into plane point clouds of different dimensions according to the normal vector of the point in the fourth point cloud data. If the number of plane point clouds is greater than a third quantity threshold, plane fitting is performed, and the vertical mean of the point cloud on the fitted plane is taken as the height position of the sling. The situation of the sling grabbing the object is determined based on whether the fitted plane exists. When the type characterizes the sling, the height position of the object corresponding to the type includes the height position of the sling, and the grasping state includes the situation of the sling grabbing the object. In this technical solution, the point cloud data is clustered, and for each type, the point clouds of planes of different dimensions are used to judge the height of the sling and the situation of the sling grabbing the object.

[0172] Based on the above embodiment, FIG5 is a flowchart of a vehicle control method based on port vertical transportation target detection according to an embodiment of the present application. As shown in FIG5 , the region of interest of the grasped object may be determined in the following manner:

[0173] Step 51: Acquire fifth point cloud data of the lifting equipment within the expected range;

[0174] The expected range may be the range in which the gripper and the grasped object of the lifting structure equipment may appear, or may refer to a certain angle range.

[0175] The point cloud information within the expected range is obtained, and then any fixed point on the vehicle itself is used as the origin of the coordinate system to obtain the fifth point cloud data after coordinate transformation.

[0176] Step 52: Based on the point clouds of various types of parts obtained by clustering the fifth point cloud data, the side door plane area and tire area of ​​the lifting equipment are determined, and the center position of the lifting equipment is determined;

[0177] In this step, based on the positional relationship of the point clouds, the k-means method is used to cluster the fifth point cloud data to obtain multiple types of point clouds.

[0178] In addition, after clustering, the non-clutter point cloud class can be retained based on the lane information of the vehicle and the number of each type of point cloud.

[0179] Furthermore, the corresponding area of ​​interest is determined through the range of various point clouds, and the random sampling consistency method is used to calculate the side door plane of the lifting equipment using the point cloud in the area of ​​interest. The tire area of ​​interest is determined using the plane position, and the random sampling consistency method is used to calculate the tire plane using the point cloud in the area of ​​interest (this is because the point cloud features of the tire and side door are relatively obvious and easy to determine).

[0180] Afterwards, the center position of the lifting equipment is obtained by mutual verification based on the side door plane and the tire plane.

[0181] In addition, based on the center position, the (x, y) value of the grasped object position can be converted.

[0182] Step 53: Determine the region of interest of the grasped object based on the center position.

[0183] In this step, the region of interest of the grasped object (such as a container) is determined based on the center position of the lifting equipment, and the normal vector of each point in the point cloud of the region of interest is estimated, (n x ,n y ,n z ), Points are divided into three categories by normal vector direction: xy plane, xz plane, yz plane

[0184] That is, the xy plane (horizontal plane), the yz plane (left and right of the vehicle, vertical plane), and the xz plane (front and back of the vehicle, vertical plane).

[0185] The vehicle control method based on port vertical transport target detection provided in an embodiment of the present application obtains fifth point cloud data of a crane within a desired range. Based on clustering the fifth point cloud data into point clouds of various part types, the center position of the crane is determined by identifying the side door plane area and tire area of ​​the crane. Based on this center position, the region of interest for the grasped object is determined. In this technical solution, the area of ​​interest for the grasped object is determined by using the side door plane area and tire area of ​​the crane to determine the area where the grasped object may appear, thereby enabling the subsequent point cloud to be filtered to improve the efficiency and accuracy of subsequent processing.

[0186] The following are two examples of specific implementation methods provided in the embodiments of the present application, taking a Qiaolong crane and a forklift as examples respectively:

[0187] FIG6 is a schematic diagram of the detection process of the Qiaolong large machine provided in an embodiment of the present application. As shown in FIG6 , the schematic diagram includes:

[0188] Step 611: Install the sensor, usually a camera or a lidar. The installation position and angle need to cover the area above the vehicle body.

[0189] Step 612: Coordinate conversion, converting the sensor data to a coordinate system fixed to the vehicle, where the direction directly in front of the vehicle is the positive x-axis direction, the direction directly to the left of the vehicle is the positive y-axis direction, and the direction directly above the vehicle is the positive z-axis direction.

[0190] Step 613: Determine the crane beam region of interest (ROI) based on the crane beam's position, beam dimensions, and the crane's approximate position determined by other methods (with meter-level accuracy). Each of the two beams has an ROI. Use these two ROIs to filter the point cloud, obtaining point clouds that fall within different beam regions.

[0191] Step 614: Estimate the normal vector (n x ,n y ,n z ), Project the normal vector to the xy plane (horizontal plane) to obtain a two-dimensional vector (n x ,n y ), using relationships Filter the points in the point cloud whose normal vectors are approximately parallel to the xy plane, that is, the points that fall on the vertical surface of the beam. T is a set threshold, such as 0.99.

[0192] Step 615: Use the Random Sample Consensus (RANSAC) method to perform plane fitting on the points on the vertical plane of the yz beam obtained in step 614. This effectively eliminates the influence of noise on the fitting. The fitted vertical plane equation of the beam is ax + by + cz + d = 0. The inliers in the fitting process are also obtained, i.e., the points ultimately used to determine the vertical plane equation during the fitting process.

[0193] Step 616: Obtain the precise crane position (within 10 cm accuracy) using the vertical plane equations of the two beams in step 615. Combined with the vehicle's lane information, determine the crane's spreader region of interest (RoI) and obtain a spreader region of interest point cloud.

[0194] Step 617: Estimate the normal vector (n x ,n y ,n z ), The points are divided into three categories according to the direction of the normal vector: xy plane, xz plane, and yz plane.

[0195] Step 618: Check the number of each type of plane point cloud respectively. When the number threshold is reached, use the random sampling consensus (RANSAC) method to perform plane fitting. The z-mean value of the point cloud of each plane can represent the spreader height calculated by the feature.

[0196] Step 619: Evenly segment the crane spreader region of interest (RoI) in the z direction to obtain the segmented region with the most point clouds. When the number of point clouds in this region exceeds a threshold, the z-mean of the point clouds in this region represents the spreader height calculated for this feature.

[0197] Step 620 : Compare and vote on the spreader heights calculated based on the features obtained in steps 618 and 619 to obtain a spreader detection height z value.

[0198] Step 621 : Obtain the detection position (x, y) value of the spreader according to the crane position x and the vehicle lane information y obtained in step 616 .

[0199] Step 622: Based on the feature planes obtained in step 618, when the density of the xy plane point cloud is greater than the threshold, it is considered that the spreader has grasped the box; when the z-range of the xz plane point cloud is greater than the threshold, it is considered that the spreader has grasped the box; when the z-range of the yz plane point cloud is greater than the threshold, it is considered that the spreader has grasped the box.

[0200] Step 623: Compare and vote on whether the spreader has grabbed a box based on the results of each feature obtained in step 620 to obtain the spreader's state of grabbing a box.

[0201] Step 624 : Based on the currently calculated spreader height and the historically stored spreader heights of n frames, the spreader height of the next frame is estimated using the least squares method to obtain the current spreader speed and direction.

[0202] FIG7 is a schematic diagram of a detection process of a forklift provided in an embodiment of the present application. As shown in FIG7 , the schematic diagram includes:

[0203] Step 711: Install the sensor, usually a camera or a lidar. The installation position and angle need to cover the area above the vehicle body.

[0204] Step 712: Coordinate conversion. Convert the sensor data to a coordinate system fixed to the vehicle, where the direction directly in front of the vehicle is the positive x-axis direction, the direction directly to the left of the vehicle is the positive y-axis direction, and the direction directly above the vehicle is the positive z-axis direction.

[0205] Step 713: Based on the positional relationship of the point clouds, the k-means method is used to cluster the point clouds, and the non-cluttered point cloud classes within the expected position range are retained according to the lane where the vehicle is located and the number of each type of point cloud.

[0206] Step 714: Determine the region of interest based on the ranges of various point clouds, and use the Random Sampling Consensus (RANSAC) method to calculate the plane of the forklift side door on the point cloud of the region of interest.

[0207] Step 715 : Determine the tire region of interest based on the plane position in step 714 , and calculate the tire plane using a random sampling consensus (RANSAC) method on the point cloud in the region of interest.

[0208] Step 716: Based on the mutual verification of the side door plane and the tire plane in step 714 and step 715, the center position of the forklift is obtained.

[0209] Step 717: Determine the interest region of the forklift container based on the center position of the forklift in step 716, and estimate the normal vector of each point in the point cloud of the interest region, (n x ,n y ,n z ), Points are divided into three categories by the direction of the normal vector: xy plane, xz plane, and yz plane.

[0210] Step 718: Check the number of plane point clouds of each type respectively. When the number threshold is reached, use the random sampling consensus (RANSAC) method to perform plane fitting. The z-mean value of the point cloud of each plane can represent the spreader height calculated by the feature.

[0211] Step 719: According to whether the three types of planes exist in step 718, the state of the forklift grabbing the box is obtained.

[0212] Step 720: Compare and vote on the spreader heights calculated based on the various features in step 718 to obtain a z value for the container detection height of the forklift.

[0213] Step 721: According to the center position of the forklift in step 716, the forklift container position (x, y) value is converted.

[0214] Step 722: Based on the currently calculated forklift container height and the historically stored n-frame spreader heights, the next frame spreader height is estimated using the least squares method to obtain the current forklift container speed and direction.

[0215] The technical solutions and technical effects of the embodiments shown in FIG6 and FIG7 are similar to those of the above-mentioned embodiments and will not be described in detail here.

[0216] The following are device embodiments of the present application, which can be used to implement the method embodiments of the present application. For details not disclosed in the device embodiments of the present application, please refer to the method embodiments of the present application.

[0217] FIG8 is a schematic diagram of the structure of a vehicle control device based on port vertical transportation target detection provided by an embodiment of the present application. As shown in FIG8 , the device is applied to a vehicle and includes:

[0218] An acquisition module 81 is configured to acquire first point cloud data of an area above the vehicle;

[0219] A first determining module 82 is configured to determine, based on the first point cloud data and a predetermined region of interest of a spreader or a grasping object in the lifting equipment, second point cloud data located in the region of interest of the spreader or grasping object in the first point cloud data;

[0220] A second determining module 83 is configured to determine the position information and the gripping state of the spreader corresponding to the second point cloud data based on the feature information of the second point cloud data;

[0221] The control module 84 is used to control the vehicle's movement and stop status according to the position information and the grasping state.

[0222] In one or more embodiments, the lifting device is a device with a beam structure;

[0223] Accordingly, the second determining module 83 is specifically configured to:

[0224] dividing the point cloud in the second point cloud data into plane point clouds of different dimensions based on the normal vectors of the midpoints in the second point cloud data, and if the number of point clouds in the plane is greater than a first threshold value, determining the vertical mean of the point clouds on the plane as the height position of the spreader; or / and, vertically slicing the second point cloud data, and obtaining the vertical mean of the point clouds in the cut area having the largest number of point clouds after slicing and a number of point clouds after slicing greater than the second threshold value as the height position of the spreader;

[0225] determining whether the spreader grasps the grasped object based on a point cloud density of a plane in a horizontal dimension and a first density threshold, or a vertical value of a point cloud of a plane in another dimension and a first value threshold;

[0226] The position information includes the height of the spreader; the object grasping status includes the situation of the spreader grasping the object.

[0227] In one or more embodiments, the region of interest of the spreader is the area formed by the beam of the lifting equipment and the lane in which the vehicle is located;

[0228] Accordingly, before determining the second point cloud data located in the interest region of the spreader in the first point cloud data based on the first point cloud data and the predetermined interest region of the spreader in the lifting equipment, the second determining module 83 is further configured to:

[0229] Obtain the third point cloud data of the beam;

[0230] Fitting the point cloud corresponding to the projection of the normal vector of the midpoint of the third point cloud data onto the vertical plane of the beam to obtain the vertical plane equation of the beam;

[0231] The region of interest of the spreader in the lifting equipment is determined based on the lane information of the vehicle and the vertical plane equation of the beam.

[0232] In one or more embodiments, before performing fitting processing on the point cloud corresponding to the projection of the normal vector of the point in the third point cloud data onto the vertical plane of the beam to obtain the vertical plane equation of the beam, the second determining module 83 is further configured to:

[0233] Project the normal vector of the point in the third point cloud data onto the horizontal plane to obtain a two-dimensional vector;

[0234] The two-dimensional vectors are filtered using a preset strategy to obtain a point cloud corresponding to the vertical plane projected onto the beam. The preset strategy is to filter the points corresponding to the two-dimensional vectors whose sum of the squares of the data in the two-dimensional vectors is greater than a preset threshold.

[0235] In one or more embodiments, the lifting device is a lifting structure device;

[0236] Accordingly, the second determining module 83 is specifically configured to:

[0237] performing clustering and filtering processing on the second point cloud data to obtain fourth point cloud data;

[0238] dividing the point cloud in the fourth point cloud data into plane point clouds of different dimensions based on the normal vectors of the midpoints of the fourth point cloud data, performing plane fitting if the number of the plane point clouds is greater than a third number threshold, and taking the vertical mean of the point clouds on the fitted plane as the height position of the spreader;

[0239] The situation of the spreader grabbing the object is determined based on whether the fitted plane exists;

[0240] The height position includes the height position of the spreader; and the object grasping state includes the state of the spreader grasping the object.

[0241] In one or more embodiments, the region of interest for grabbing an object is the region formed by the lifting of the object by the sling belt of the lifting equipment;

[0242] Accordingly, before determining the second point cloud data located in the region of interest of the grasped object in the first point cloud data based on the first point cloud data and the predetermined region of interest of the grasped object in the lifting device, the second determining module 83 is further configured to:

[0243] Acquire fifth point cloud data of the lifting equipment within the expected range;

[0244] Based on the point clouds of various types of parts clustered from the fifth point cloud data, the side door plane area and tire area of ​​the lifting equipment are determined, and the center position of the lifting equipment is determined;

[0245] Based on the center position, the region of interest of the grasped object is determined.

[0246] In one or more embodiments, the second determining module 83 is further configured to:

[0247] The motion state of the spreader is determined based on the position information and the position information of the previous frame of the position information.

[0248] The device provided in the embodiments of the present application can be used to execute the method in any of the above embodiments. Its implementation principles and technical effects are similar and will not be repeated here.

[0249] It should be noted that it should be understood that the division of the various modules of the above device is merely a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. Moreover, these modules can all be implemented in the form of software called by processing elements; they can also all be implemented in the form of hardware; some modules can also be implemented in the form of software called by processing elements, and some modules can be implemented in the form of hardware. In addition, these modules can be fully or partially integrated together or implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each of the above modules can be completed by the hardware integrated logic circuit in the processor element or by instructions in the form of software.

[0250] Figure 9 is a structural schematic diagram of a vehicle provided in an embodiment of the present application. As shown in Figure 9, the vehicle may include: a processor 91, a memory 92, and computer program instructions stored in the memory 92 and executable on the processor 91. When the processor 91 executes the computer program instructions, the method provided in any of the aforementioned embodiments is implemented.

[0251] Optionally, the above-mentioned components of the vehicle can be connected via a system bus.

[0252] The memory 92 may be a separate storage unit or a storage unit integrated in the processor 91. The number of the processor 91 may be one or more.

[0253] It should be understood that the processor 91 can be a central processing unit (CPU), or other general-purpose processors 91, digital signal processors 91 (DSP), application-specific integrated circuits (ASIC), etc. The general-purpose processor 91 can be a microprocessor 91 or any conventional processor 91. The steps of the method disclosed in this application can be directly implemented by the hardware processor 91 or performed by a combination of hardware and software modules in the processor 91.

[0254] The system bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, among others. A system bus can be divided into an address bus, a data bus, a control bus, and so on. For ease of illustration, the figure uses only one thick line, but this does not imply that there is only one bus or only one type of bus. Memory 92 may include random access memory 92 (RAM) and may also include non-volatile memory 92 (NVM), such as at least one disk storage 92.

[0255] All or part of the steps of the above-mentioned method embodiments can be completed by hardware associated with program instructions. The aforementioned program can be stored in a readable memory 92. When executed, the program performs the steps of the above-mentioned method embodiments; and the aforementioned memory 92 (storage medium) includes: read-only memory 92 (ROM), RAM, flash memory 92, hard disk, solid-state drive, magnetic tape, floppy disk, optical disc, and any combination thereof.

[0256] The vehicle provided in the embodiments of the present application can be used to execute the method provided in any of the above-mentioned method embodiments. The implementation principles and technical effects are similar and will not be repeated here.

[0257] An embodiment of the present application provides a computer-readable storage medium, in which computer instructions are stored. When the computer instructions are executed on a computer, the computer executes the above method.

[0258] The computer-readable storage medium mentioned above may be implemented by any type of volatile or non-volatile memory device, or a combination thereof, such as static random access memory, electrically erasable programmable read-only memory, erasable programmable read-only memory, programmable read-only memory, read-only memory, magnetic storage, flash memory, magnetic disk, or optical disk. The computer-readable storage medium may be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0259] Optionally, a readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be an integral part of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0260] An embodiment of the present application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor can read the computer program from the computer-readable storage medium, and the at least one processor can implement the above method when executing the computer program.

[0261] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A vehicle control method based on port vertical transportation target detection, characterized in that: Applied to a vehicle, the method comprises: Acquiring first point cloud data of the area above the vehicle; Determining, based on the first point cloud data and a predetermined region of interest of a spreader or a grasped object in the lifting equipment, second point cloud data located in the region of interest of the spreader or grasped object in the first point cloud data; determining, based on the characteristic information of the second point cloud data, position information and a grasping state of the spreader corresponding to the second point cloud data; The vehicle's moving and stopping states are controlled according to the position information and the grasping state.

2. The method according to claim 1, characterized in that The lifting equipment is a beam-type structured equipment; Accordingly, determining the position information and the grasping state of the spreader corresponding to the second point cloud data according to the feature information of the second point cloud data includes: dividing the point clouds in the second point cloud data into plane point clouds of different dimensions based on normal vectors of points in the second point cloud data, and if the number of point clouds in the plane is greater than a first threshold value, determining the vertical mean of the point clouds on the plane as the height position of the spreader; or / and, vertically slicing the second point cloud data, and obtaining the vertical mean of the point clouds in the cut area having the largest number of point clouds after slicing and a number of point clouds after slicing greater than a second threshold value as the height position of the spreader; determining, based on a point cloud density of a plane in a horizontal dimension and a first density threshold, or a vertical value of a point cloud of a plane in another dimension and a first value threshold, whether the spreader grasps the grasped object; The position information includes the height of the sling; and the object-grabbing state includes the situation in which the sling grasps the object.

3. The method according to claim 2, characterized in that The region of interest of the spreader is the area formed by the crossbeam of the lifting equipment and the lane where the vehicle is located; Accordingly, before determining second point cloud data located in the region of interest of the spreader in the first point cloud data based on the first point cloud data and the predetermined region of interest of the spreader in the lifting equipment, the method further includes: Acquiring third point cloud data of the beam; Performing fitting processing on a point cloud corresponding to a projection of a normal vector of a point in the third point cloud data onto a vertical plane of the beam to obtain a vertical plane equation of the beam; An area of ​​interest of a sling in the lifting equipment is determined according to lane information of the vehicle and a vertical plane equation of the beam.

4. The method according to claim 3, characterized in that Before performing fitting processing on the point cloud corresponding to the projection of the normal vector of the midpoint of the third point cloud data onto the vertical plane of the beam to obtain the vertical plane equation of the beam, the method further includes: Projecting the normal vector of the midpoint of the third point cloud data onto the horizontal plane to obtain a two-dimensional vector; The two-dimensional vector is filtered using a preset strategy to obtain a point cloud corresponding to the vertical plane projected onto the beam. The preset strategy is to filter the points corresponding to the two-dimensional vector whose sum of the squares of the data in the two-dimensional vector is greater than a preset threshold.

5. The method according to claim 1, wherein The lifting equipment is a lifting structure equipment; Accordingly, determining the position information and the grasping state of the spreader corresponding to the second point cloud data according to the feature information of the second point cloud data includes: performing clustering and filtering processing on the second point cloud data to obtain fourth point cloud data; dividing the point cloud in the fourth point cloud data into plane point clouds of different dimensions based on the normal vectors of the points in the fourth point cloud data, performing plane fitting if the number of the point clouds in the plane is greater than a third number threshold, and taking the vertical mean of the point clouds on the fitted plane as the height position of the spreader; determining whether the spreader grasps the object based on whether the fitted plane exists; The height position includes the height position of the sling; and the object grasping state includes the state of the sling grasping the object.

6. The method according to claim 5, characterized in that The area of ​​interest for the grasped object is the area formed by the lifting of the object by the sling of the lifting equipment; Accordingly, before determining second point cloud data located in the region of interest of the grasped object in the first point cloud data based on the first point cloud data and the predetermined region of interest of the grasped object in the lifting equipment, the method further includes: acquiring fifth point cloud data of the lifting equipment within an expected range; Determining the center position of the lifting equipment based on the side door plane area and tire area of ​​the lifting equipment determined after clustering the point clouds of multiple part types of the fifth point cloud data; The region of interest of the grasped object is determined according to the center position.

7. The method according to any one of claims 1 to 6, characterized in that The method further comprises: The motion state of the spreader is determined according to the position information and position information of a previous frame of the position information.

8. A vehicle control device based on port vertical transportation target detection, characterized in that: Applied to a vehicle, the device comprises: An acquisition module, configured to acquire first point cloud data of an area above the vehicle; a first determining module configured to determine, based on the first point cloud data and a predetermined region of interest of a spreader or a grasped object in the lifting equipment, second point cloud data located in the region of interest of the spreader or grasped object in the first point cloud data; a second determining module, configured to determine position information and a grasping state of a spreader corresponding to the second point cloud data based on feature information of the second point cloud data; The control module is used to control the moving and stopping state of the vehicle according to the position information and the grasping state.

9. A vehicle, characterized in that: include: a processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 7 when executed by a processor.

Images