Boom-mounted motion-following sensing system, boom operation method, and operation machine

Abstract

A boom-mounted motion-following sensing system, an operation machine, a vehicle, and a boom operation method, wherein the boom-mounted motion-following sensing system comprises: a scene detection unit; a boom-mounted gimbal, which is arranged on a boom (21) and comprises a pitching mechanism (14), a rotation mechanism (12) and a swing mechanism (11), the scene detection unit being arranged on the swing mechanism (11); and a processing device (17), which communicates with the boom-mounted gimbal and the scene detection unit respectively, and is configured to: acquire an operation pose of the boom (21); control the movement of the pitching mechanism (14) on the basis of the operation pose of the boom (21), such that the rotation axis is adjusted to be perpendicular to the ground plane; and control the movements of the swing mechanism (11) and the rotation mechanism (12), such that the scene detection unit performs rotation scanning around the rotation axis at a preset swing angle.

Description

Arm-mounted follow-up sensing system, arm operation method and operation machine

[0001] This application claims priority to the Chinese patent application No. 202410884245.9, filed on July 03, 2024, and entitled "Arm-mounted follow-up sensing system, arm operation method and operation machine", the whole content of which is incorporated herein by reference. TECHNICAL FIELD

[0002] The present application belongs to the field of operation machines, and in particular relates to an arm-mounted follow-up sensing system, an arm operation method and an operation machine. BACKGROUND

[0003] In the operation process of the arm operation machine, the environmental information around the arm is usually needed to be acquired to ensure the safety of the operation. Taking the operation machine as a crane for example, since the hoisting operation area of the crane is large, for example, the longest arm length of a large crane can reach hundreds of meters, and the maximum width and the maximum operation radius can reach more than sixty meters. The operator is in the operation room, and the distance from the operator to the hoisting target can reach more than sixty meters. The operator's operation field of view is limited when hoisting, which further brings potential safety hazards. TECHNICAL SOLUTION

[0004] The purpose of the present application is to provide an arm-mounted follow-up sensing system, an arm operation method and an operation machine, which can realize large field of view monitoring in a large operation range.

[0005] In order to achieve the above-mentioned purpose, the first aspect of the present application provides an arm-mounted follow-up sensing system, which comprises:

[0006] a scene detection unit, configured to acquire scanning data of an operation scene;

[0007] an arm-mounted pan-tilt head, provided on an arm and comprising a tilting mechanism, a rotating mechanism and a swinging mechanism, the rotating mechanism is connected with the swinging mechanism and can drive the swinging mechanism to rotate around a rotating axis, the scene detection unit is provided on the swinging mechanism, the swinging mechanism can drive the scene detection unit to swing to adjust a swing angle between an optical axis of the scene detection unit and the rotating axis, the tilting mechanism is connected between the arm and the rotating mechanism and can drive the rotating mechanism to perform up-down tilting motion around a tilting axis; and

[0008] a processing device, in communication with the arm-mounted pan-tilt head and the scene detection unit respectively, and configured to:

[0009] acquire an operation pose of the arm;

[0010] control the tilting mechanism to act according to the operation pose of the arm, so that the rotating axis is adjusted to be perpendicular to the ground plane;

[0011] The total rotation number of the scene detection unit rotating around the rotation axis and the swing angle corresponding to each rotation cycle are determined according to the working scene size of the arm support;

[0012] The swing mechanism and the rotation mechanism are controlled to act, so that the scene detection unit rotates and scans around the rotation axis at a preset swing angle;

[0013] A working scene global map is generated according to the scanning data of the scene detection unit, the action data of the rotation mechanism and the action data of the swing mechanism, including:

[0014] The scanning data of the scene detection unit, the action data of the rotation mechanism and the action data of the swing mechanism are obtained;

[0015] The scanning data is processed and spliced in combination with the action data of the rotation mechanism and the action data of the swing mechanism, to generate the working scene global map.

[0016] In some specific embodiments, the processing device is configured to:

[0017] The working scene size of the arm support is determined according to the length of the arm support.

[0018] In some specific embodiments, the swing mechanism and the rotation mechanism are controlled to act, so that the scene detection unit rotates and scans around the rotation axis at the swing angle corresponding to the rotation cycle, until the last rotation scan is completed, including:

[0019] The current rotation cycle of the scene detection unit is determined, and the swing mechanism is controlled to act to adjust the swing angle corresponding to the current rotation cycle;

[0020] The rotation mechanism is controlled to drive the scene detection unit to rotate at a preset rotation angle interval and a preset time period interval, until the scene detection unit rotates one cycle at the current rotation cycle;

[0021] It is determined that the scene detection unit has completed the last rotation scan, and the scene detection unit is controlled to stop scanning.

[0022] In some specific embodiments, the arm-mounted cloud platform further includes a roll mechanism and a pitch mechanism, the roll mechanism being capable of driving the rotation mechanism to rotate around a roll axis, the roll axis being perpendicular to the pitch axis, before the scene detection unit rotates and scans, the processing device is further configured to:

[0023] The roll mechanism is controlled to act, so that the rotation axis can be perpendicular to the ground plane when the arm support is twisted or tilted. In some specific embodiments, the pitch mechanism, the roll mechanism, the rotation mechanism and the swing mechanism are connected in sequence; or the roll mechanism, the pitch mechanism, the rotation mechanism and the swing mechanism are connected in sequence.

[0024] In some embodiments, the arm-mounted follow-up perception system further comprises a human-machine interaction device configured to receive and display the image captured by the scene detection unit and obtain a target selection operation of a user on the image.

[0025] The processing device is in communication with the human-machine interaction device and is further configured to:

[0026] Control the rotation mechanism and the swing mechanism to move so that the work target enters the field of view of the scene detection unit.

[0027] Receive the target selection operation and determine the position of the work target in the global map of the work scene according to the target selection operation.

[0028] In some embodiments, the processing device is further configured to:

[0029] According to the current state of the arm support and the spatial position of the work target in the global map of the work scene, the work path is planned and safety calculation is performed.

[0030] In some embodiments, the processing device is further configured to:

[0031] When the arm support moves according to the work path planning, the arm-mounted gimbal is controlled to move so that the optical axis of the scene detection unit is perpendicular to the ground plane, and when the work target enters the field of view of the scene detection unit, the work target is identified and repositioned.

[0032] In some embodiments, the arm-mounted follow-up perception system further comprises a first angle sensor and a second angle sensor, the first angle sensor is arranged on the swing mechanism and is configured to detect the swing angle of the swing mechanism, and the second angle sensor is arranged on the rotation mechanism and is configured to detect the rotation angle of the rotation mechanism; and / or, the scene detection unit comprises an image acquisition component and / or a radar.

[0033] The second aspect of the present application further provides a work machine, comprising: an arm support and the above-mentioned arm-mounted follow-up perception system, and the arm-mounted gimbal is arranged on the arm support.

[0034] The third aspect of the present application further provides an arm support work method, the arm support is provided with an arm-mounted gimbal, the arm-mounted gimbal comprises a pitch mechanism, a rotation mechanism and a swing mechanism arranged on the rotation mechanism, the rotation mechanism can drive the swing mechanism to rotate around a rotation axis, the swing mechanism is provided with a scene detection unit, the swing mechanism can drive the scene detection unit to swing to adjust the swing angle between the optical axis of the scene detection unit and the rotation axis, the pitch mechanism is connected between the arm support and the rotation mechanism and can drive the rotation mechanism to perform up-down pitch movement around a pitch axis, the pitch axis is perpendicular to the luffing direction of the arm support, and the arm support work method comprises:

[0035] Obtaining the work pose of the arm support;

[0036] controlling the pitch mechanism to act according to the working posture of the arm support, so that the rotation axis is adjusted to be perpendicular to the ground plane;

[0037] determining the total number of rotation cycles and the swing angle corresponding to each rotation cycle for the scene detection unit to rotate around the rotation axis according to the working scene size of the arm support;

[0038] controlling the swing mechanism and the rotation mechanism to act, so that the scene detection unit rotates around the rotation axis at a preset swing angle for rotation scanning;

[0039] generating a working scene global map according to the scanning data of the scene detection unit, the action data of the rotation mechanism and the action data of the swing mechanism, including:

[0040] obtaining the scanning data of the scene detection unit, the action data of the rotation mechanism and the action data of the swing mechanism;

[0041] processing and splicing the scanning data in combination with the action data of the rotation mechanism and the action data of the swing mechanism to generate the working scene global map. Advantages

[0042] Through the above technical solution, the rotation mechanism is arranged on the arm support through the pitch mechanism. When the arm support is extended or the amplitude is changed, the working posture of the arm support changes. By obtaining the working posture of the arm support, the pitch mechanism can be controlled to drive the rotation mechanism to pitch according to the working posture of the arm support, so that the rotation axis can be adjusted to be perpendicular to the ground plane, thereby facilitating the scene detection unit to rotate and scan in each direction of the working area, and enabling the scene detection unit to comprehensively and effectively scan the environment below and around, which helps to reduce the control difficulty of the arm-mounted gimbal and simplify the subsequent scanning data processing steps. The scene detection unit is arranged on the rotation mechanism through the swing mechanism. The swing mechanism can drive the scene detection unit to swing around the swing axis to adjust the swing angle between the optical axis of the scene detection unit and the rotation axis. In this way, the processing device can control the arm-mounted gimbal to act so that the scene detection unit can rotate and scan at different inclination angles, thereby expanding the scanning coverage area of the scene detection unit, so as to realize large field of view monitoring in a large working range, which is conducive to eliminating safety hazards in the working process. BRIEF DESCRIPTION OF DRAWINGS

[0043] The accompanying drawings are included to provide a further understanding of the embodiments of the application, and constitute a part of the specification, and are used together with the following detailed description to explain the embodiments of the application, but do not constitute a limitation on the embodiments of the application. For those skilled in the art, other drawings can be obtained from the structures shown in the drawings without creative labor. In the drawings:

[0044] Fig. 1 shows a structural schematic diagram of an arm-mounted follow-up perception device according to an embodiment of the present application;

[0045] Fig. 2 shows a structural schematic diagram of the arm-mounted follow-up perception device in Fig. 1 from another perspective;

[0046] Fig. 3 shows a partial structural perspective view of the arm-mounted follow-up perception device in Fig. 1;

[0047] Fig. 4 shows a working schematic diagram of a crane according to an embodiment of the present application;

[0048] Fig. 5 shows a scanning state schematic diagram of the arm-mounted follow-up perception device according to the present application when the boom of the crane is not extended;

[0049] Fig. 6 shows a scanning state schematic diagram of the arm-mounted follow-up perception device according to the present application when the boom of the crane is extended;

[0050] Fig. 7 shows a scanning state schematic diagram of the arm-mounted follow-up perception device according to the present application when the optical axis of the scene detection unit is inclined to the rotation axis;

[0051] Fig. 8 shows a control logic flowchart of the arm-mounted follow-up perception device according to an embodiment of the present application.

[0052] Reference signs: 1, arm-mounted follow-up perception device; 11, swing mechanism; 111, swing driving member; 112, swing support; 12, rotation mechanism; 121, rotation driving member; 122, rotation support; 13, horizontal roll mechanism; 131, horizontal roll driving member; 132, horizontal roll support; 14, pitch mechanism; 141, pitch driving member; 142, pitch support; 151, laser radar; 152, image acquisition component; 17, processing device; 2, crane; 21, boom; 22, human-machine interaction device. DETAILED DESCRIPTION

[0053] The specific embodiments of the present application are described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely intended to illustrate and explain the present application, and are not intended to limit the present application.

[0054] To improve the operation safety and intelligence of large-scale operation machinery, as shown in FIG. 1 and FIG. 2, the first aspect of the present application provides an arm-mounted cloud platform, which can be installed on the boom 21 of the operation machinery and used to install a scene detection unit for obtaining scanning data of the operation scene. In this way, it is beneficial to realize online reconstruction of hoisting operation scene, large space and long distance target recognition and positioning, and dynamic detection of spatial obstacles, thereby eliminating the operation visual field limitation of the operator, greatly reducing the operation difficulty of the operation machinery, and also making the operation machinery more suitable for automatic operation or unmanned operation in dangerous operation scene or repeated operation demand.

[0055] The arm-mounted cloud platform of the present application includes a pitching mechanism 14, a rotating mechanism 12 and a swinging mechanism 11, the rotating mechanism 12 is connected with the swinging mechanism 11, and the scene detection unit can be installed on the swinging mechanism 11. The rotating mechanism 12 can drive the swinging mechanism 11 to rotate around the rotation axis to drive the scene detection unit to perform 360° rotation scanning, and the swinging mechanism 11 can drive the scene detection unit to swing around the swing axis to adjust the swing angle between the optical axis of the scene detection unit and the rotation axis, and the swing axis can be parallel to the ground plane.

[0056] In the embodiment of the present application, the operation machinery can be an operation device with a boom, such as a crane, a pump truck, a high-altitude spraying fire truck, a high-altitude operation machinery or a boom-type robot, etc. Taking the operation machinery as a crane as an example, since the maximum amplitude and the maximum operation radius of the large crane operation can reach more than sixty meters, the minimum amplitude and the minimum operation radius can be as low as a few meters, and the difference between the maximum amplitude and the minimum amplitude is large, and the difference between the maximum operation radius and the minimum operation radius is also large, up to dozens of meters. Therefore, the arm-mounted cloud platform of the present application can also realize large field of view monitoring and three-dimensional reconstruction in a large operation range for the scene detection unit with a small field of view angle and relatively low equipment cost, while effectively reducing the manufacturing cost of the large crane. In addition, since the arm-mounted cloud platform is additionally provided with the swinging mechanism 11 capable of adjusting the swing angle during scanning, the arm-mounted following perception device 1 using the scene detection unit with a small field of view angle can also be applied to operation machinery of different specifications and models, especially large operation machinery, and the universality is greatly improved.

[0057] In addition, when the arm support 21 is raised or luffed, the working position of the arm support 21 changes, and the position of the rotating mechanism 12 also changes, resulting in a change in the vertical relationship between the rotating axis and the ground plane. To this end, as shown in FIGS. 1 and 2, the arm-mounted gimbal further includes a pitching mechanism 14, which can drive the rotating mechanism 12 to perform up-and-down pitching movement about a pitching axis when the arm support 21 is luffed. In this way, when the arm support 21 is raised (luffed), the rotating axis of the scene detection unit can be adjusted by the pitching mechanism 14 to ensure verticality with the ground plane, thereby facilitating the rotating scanning of the scene detection unit to each direction of the working area, while enabling comprehensive and effective scanning of the environment below and around the scene detection unit, which helps to reduce the control difficulty of the arm-mounted gimbal and simplify the subsequent scanning data processing steps.

[0058] The pitching axis is perpendicular to the luffing direction of the arm support 21, and the luffing direction of the arm support 21 is the extension direction of the projection of the central axis of the arm support 21 on the ground plane.

[0059] To better embody the technical effects brought by the embodiments of the present application, the function implementation mode of the arm-mounted gimbal will be described below in combination with some application examples. In the arm-mounted gimbal of the present application, the swinging mechanism 11, the rotating mechanism 12, and the pitching mechanism 14 can belong to two different control modules, the swinging mechanism 11 and the rotating mechanism 12 being a rotating scanning working control module, and the pitching mechanism 14 being a position adjustment control module. By providing the pitching mechanism 14, dynamic and real-time control of the action of the pitching mechanism 14 can be achieved when the working position of the arm support 21 changes, thereby changing the position of the rotating mechanism 12 so that the rotating axis always remains perpendicular to the ground plane. As for the rotating scanning working module of the swinging mechanism 11 and the rotating mechanism 12, the scene detection unit is mounted on the swinging mechanism 11 and has a direction calibration relationship, and the swinging mechanism 11 and the rotating mechanism 12 are independently controllable for scanning the working scene in steps when rotating scanning, which, according to the calibration relationship of the scene detection unit, helps to improve the processing efficiency of the scanning data. For example, when the scanning data is applied to scene reconstruction, the efficiency of scene reconstruction and the real-time performance of the algorithm can be improved; or when the scanning data is applied to scene obstacle detection, the detection efficiency and positioning accuracy of obstacles can be effectively improved. Of course, it is easy to understand that the above is some illustrative examples of some applicable modes of scanning data, and the scanning data can also be used for other purposes in actual applications, such as pedestrian detection, etc., which will not be exemplified one by one here.

[0060] Further, taking the application of scan data to scene reconstruction as an example, in order to improve the efficiency of scene reconstruction and the real-time performance of the algorithm, the control coordinate system of the arm-mounted follow-up sensing system can be provided with two. Specifically, the pitch axis of the pitch mechanism 14 is a dynamic change axis, and the pitch axis of the pitch mechanism 14 is arranged in one of the independent control coordinate systems; and the swing axis of the swing mechanism 11 and the rotation axis of the rotation mechanism 12 driving the rotation of the scene detection unit are controllable and adjustable axes according to the working condition operation requirements, and the swing axis of the swing mechanism 11 and the rotation axis of the rotation mechanism 12 driving the rotation of the scene detection unit are arranged in the other coordinate system.

[0061] In addition, unlike the existing ordinary three-axis gimbal, since the arm-mounted gimbal of the present application is provided with the pitch mechanism 14 between the rotation mechanism 12 and the arm support 21, the arm-mounted gimbal can be installed on the arm support 21 without limitation of orientation, and when the scene detection unit rotates and scans, the rotation axis can be automatically controlled to be perpendicular to the horizontal plane, thereby enabling comprehensive and effective scanning of the environment below and around the scene detection unit, and facilitating the subsequent scanning data processing step of the processing device. Obviously, for the existing ordinary three-axis gimbal, since the camera is arranged on the pitch structure or the roll structure, the pitch structure or the roll structure is arranged on the rotation structure, and the rotation structure is arranged on the mounting platform, the ordinary three-axis gimbal cannot guarantee that the camera can perform comprehensive and effective scanning of the environment below and around. If the existing ordinary three-axis gimbal is installed on the arm support, once the arm support is tilted, the rotation axis of the camera is also tilted in a completely decoupled manner, resulting in that some orientations cannot be effectively scanned, and the shape of the scanning area will also change, thereby causing the subsequent processing of the scanning data to be very complex. Moreover, if the existing ordinary three-axis gimbal is used to obtain the full-range comprehensive scanning area in the present case, at least two degrees of freedom coupling motion is required, and the difficulty of gimbal control will also increase significantly.

[0062] In addition, as shown in FIG. 3, the scene detection unit includes detection members such as a laser radar 151 and / or an image acquisition component 152, and the field of view angles of different models and specifications of the laser radar 151 and the image acquisition component 152 are different. By providing the swing mechanism 11, the swing angle of the scene detection unit can be flexibly adjusted, and the versatility of the arm-mounted gimbal is also greatly improved. Among them, the scene detection unit rotates and scans the working area, which can facilitate the subsequent generation of a global map of the working scene, and also facilitate monitoring, tracking and obstacle identification during operation.

[0063] Optionally, the structures of the rotating mechanism 12, the swinging mechanism 11 and the pitching mechanism 14 can be various. The swinging mechanism 11 can include a swinging driving member 111 and a swinging support 112, and the rotating mechanism 12 can include a rotating driving member 121 and a rotating support 122. The swinging driving member 111 is arranged on the swinging support 112, and a driving shaft of the swinging driving member 111 can be drivingly connected with the scene detection unit to drive the scene detection unit to swing. The rotating driving member 121 is arranged on the rotating support 122, and a driving shaft of the rotating driving member 121 is drivingly connected with the swinging support 112 to drive the swinging mechanism 11 to rotate. The swinging driving member 111 can include a servo motor, a stepping motor or an electric cylinder, etc., the swinging support 112 can be a U-shaped support or an irregularly shaped support, etc., and the scene detection unit can be directly connected with the driving shaft of the swinging driving member 111 or indirectly connected with the driving shaft of the swinging driving member 111 through a connecting frame, an intermediate gear assembly or a connecting rod assembly. Similarly, the rotating driving member 121 can include a servo motor or a stepping motor, etc., the rotating support 122 can be an L-shaped support or an irregularly shaped support, etc., and the swinging support 112 can be directly connected with the driving shaft of the rotating driving member 121 or indirectly connected with the driving shaft of the rotating driving member 121 through a connecting frame or an intermediate gear assembly. The pitching mechanism 14 can include a pitching driving member 141 and a pitching support 142, the pitching driving member 141 can include a servo motor, a stepping motor or an electric cylinder, etc., and the pitching support 142 can be a U-shaped support or an irregularly shaped support, etc.

[0064] Since the scene detection unit needs to be rotated for scanning, the rotation axis needs to be perpendicular to the ground plane for subsequent online reconstruction of a three-dimensional scene map. However, in actual operation, factors such as swinging of the load on the arm support 21 or tilting of the vehicle body can cause the end of the arm support 21 to twist. The twisting of the arm support 21 can cause the rotating mechanism 12 to tilt, and the perpendicular relationship between the rotation axis and the ground plane is also changed.

[0065] Therefore, in some specific embodiments, as shown in FIGS. 1 and 2, the arm load holder of the present application further includes a roll mechanism 13 connected with the pitching mechanism 14, and the roll mechanism 13 and the pitching mechanism 14 are connected between the arm support 21 and the rotating mechanism 12. The roll mechanism 13 can drive the rotating mechanism 12 to rotate around a roll axis when the arm support 21 twists, the roll axis is parallel to the luffing direction of the arm support 21 and perpendicular to the pitching axis. In this way, when the arm support 21 twists, the rotation axis of the scene detection unit can be adjusted by the roll mechanism 13 to ensure that it is perpendicular to the ground plane. The roll mechanism 13 can include a roll driving member 131 and a roll support 132, the roll driving member 131 can include a servo motor, a stepping motor or an electric cylinder, etc., and the roll support 132 can be a U-shaped support or an irregularly shaped support, etc.

[0066] Further, the roll mechanism 13 and the pitch mechanism 14 can belong to the same control module, that is, the roll mechanism 13 and the pitch mechanism 14 belong to the pose adjustment control module. By setting the roll mechanism 13 and the pitch mechanism 14, the action of the roll mechanism 13 and the pitch mechanism 14 can be dynamically and real-timely controlled when the working pose of the boom 21 changes, so as to change the pose of the rotating mechanism 12, and the rotating axis is always kept perpendicular to the ground.

[0067] Further, taking the application of the scanning data to the scene reconstruction as an example, in order to improve the efficiency of the scene reconstruction and the real-time performance of the algorithm, when the arm-mounted follow-up sensing system of the present application is provided with two control coordinate systems, the roll axis of the roll mechanism 13 is also a dynamically changing axis, and the pitch axis of the pitch mechanism 14 and the roll axis of the roll mechanism 13 are arranged in the same control coordinate system.

[0068] Since the arm-mounted holder of the present application is provided with the roll mechanism 13 and the pitch mechanism 14 between the rotating mechanism 13 and the boom 21, when the boom 21 is luffing and tilting, the environment below and around the scene detection unit can also be comprehensively and effectively scanned, which is helpful to simplify the subsequent scanning data processing steps and reduce the control difficulty of the arm-mounted holder.

[0069] Optionally, the adjustment range of the swing angle of the swing mechanism 11 is -32°-32°. The adjustment range of the pitch angle of the pitch mechanism 14 is -10°-90°. The adjustment range of the roll angle of the roll mechanism 13 is -15°-15°. The control range of the rotating angle of the rotating mechanism 12 is positive rotation -20°-360° or negative rotation -20°-360°.

[0070] In some specific embodiments, as shown in FIGS. 1 and 2, the swing mechanism 11, the rotating mechanism 12, the roll mechanism 13 and the pitch mechanism 14 are sequentially connected. Specifically, the swing driving member 111 can be arranged on the swing bracket 112, and the driving shaft of the swing driving member 111 can be drivingly connected with the scene detection unit. The rotating driving member 121 is arranged on the rotating bracket 122, and the driving shaft of the rotating driving member 121 is drivingly connected with the swing bracket 112. The roll driving member 131 is arranged on the roll bracket 132, and the driving shaft of the roll driving member 131 is drivingly connected with the rotating bracket 122. The pitch driving member 141 is arranged on the pitch bracket 142, and the driving shaft of the pitch driving member 141 is drivingly connected with the roll bracket 132. The rotating bracket 122 can be directly connected with the driving shaft of the roll driving member 131, or indirectly connected with the driving shaft of the roll driving member 131 through a connecting frame body, an intermediate gear assembly, etc. The roll bracket 132 can be directly connected with the driving shaft of the pitch driving member 141, or indirectly connected with the driving shaft of the pitch driving member 141 through a connecting frame body, an intermediate gear assembly, etc.

[0071] Alternatively, in some embodiments, the swing mechanism 11, the rotation mechanism 12, the tilt mechanism 14 and the roll mechanism 13 are connected in sequence (not shown in the figure). Specifically, the swing driving member 111 is arranged on the swing support 112 and the driving shaft of the swing driving member 111 is drivingly connected with the scene detection unit, the rotation driving member 121 is arranged on the rotation support 122 and the driving shaft of the rotation driving member 121 is drivingly connected with the swing support 112, the tilt driving member 141 is arranged on the tilt support 142 and the driving shaft of the tilt driving member 141 is drivingly connected with the rotation support 122, and the roll driving member 131 is arranged on the roll support 132 and the driving shaft of the roll driving member 131 is drivingly connected with the tilt support 142. The rotation support 122 can be directly connected with the driving shaft of the tilt driving member 141 or indirectly connected with the driving shaft of the tilt driving member 141 through a connecting frame, an intermediate gear assembly or the like. The tilt support 142 can be directly connected with the driving shaft of the roll driving member 131 or indirectly connected with the driving shaft of the roll driving member 131 through a connecting frame, an intermediate gear assembly or the like.

[0072] Optionally, the arm-mounted holder further comprises a holder fixing support (not shown in the figure) which is connectable with the arm support 21, and the rotation mechanism 12 is arranged on the holder fixing support through the tilt mechanism 14 and the roll mechanism 13. When the swing mechanism 11, the rotation mechanism 12, the roll mechanism 13 and the tilt mechanism 14 are connected in sequence, the tilt support 142 is connected with the holder fixing support; when the swing mechanism 11, the rotation mechanism 12, the tilt mechanism 14 and the roll mechanism 13 are connected in sequence, the roll support 132 is connected with the holder fixing support.

[0073] Optionally, the holder fixing support of the arm-mounted holder is L-shaped and provided with reinforcing ribs, the tilt support 142 is formed as two vertical plates which are arranged in space and extend downward from the horizontal top plate of the holder fixing support, and the tilt driving member 141 is arranged on the tilt support 142 and located outside the tilt support 142. The roll support 132 is cylindrical and rotatably arranged between the two vertical plates of the tilt support 142, and the tilt driving member 141 can drive the roll support 132 to perform tilt action about the tilt axis. The rotation support 122 is U-shaped and located in the roll support 132, the roll driving member 131 is arranged on the roll support 132 and can drive the rotation support 122 to perform roll swing about the roll axis, and the tilt axis is perpendicular to the roll axis. The swing support 112 is U-shaped and located in the rotation support 122, the rotation driving member 121 is arranged on the rotation support 122 and can drive the swing support 112 to rotate about the rotation axis. The scene detection unit is arranged in the swing support 112, and the swing driving member 111 is arranged outside the swing support 112 and can drive the scene detection unit to swing about the swing axis.

[0074] Alternatively, the gimbal fixing support of the arm-mounted gimbal is L-shaped and provided with reinforcing ribs, one outer side panel of the gimbal fixing support is used to be connected with the arm support 21, the other outer side panel is used to mount the processing device 17, and the pitching support 142 is arranged between the two inner side panels. The roll support 132, the rotating support 122 and the swinging support 112 are all L-shaped. The arm-mounted gimbal is further provided with a scene detection unit mounting rack, the scene detection unit mounting rack is used to mount the scene detection unit and is drivingly connected with the swinging driving member 111, and the swinging driving member 111 can drive the scene detection unit mounting rack to swing around the swinging axis.

[0075] The second aspect of the present application further provides an arm-mounted follow-up sensing device 1, which comprises the above-mentioned arm-mounted gimbal and a scene detection unit, the scene detection unit is arranged on the arm-mounted gimbal and is used to acquire scanning data of a working scene. Since the arm-mounted follow-up sensing device 1 comprises the above-mentioned arm-mounted gimbal, it also has all the technical effects brought by the arm-mounted gimbal, and thus will not be repeated here.

[0076] The third aspect of the present application further provides an arm-mounted follow-up sensing system, which comprises the above-mentioned arm-mounted follow-up sensing device 1. Since the arm-mounted follow-up sensing system comprises the above-mentioned arm-mounted follow-up sensing device 1, it also has all the technical effects brought by the arm-mounted follow-up sensing device 1, and thus will not be repeated here.

[0077] Optionally, the arm-mounted follow-up sensing device 1 can further comprise a first angle sensor, a second angle sensor, a third angle sensor and a fourth angle sensor (not shown in the figure). The first angle sensor is arranged on the swinging mechanism 11 and is used to detect a swinging angle of the swinging mechanism 11, the second angle sensor is arranged on the rotating mechanism 12 and is used to detect a rotating angle of the rotating mechanism 12, the third angle sensor is arranged on the roll mechanism 13 and is used to detect a first inclination angle value in a direction perpendicular to the luffing direction of the arm support 21, and the fourth angle sensor is arranged on the pitching mechanism 14 and is used to detect a second inclination angle value in the luffing direction of the arm support 21.

[0078] Among them, the first angle sensor, the second angle sensor, the third angle sensor and the fourth angle sensor can be independently arranged angle sensors, or can be partially integrated in an IMU attitude detection device. The IMU attitude detection device can have automatic and manual control modes. The automatic control mode is mainly controlled by the processing device receiving the IMU attitude angle; the manual control mode is mainly controlled by the user collecting the IMU attitude angle and calling the SDK to control the angle rotation of the axis of the arm-mounted follow-up sensing device 1.

[0079] Optionally, the scene detection unit can comprise an image acquisition component 152, or the scene detection unit can comprise a radar, or the scene detection unit can comprise both the image acquisition component 152 and the radar. The radar can be a laser radar 151, and the scene detection unit of the present application can adopt a multi-line laser radar with less wiring, which can also obtain a large scanning space and reduce manufacturing costs. The image acquisition component 152 can be a camera or a camera head, and the camera can be, for example, a variable-focus monocular camera. The optical axis of the scene detection unit can be the central axis of the field of view of the image acquisition component 152 or the radar, and the central axes of the fields of view of the image acquisition component 152 and the radar are parallel.

[0080] Specifically, when the arm-mounted follow-up sensing device 1 is installed at the arm tip of the arm support 21, the detection range of the radar is a conical region with the sling / hook as the axis. By controlling the radar scanning, a three-dimensional scene map within the radar scanning range can be reconstructed in real time. Subsequently, the coarse positioning of the work target position can be performed in the form of point selection in the three-dimensional scene map, and the value can be transmitted to the camera for accurate positioning (secondary positioning) and the camera or camera head is controlled to follow. The radar can detect the spatial three-dimensional coordinates and azimuth angle of all objects (including target points and obstacles) within the conical region with the sling / hook as the axis in real time, and distinguish target objects and obstacles by fusing images. In addition, the arm support 21 can be automatically avoided by combining a dynamic path correction algorithm according to the motion trend of the hook / sling load.

[0081] In addition, according to the coarse positioning value of the three-dimensional scene map, the arm-mounted follow-up sensing device 1 can control the arm-mounted gimbal to automatically follow the camera to the work target position, extract the high-definition image region block of the work target through focal length stretching, and perform work target image block feature extraction. According to the work target coarse positioning value, the rotation / amplitude action of the arm support 21 is calculated, and then the automatic hoist is started to control the gimbal to make the camera lens vertically downward and track the hook and the vertical projection point of the hook. When the field of view of the camera overlaps with the accurate positioning coarse determination region or the work target enters the field of view of the camera, the camera is started to detect the target feature and perform accurate positioning.

[0082] In some embodiments, the arm-mounted follow-up sensing system can further comprise a processing device 17 for controlling the arm-mounted gimbal to rotate the work area at a preset swing angle to enable the scene detection unit to perform a rotational scan. The processing device 17 is in communication with the arm-mounted gimbal and the scene detection unit, and can perform data interaction communication in the form of a network port. Specifically, the processing device can be configured to:

[0083] acquire the work pose of the arm support 21;

[0084] control the pitch mechanism 14 to act according to the work pose of the arm support 21, so that the rotation axis is adjusted to be perpendicular to the ground plane;

[0085] The swing mechanism 11 and the rotation mechanism 12 are controlled to operate so that the scene detection unit performs a rotational scan around the rotation axis at a preset swing angle.

[0086] Due to the small field of view angle of the camera and the laser radar 151, the light axis swing direction of the camera and the laser radar 151 is adjusted in real time by the swing mechanism 11 and the rotation mechanism 12, so as to expand the detectable range of the camera and the laser radar 151. The processing device can automatically obtain the data of the working pose of the arm support 21 by communicating with the motion control module of the arm support 21, or the data can also be manually input into the processing device.

[0087] Optionally, the processing device 17 is further configured to:

[0088] Generate a global map of the working scene according to the scan data of the scene detection unit, the action data of the rotation mechanism 12, and the action data of the swing mechanism 11.

[0089] When the arm support 21 is stationary, the rotation mechanism 12 is controlled to perform a rotation action, and the swing mechanism 11 is controlled to gradually adjust the scan area of the camera and the laser radar, so as to obtain actual scene point cloud data and images of the entire working area, realize online reconstruction of the working scene, and generate a global map of the working scene.

[0090] Optionally, the processing device 17 is further configured to:

[0091] Determine the total number of rotations of the scene detection unit around the rotation axis and the swing angle corresponding to each rotation according to the size of the working scene of the arm support.

[0092] In the determination of the total number of rotations of the scene detection unit around the rotation axis and the swing angle corresponding to each rotation according to the size of the working scene of the arm support, the processing device determines the maximum swing angle by combining the size of the working scene of the arm support and the field of view angle of the scene detection unit, and then determines the rotation scan parameters of the scene detection unit by combining the maximum swing angle and the field of view angle of the scene detection unit. The rotation scan parameters of the scene detection unit include the total number of rotations of the scene detection unit around the rotation axis and the swing angle corresponding to each rotation.

[0093] In addition, in order to balance the operation efficiency and the equipment cost, the total number of rotations of the scene detection unit around the rotation axis can be set to be less than or equal to 5. As shown in FIG. 6, when the optical axis of the scene detection unit is downwardly perpendicular to the ground, the rotational scanning of the scene detection unit can obtain scanning data of a conical scanning region with the vertical intersection point of the optical axis on the ground as the center; as shown in FIG. 7, when the swing angle is adjusted so that the optical axis of the scene detection unit is inclined to the ground, the rotational scanning of the scene detection unit can obtain scanning data of a double-cone annular scanning region arranged around the rotation axis; and the scanning data of the scanning regions of all the rotation times at different swing angles can be spliced to obtain the scanning data of the entire operation region.

[0094] In addition, for the step of generating the operation scene global map according to the scanning data of the scene detection unit, the action data of the rotating mechanism 12 and the action data of the swing mechanism 11, the step can include:

[0095] obtaining the scanning data of the scene detection unit, the action data of the rotating mechanism 12 and the action data of the swing mechanism 11;

[0096] processing and splicing the scanning data in combination with the action data of the rotating mechanism 12 and the action data of the swing mechanism 11 to generate the operation scene global map.

[0097] The scanning data of the scene detection unit includes point cloud data obtained by the radar, and the point cloud data can be converted and spliced in coordinates to construct the operation scene global map of the entire operation region. After the construction of the operation scene global map is completed, the rotating mechanism 12 and the swing mechanism 11 are controlled to be returned to normal so that the optical axis of the scene detection unit is perpendicular to the ground, and whether the operation scene global map needs to be reconstructed can be determined according to the construction quality of the operation scene global map. If not, the arm-mounted cloud platform can be directly controlled to move so that the next operation target enters the field of view of the scene detection unit, and the operation target is positioned for subsequent path planning and hoisting operation; if so, the arm-mounted follow-up sensing device 1 can be controlled to start scanning operation again.

[0098] In addition, for the step of controlling the swing mechanism 11 and the rotating mechanism 12 to move so that the scene detection unit rotates around the rotation axis at a preset swing angle, the step can include:

[0099] determining the current rotation time of the scene detection unit, and controlling the swing mechanism 11 to move to adjust the swing angle corresponding to the current rotation time;

[0100] controlling the rotating mechanism 12 to drive the scene detection unit to rotate at a preset rotation angle interval and a preset time period interval until the scene detection unit rotates one revolution at the current rotation time;

[0101] When it is determined that the scene detection unit has completed the last rotation scan, the control unit controls the scene detection unit to stop scanning.

[0102] In the sub-step of controlling the rotation mechanism 12 to drive the scene detection unit to rotate at preset rotation angle intervals and preset time period intervals until the scene detection unit rotates a full circle in the current rotation cycle, the processing device controls the rotation mechanism 12 to rotate at preset rotation angle intervals every preset time period, and then acquires the scan data of the scene detection unit, until the rotation mechanism 12 drives the scene detection unit to rotate 360° in the current rotation cycle to complete a full scan. For example, the preset rotation angle can be 15° to 30°, and can be preferably 20°; the preset time period can be 0.5 seconds to 3 seconds, and can be preferably 1 second to 2 seconds. Specifically, in any rotation cycle, the rotation mechanism 12 can be controlled to pause for a preset time period after rotating a preset rotation angle, and then continue to rotate until the 360° rotation scan of the rotation cycle is completed. In this way, the point cloud data can be more dense, thereby more conducive to improving the construction quality of the global map of the work scene.

[0103] Specifically, as shown in FIG. 4, the axis parallel to the luffing direction of the arm support 21 is the Y axis, the axis perpendicular to the luffing direction of the arm support 21 and parallel to the ground plane is the X axis, and the axis perpendicular to the ground plane is the Z axis. When the arm support 21 is extended or luffed and the arm support 21 is twisted, the inclination angles of the laser radar and the camera in the X axis direction and the Y axis direction change. By acquiring the inclination angle values of the third angle sensor and the fourth angle sensor in the X and Y axis directions in real time, and then controlling the action of the arm-mounted gimbal, the adjustment of the optical axis angle of the laser radar and the camera is realized.

[0104] As shown in FIG. 5, in the initial state of the arm support 21, the arm support 21 is fully retracted (not unfolded), the angle sensors of the arm-mounted gimbal remain parallel to the ground plane, and the initial state angles of the third angle sensor and the fourth angle sensor are both 0°, and the Y axis remains parallel to the luffing direction.

[0105] When the work machine starts the gimbal self-adjustment control thread and enters the arm unfolding step, the first inclination angle value and the second inclination angle value of the third angle sensor and the fourth angle sensor in the X and Y axis directions and the height of the arm tip to the ground are read in real time, the control angle is calculated according to the initial attitude angle of the arm-mounted gimbal, and the self-adjustment of the pitch angle, the roll angle and the camera focal length is realized through the SDK of the arm-mounted gimbal, so that the central axis of the field of view of the radar and the camera is always perpendicular to the ground plane.

[0106] The central axis of the field of view of the laser radar and the camera is perpendicular to the ground control method: when the arm support 21 changes in length (in the case of luffing or retraction / unfolding), the camera lens angle changes with the inclination, at the same time, the third angle sensor and the fourth angle sensor detect the first inclination angle value ∝ x and the second inclination angle value ∝y , a x is a first tilt angle value of the roll mechanism 13 in the X-axis direction when the included angle between the arm support and the ground surface is a y is a second tilt angle value of the pitch mechanism 14 in the Y-axis direction when the included angle between the arm support and the ground surface is a. The processing device obtains the angle values in real time from the vehicle-mounted controller through the CAN bus. When the arm support 21 is in the initial state (not lifted), a x = 0°, a y ≠ 0°. When the arm support 21 is extended (the arm is extended and the amplitude is changed), and only a y changes, the control amount of the rotation angle P of the arm-mounted gimbal and the control amount of the pitch angle T are P = 0° and T = a y , respectively. When the arm support 21 is twisted in the X-axis direction, a x ≠ 0°, and the control amount of the roll angle R of the arm-mounted gimbal is R = a x . Thus, the control of the vertical downward axis in the field of view of the radar and the camera can be completed, and the image plane coordinate system of the camera and the radar is consistent with the axis direction of the coordinate system OXYZ set by the arm support. After the arm support 21 is fully extended, the self-adjustment control thread of the arm-mounted gimbal is exited, and the gimbal control mode is switched, so that the arm-mounted gimbal is automatically rotated, and the radar is controlled to scan the scene to obtain three-dimensional point cloud data of the working scene.

[0107] When the radar is controlled to scan the scene to obtain three-dimensional point cloud data of the working scene, when the arm support 21 is stationary, the central axis of the field of view of the radar and the camera is perpendicular to the ground surface, and the radar point cloud data and the camera image data of the scene at this moment are obtained, as shown in FIG. 6. The arm-mounted gimbal is controlled to move, so that the swing angle a of the camera / radar in the i-th rotation is a preset swing angle a (set according to the size of the scanned scene), and the gimbal is controlled to rotate horizontally by 360° at an equal angle interval, and the interval time period is T1, as shown in FIG. 7. Then, the obtained point cloud data is converted, and the hoisting scene map of the entire working area can be obtained.

[0108] Alternatively, the working scene size of the arm support can be determined according to the length of the arm support 21. The processing device can obtain the working scene size of the arm support by automatically obtaining the length data of the arm support 21 and converting according to the built-in algorithm; or the working scene size of the arm support can also be determined according to manual measurement and input into the processing device, and the processing device converts according to the built-in algorithm to obtain the working scene size of the arm support.

[0109] Alternatively, the processing device 17 is further configured to:

[0110] When the arm support 21 is extended or the amplitude is changed, the swing mechanism 11, the roll mechanism 13, and the pitch mechanism 14 are controlled to move so that the optical axis of the scene detection unit is perpendicular to the ground surface.

[0111] The fourth aspect of the present application also provides a working machine, which comprises an arm support 21 and the arm load follow-up sensing system described above, and the arm load follow-up sensing device 1 of the arm load follow-up sensing system is arranged on the arm support 21. Since the working machine comprises the arm load follow-up sensing system described above, it also has all the technical effects brought by the arm load follow-up sensing system, and thus will not be repeated here.

[0112] The working machine can be a crane, such as a truck crane, a crawler crane, or a boom tower crane, or can be a pump truck, an aerial work vehicle, a fire truck, an arm support working robot, or other working machines. The power of the working machine can be provided by a new energy battery (such as a lithium battery or a hydrogen fuel cell), a fuel engine, or a hybrid power device, or can also be provided by a city power supply, etc., which is not limited here.

[0113] Optionally, the working machine can be provided with two arm load follow-up sensing devices 1, wherein the scene detection unit of one arm load follow-up sensing device 1 comprises a camera or a camera head, and the scene detection unit of the other arm load follow-up sensing device 1 comprises a radar. Alternatively, the working machine can be provided with only one arm load follow-up sensing device 1, and the scene detection unit of the arm load follow-up sensing device 1 comprises a radar and an image acquisition component 152, i.e., the radar and the image acquisition component 152 share one arm load holder, so that the structure is more simple and compact, and the equipment cost is lower. The image acquisition component 152 can be a camera or a camera head.

[0114] Optionally, as shown in FIG. 4, the arm load follow-up sensing device 1 is arranged on the arm tip of the arm support 21.

[0115] According to the structural characteristics of the top end of the arm support 21, the movement form (amplitude change, rotation, hoisting) of the arm support 21, the tilting of the vehicle body, the torsion of the arm support 21, etc., the arm load follow-up sensing device has changes in three axial directions of the rotation angle, the pitch angle, and the roll angle. Therefore, the arm load holder of the present application is a four-axis holder with a swing mechanism 11, a rotation mechanism 12, a roll mechanism 13, and a pitch mechanism 14, and the laser radar 151 and the camera are integrally arranged on the arm load holder. When the arm support 21 is extended or the amplitude is changed, the processing device can control the self-adjustment of the posture of the radar and the camera by collecting the angle values of each angle sensor and controlling the actions of the swing mechanism 11, the roll mechanism 13, and the pitch mechanism 14, so that the optical axis of the scene detection unit is perpendicular to the ground. When the arm support 21 moves, it ensures that the optical axis of the scene detection unit is always perpendicular to the ground, which can monitor the obstacles around the load and the hook in real time and perform obstacle avoidance, and use the camera to accurately identify and detect the position of the target of the working machine, so as to provide support for the accurate positioning of intelligent hoisting according to the current posture of the arm support 21 of the working machine and the deviation of the target position.

[0116] Optionally, as shown in FIG. 4, the working machine further comprises a human-machine interaction device 22 for receiving and displaying the image collected by the scene detection unit and obtaining a target selection operation of the user for the image. The processing device is in communication with the human-machine interaction device 22 and is further configured to:

[0117] controlling the rotation mechanism 12 and the swing mechanism 11 to act so that the work target enters the field of view collected by the scene detection unit;

[0118] receiving the target selection operation and determining the position of the work target in the global map of the work scene according to the target selection operation.

[0119] Wherein, the human-machine interaction device 22 can include a display screen which can display the image collected by the scene detection unit, and the display screen can be a touch display screen, so that the user can perform the target selection operation by clicking the position of the work target in the panoramic image. It can be understood that the target selection operation can also be realized by other means, for example, the position of the work target in the image can also be selected through a peripheral device connected to the display screen. The processing device 17 is in communication with the human-machine interaction device 22 and has an image processing function, which can receive the collected image from the scene detection unit and transmit it to the human-machine interaction device 22 for display, and receive the target selection operation of the user for the image, so as to determine the position of the work target in the global map of the work scene according to the target selection operation. In this way, the operator of the working machine can intuitively view the actual lifting scene through the human-machine interaction device 22, thereby reducing the need for a command personnel to coordinate the command and reducing labor costs. And by automatically determining the position of the work target according to the target selection operation of the user for the image, automatic positioning of the working machine can be realized, thereby reducing the operation intensity and proficiency of the operator and improving work efficiency.

[0120] Further, the processing device 17 can be further configured to:

[0121] According to the current state of the boom 21 and the spatial position of the work target in the global map of the work scene, the work path is planned and safety calculation is performed.

[0122] Further, after the intelligent lifting function is turned on, the boom 21 moves according to the work path planning, and when the boom 21 moves close to the work target, the processing device can be further configured to:

[0123] controlling the arm-mounted cloud platform to act so that the optical axis of the scene detection unit is perpendicular to the ground plane, and when the work target enters the field of view collected by the scene detection unit, identifying and repositioning the work target.

[0124] Wherein, the repositioning is accurate positioning, which is conducive to realizing the automation and intelligent operation of the working machine.

[0125] Further, when the hook approaches the operation target, at this time, the distance between the hook and the operation target is within the set threshold range, it can be considered that this time the follow-up sensing operation is completed, and then the next follow-up sensing operation can be entered according to the working condition. When the next follow-up sensing operation starts, it can be judged whether it is necessary to rebuild the global map of the operation scene. If it is necessary to rebuild the global map of the operation scene, the arm-mounted follow-up sensing device 1 can be controlled to start scanning the operation again after the boom is unfolded; if it is not necessary to rebuild the global map of the operation scene, the next operation target can be directly controlled to enter the field of view of the scene detection unit by the action of the arm-mounted gimbal, and the operation target is positioned to perform subsequent path planning and hoisting operation.

[0126] It should be noted that the structural principles of the boom 21, the radar, the camera, the swing mechanism 11, the rotation mechanism 12, the roll mechanism 13, the pitch mechanism 14 and other component mechanisms, and the method principles of coordinate conversion and splicing of point cloud data are well known to those skilled in the art, and do not belong to the core improvement part of the present application, and thus will not be described here.

[0127] The fifth aspect of the present application also provides a vehicle, which comprises a vehicle body, a boom 21 and the above-mentioned arm-mounted follow-up sensing system, the boom 21 is arranged on the vehicle body, and the arm-mounted gimbal is arranged on the boom 21. Since the vehicle comprises the above-mentioned arm-mounted follow-up sensing system, it also has all the technical effects brought by the arm-mounted follow-up sensing system, and thus will not be repeated here. The vehicle can be a pure electric vehicle, a range-extended electric vehicle, a hybrid vehicle, a fuel cell electric vehicle, a hydrogen engine vehicle and the like.

[0128] The sixth aspect of the present application also provides an arm boom operation method, as shown in FIGS. 1 and 2, the arm boom 21 is provided with an arm-mounted gimbal, the arm-mounted gimbal comprises a pitch mechanism 14, a rotation mechanism 12 and a swing mechanism 11 arranged on the rotation mechanism 12, the rotation mechanism 12 can drive the swing mechanism 11 to rotate around a rotation axis, the swing mechanism 11 is provided with a scene detection unit, the swing mechanism 11 can drive the scene detection unit to swing to adjust a swing angle between an optical axis of the scene detection unit and the rotation axis, the pitch mechanism 14 is connected between the boom 21 and the rotation mechanism 12 and can drive the rotation mechanism 12 to perform up-down pitch movement around a pitch axis, the pitch axis is perpendicular to the luffing direction of the boom 21, and the arm boom operation method comprises:

[0129] obtaining an operation pose of the boom 21;

[0130] controlling the pitch mechanism 14 to act according to the operation pose of the boom 21, so that the rotation axis is adjusted to be perpendicular to the ground plane;

[0131] controlling the swing mechanism 11 and the rotation mechanism 12 to act, so that the scene detection unit performs rotation scanning around the rotation axis at a preset swing angle.

[0132] Optionally, the arm support operation method further comprises:

[0133] A global operation scene map is generated according to the scanning data of the scene detection unit, the action data of the rotating mechanism 12, and the action data of the swinging mechanism 11.

[0134] Optionally, the arm support operation method further comprises:

[0135] The total number of rotation rounds of the scene detection unit rotating around the rotation axis and the swinging angle corresponding to each rotation round are determined according to the operation scene size of the arm support 21.

[0136] Optionally, the arm support operation method further comprises:

[0137] The operation scene size of the arm support 21 is determined according to the length of the arm support 21.

[0138] Optionally, the control of the swinging mechanism 11 and the rotating mechanism 12 to operate so that the scene detection unit rotates around the rotation axis at a preset swinging angle comprises:

[0139] The current rotation round of the scene detection unit is determined, and the swinging mechanism 11 is controlled to operate to adjust the swinging angle corresponding to the current rotation round;

[0140] The rotating mechanism 12 is controlled to drive the scene detection unit to rotate at a preset rotation angle interval and a preset time period interval until the scene detection unit rotates one round at the current rotation round;

[0141] The completed last round of rotation scanning of the scene detection unit is determined, and the scene detection unit is controlled to stop scanning.

[0142] Optionally, the global operation scene map is generated according to the scanning data of the scene detection unit, the action data of the rotating mechanism 12, and the action data of the swinging mechanism 11, comprising:

[0143] The scanning data of the scene detection unit, the action data of the rotating mechanism 12, and the action data of the swinging mechanism 11 are obtained;

[0144] The scanning data is processed and spliced in combination with the action data of the rotating mechanism 12 and the action data of the swinging mechanism 11 to generate the global operation scene map.

[0145] Optionally, the arm support cloud platform further comprises a roll mechanism 13, the roll mechanism 13 being capable of driving the rotating mechanism 12 to rotate around a roll axis, the roll axis being perpendicular to the pitch axis, and the arm support operation method further comprises, before the scene detection unit rotates and scans:

[0146] The roll mechanism 13 is controlled to operate so that the rotation axis can be perpendicular to the ground plane when the arm support 21 is twisted or tilted.

[0147] Optionally, the boom operation method further comprises:

[0148] detecting the swing angle, a first tilt angle value of the roll mechanism 13 in the luffing direction of the boom 21, and a second tilt angle value of the pitch mechanism 14 in the luffing direction of the boom 21;

[0149] controlling the swing mechanism 11, the roll mechanism 13, and the pitch mechanism 14 to act according to the swing angle, the first tilt angle value, and the second tilt angle value, so that the optical axis of the scene detection unit is perpendicular to the ground plane.

[0150] Optionally, the boom operation method further comprises:

[0151] controlling the rotation mechanism 12 and the swing mechanism 11 to act so that the operation target enters the field of view collected by the scene detection unit;

[0152] displaying the image collected by the scene detection unit and obtaining a target selection operation of a user for the image;

[0153] receiving the target selection operation and determining the position of the operation target in the global map of the operation scene according to the target selection operation.

[0154] Optionally, the boom operation method further comprises:

[0155] performing operation path planning and safety calculation according to the current state of the boom 21 and the spatial position of the operation target in the global map of the operation scene.

[0156] Optionally, the boom operation method further comprises:

[0157] when the boom 21 acts according to the operation path planning, controlling the arm-mounted gimbal to act so that the optical axis of the scene detection unit is perpendicular to the ground plane, and when the operation target enters the field of view collected by the scene detection unit, identifying and repositioning the operation target.

[0158] Optionally, the scene detection unit comprises an image acquisition component 152 and / or a radar.

[0159] As shown in FIG. 8, the control process of the arm-mounted follow-up sensing device 1 of the present application will be briefly described below by taking a crane as an example:

[0160] 1. As shown in FIG. 5, before the boom 21 of the crane is unfolded, the boom 21 of the crane is parallel to the ground plane. The arm-mounted follow-up sensing device 1 is powered on, and the initial state of the arm-mounted gimbal is obtained. The initial state of the arm-mounted gimbal includes the rotation angle, the swing angle, the pitch angle, and the roll angle.

[0161] 2. Crane spreader step: as shown in FIG. 6, the swing mechanism 11, the roll mechanism 13 and the pitch mechanism 14 are controlled in real time to adjust the swing angle, the pitch angle and the roll angle, so that the optical axis of the scene detection unit is perpendicular to the ground. After the spreader is completed, the boom 21 of the crane is in the initial state of operation, at which time the optical axis of the scene detection unit is perpendicular to the ground.

[0162] 3. Scene scanning step: determine the current rotation cycle of the scene detection unit, control the swing mechanism 11 to act to adjust the swing angle corresponding to the current rotation cycle, control the rotation mechanism 12 to drive the scene detection unit to rotate at a preset rotation angle interval and a preset time period interval, until the scene detection unit rotates one cycle at the current rotation cycle, determine that the scene detection unit has completed the last cycle of rotation scanning, and control the scene detection unit to stop scanning.

[0163] 4. Based on the radar point cloud and the camera image obtained by scanning, a three-dimensional reconstruction of the operation scene is performed to generate a global map of the operation scene.

[0164] 5. After generating the global map of the operation scene, the swing mechanism 11 and the rotation mechanism 12 are controlled to adjust the swing angle and the rotation angle, so that the optical axes of the radar and the camera are perpendicular to the ground. At this time, it can be judged whether it is necessary to re-construct the global map of the operation scene, if necessary, re-enter the scene scanning step; if not, enter the initial positioning step of the operation target.

[0165] 6. Initial positioning step of operation target: control the swing mechanism 11 and the rotation mechanism 12 to adjust the swing angle and the rotation angle, so that the operation target (hoisting object or just-in-place point) enters the camera field of view, and the operation target is selected in the composite image, and the spatial position of the operation target in the global map of the operation scene is calculated and obtained.

[0166] 7. After completing the positioning of the operation target, the swing mechanism 11, the roll mechanism 13, the pitch mechanism 14 and the rotation mechanism 12 are controlled to adjust the swing angle, the roll angle, the pitch angle and the rotation angle, so as to correct the radar and camera field of view and make the optical axes of the radar and the camera perpendicular to the ground. And according to the current state of the boom 21 of the crane and the spatial position of the operation target, the operation path planning and safety calculation are automatically performed.

[0167] 8. Start the intelligent hoisting function, control the boom 21 to act according to the planned operation path. During this process, the swing mechanism 11, the roll mechanism 13 and the pitch mechanism 14 need to be controlled in real time to adjust the swing angle, the roll angle and the pitch angle, so that the optical axes of the radar and the camera are perpendicular to the ground.

[0168] 9. When the arm 21 moves close to the work target, the work target enters the radar and camera field of view, and automatic identification and secondary positioning (precise positioning) of the work target are performed based on the radar and camera. In this process, the swing mechanism 11, the roll mechanism 13 and the pitch mechanism 14 also need to be controlled in real time to adjust the swing angle, the roll angle and the pitch angle, so that the optical axis of the radar and the camera is perpendicular to the ground.

[0169] 10. When the hook approaches the work target and the distance between the hook and the work target is within a set threshold range, the work is considered to be completed.

[0170] 11. After the completion of the previous work and entering the next work process, it can be judged whether it is necessary to rebuild the global map of the work scene, if necessary, re-enter the scene scanning step; if not, enter the initial positioning step of the work target. If the next work process is not needed after the completion of the previous work, the work can be ended at this time.

[0171] In summary, the arm-mounted follow-up sensing device 1, the arm-mounted follow-up sensing system and the working machine of the present application have a four-axis motion controllable arm-mounted holder, the camera and radar sensors are installed on the arm-mounted holder, the horizontal 360 rotation of the camera and radar, the inclination control of the amplitude direction and the left-right direction perpendicular to the amplitude direction can be realized, and the optical axis of the camera and radar can be kept perpendicular downward during the amplitude and rotation of the arm 21. In addition, the arm-mounted holder, the arm-mounted follow-up sensing device, the working machine and the vehicle of the present application can realize the rotation scanning of the camera and radar at different swing angles, so as to realize the large field of view monitoring and three-dimensional reconstruction of a large work range; also, during the movement of the arm 21, whether the arm 21 has roll deformation or not, the posture of the camera and radar can be kept perpendicular downward, so as to realize the monitoring of the hook load and the real-time detection of the surrounding obstacles during the movement of the hook load, and help to ensure the coverage of the scene detection area. In addition, when the arm-mounted holder, the arm-mounted follow-up sensing device and the arm-mounted follow-up sensing system of the present application are applied to the working machine such as a crane, the arm-mounted follow-up three-dimensional reconstruction, obstacle detection, target following and the like can be realized, which provides a technical basis for the automatic hoisting and intelligent hoisting of the crane, and the "eye" function of the crane is guaranteed.

[0172] In the description of the present application, it should be understood that the terms "first", "second" are only for the purpose of description, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "multiple" is at least two, such as two, three, etc., unless otherwise specifically limited.

[0173] In this application, unless otherwise explicitly specified and limited, the terms "mounting", "connection", "connecting", "fixing" and the like should be understood in a broad sense, for example, can be fixed connection, can also be detachable connection, or integral; can be mechanical connection, can also be electrical connection or communication with each other; can be directly connected, can also be indirectly connected through an intermediate medium, can be the internal communication of two elements or the interaction relationship between two elements, unless otherwise explicitly limited. For those skilled in the art, the specific meaning of the above terms in this application can be understood according to the specific circumstances.

[0174] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "example", "specific example" or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. Furthermore, the skilled in the art can combine and combine the different embodiments or examples described in the present specification and the features of the different embodiments or examples without contradiction.

[0175] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above embodiments within the scope of the present application.

Claims

1. An arm-mounted servo-aware system, wherein, Comprising: a scene detection unit configured to acquire scanning data of a work scene; an arm-mounted holder provided on an arm support (21) and comprising a tilting mechanism (14), a rotating mechanism (12) and a swinging mechanism (11), the rotating mechanism (12) being connected with the swinging mechanism (11) and capable of driving the swinging mechanism (11) to rotate around a rotating axis, the scene detection unit being provided on the swinging mechanism (11), the swinging mechanism (11) being capable of driving the scene detection unit to swing to adjust a swing angle between an optical axis of the scene detection unit and the rotating axis, the tilting mechanism (14) being connected between the arm support (21) and the rotating mechanism (12) and capable of driving the rotating mechanism (12) to perform up-and-down tilting movement around a tilting axis, the tilting axis being perpendicular to a luffing direction of the arm support (21); and a processing device (17) in communication with the arm-mounted holder and the scene detection unit respectively and configured to: acquire a work pose of the arm support (21); control the tilting mechanism (14) to act according to the work pose of the arm support (21) so that the rotating axis is adjusted to be perpendicular to a ground plane; control the swinging mechanism (11) and the rotating mechanism (12) to act so that the scene detection unit performs rotational scanning around the rotating axis at a preset swing angle.

2. The arm servo-aware system of claim 1, wherein, The processing device (17) is further configured to: generate a global map of a work scene according to the scanning data of the scene detection unit, action data of the rotating mechanism (12) and action data of the swinging mechanism (11).

3. The arm servo-aware system of claim 2, wherein, The processing device (17) is further configured to: determine a total number of rotations of the scene detection unit around the rotating axis and a swing angle corresponding to each rotation according to a size of the work scene of the arm support (21).

4. The arm servo-aware system of claim 2, wherein, The processing device (17) is configured to: determine the size of the work scene of the arm support (21) according to a length of the arm support (21).

5. The arm servo-aware system of claim 2, wherein, The generating of the global map of the work scene according to the scanning data of the scene detection unit, the action data of the rotating mechanism (12) and the action data of the swinging mechanism (11) comprises: acquiring the scanning data of the scene detection unit, the action data of the rotating mechanism (12) and the action data of the swinging mechanism (11); processing and splicing the scanning data in combination with the action data of the rotating mechanism (12) and the action data of the swinging mechanism (11) to generate the global map of the work scene.

6. The arm servo-aware system of claim 1, wherein, The controlling of the swinging mechanism (11) and the rotating mechanism (12) to act so that the scene detection unit performs rotational scanning around the rotating axis at a preset swing angle comprises: determining a current rotation of the scene detection unit, controlling the swinging mechanism (11) to act to adjust the swing angle corresponding to the current rotation; controlling the rotating mechanism (12) to drive the scene detection unit to rotate at a preset rotation angle interval and a preset time period interval until the scene detection unit rotates one round at the current rotation. determining that the scene detection unit has completed a last round of rotation scanning, controlling the scene detection unit to stop scanning.

7. The arm servo-aware system of claim 1, wherein, The arm-mounted gimbal further comprises a roll mechanism (13) capable of driving the rotation mechanism (12) to rotate around a roll axis perpendicular to the pitch axis, and the processing device (17) is further configured to: control the roll mechanism (13) to act so that the rotation axis can be perpendicular to the ground plane when the arm support (21) is twisted or tilted.

8. The arm servo-aware system of claim 7, wherein, The pitch mechanism (14), the roll mechanism (13), the rotation mechanism (12) and the swing mechanism (11) are connected in sequence; or the roll mechanism (13), the pitch mechanism (14), the rotation mechanism (12) and the swing mechanism (11) are connected in sequence.

9. The arm servo-aware system of claim 2, wherein, The arm-mounted follow-up sensing system further comprises a human-computer interaction device (22) for receiving and displaying images collected by the scene detection unit, and obtaining a target selection operation of a user for the images. The processing device (17) communicates with the human-computer interaction device (22) and is further configured to: control the rotation mechanism (12) and the swing mechanism (11) to act so that the work target enters the field of view collected by the scene detection unit; receive the target selection operation and determine the position of the work target in the global map of the work scene according to the target selection operation.

10. The arm servo-aware system of claim 9, wherein, The processing device (17) is further configured to: according to the current state of the arm support (21) and the spatial position of the work target in the global map of the work scene, perform work path planning and safety calculation.

11. The arm servo-aware system of claim 10, wherein, The processing device (17) is further configured to: when the arm support (21) acts according to the work path planning, control the arm-mounted gimbal to act so that the optical axis of the scene detection unit is perpendicular to the ground plane, and when the work target enters the field of view collected by the scene detection unit, identify and reposition the work target.

12. The arm servosystem of claim 1, wherein, The arm-mounted follow-up sensing system further comprises a first angle sensor and a second angle sensor, the first angle sensor is arranged on the swing mechanism (11) and is used to detect the swing angle of the swing mechanism (11), and the second angle sensor is arranged on the rotation mechanism (12) and is used to detect the rotation angle of the rotation mechanism (12); and / or, the scene detection unit comprises an image acquisition component (152) and / or a radar.

13. A work machine, wherein, It comprises: an arm support (21) and an arm-mounted follow-up sensing system according to any one of claims 1 to 12, and the arm-mounted gimbal is arranged on the arm support (21).

14. A vehicle, wherein, It comprises: a vehicle body, an arm support (21) and an arm-mounted follow-up sensing system according to any one of claims 1 to 12, and the arm support (21) is arranged on the vehicle body, and the arm-mounted gimbal is arranged on the arm support (21).

15. A method of operating a boom, wherein An arm support (21) is provided with an arm-mounted holder, the arm-mounted holder comprising a tilting mechanism (14), a rotating mechanism (12) and a swinging mechanism (11) arranged on the rotating mechanism (12), the rotating mechanism (12) being capable of driving the swinging mechanism (11) to rotate around a rotating axis, the swinging mechanism (11) being provided with a scene detection unit, the swinging mechanism (11) being capable of driving the scene detection unit to swing to adjust a swing angle between an optical axis of the scene detection unit and the rotating axis, the tilting mechanism (14) being connected between the arm support (21) and the rotating mechanism (12) and being capable of driving the rotating mechanism (12) to perform up-and-down tilting movement around a tilting axis, the tilting axis being perpendicular to a luffing direction of the arm support (21), and an arm support operation method comprising: acquiring an operation pose of the arm support (21); controlling the tilting mechanism (14) to act according to the operation pose of the arm support (21) so that the rotating axis is adjusted to be perpendicular to a ground plane; controlling the swinging mechanism (11) and the rotating mechanism (12) to act so that the scene detection unit performs rotating scanning around the rotating axis at a preset swing angle.

Images