Breadth of the infrared and event camera information fusion of the shovel blocked trajectory planning method

Abstract

The present application relates to a shovel blocked trajectory planning method based on wide-spectrum infrared and event camera information fusion, belonging to the technical field of excavation path planning. It includes: obtaining short-wave infrared images, thermal infrared images and space-time event streams; extracting the shovel's region of interest from the short-wave infrared images and / or thermal infrared images; screening the effective event stream corresponding to the shovel's region of interest from the space-time event stream; extracting the inherent characteristics of the shovel's back from the effective event stream and estimating the actual space-time position of the shovel at the current moment; judging whether the shovel is in a blocked working condition at the current moment according to the actual space-time position of the shovel at the current moment, the pushing torque and the lifting torque; if so, obtaining a preset trajectory planning sequence including the target space-time position at the current moment, and adjusting the preset trajectory planning sequence according to the blocked working condition type of the shovel at the current moment. The estimated actual space-time position of the shovel is more accurate, and the planned trajectory can be adjusted in real time according to the excavation blocking condition of the shovel.

Description

Technical Field

[0001] This invention relates to the field of excavation path planning technology, and in particular to a method for planning obstructed bucket trajectories by fusing broadband infrared and event camera information. Background Technology

[0002] Currently, with the rapid development of smart mines, automated construction and other scenarios, the intelligent upgrading of traditional construction machinery (such as excavators and loaders) has become an important research direction. Among them, bucket trajectory planning, as one of the important control technologies, has been widely used in construction machinery, mining machinery and other fields.

[0003] Traditional bucket trajectory planning often relies on conventional sensors (such as cameras and lidar). However, in complex environments, such as those with large changes in lighting or severe weather conditions, sensors are easily interfered with, leading to a decrease in control accuracy. Moreover, once the bucket trajectory model is generated based on traditional sensor data, it is difficult to make real-time adjustments according to the excavation obstruction. In particular, it is helpless when encountering hard rock or large rocks during the excavation process, which cause a sharp increase in resistance, resulting in the failure of bucket trajectory planning. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides a method for planning the obstructed trajectory of a bucket by fusing broadband infrared and event camera information. The technical solution of this invention is as follows:

[0005] A method for planning the obstructed trajectory of a bucket by fusing broadband infrared and event camera information, wherein a broadband infrared camera and an event camera are installed below the boom of the equipment where the bucket is located. The broadband infrared camera includes a short-wave infrared camera and a thermal infrared camera, and the event camera, the short-wave infrared camera and the thermal infrared camera are temporally and spatially aligned.

[0006] S1, acquire the short-wave infrared image and thermal infrared image collected by the short-wave infrared camera and thermal infrared camera at the current moment, respectively, and acquire the spatiotemporal event stream collected by the event camera at the current moment;

[0007] S2, Extract the region of interest of the bucket from shortwave infrared images and / or thermal infrared images;

[0008] S3: Filter the event streams corresponding to the region of interest of the bucket from the spatiotemporal event streams as valid event streams;

[0009] S4, extract the inherent features of the bucket back from the valid event stream;

[0010] S5. Based on the inherent features of the bucket's back and the relative positional relationship between the shortwave infrared camera and the event camera, estimate the actual spatiotemporal position of the bucket at the current moment.

[0011] S6, obtain the current pushing torque and lifting torque of the pushing motor shaft and lifting motor shaft of the equipment where the bucket is located, and determine whether the bucket is in an obstructed working condition based on the actual spatiotemporal position, pushing torque and lifting torque of the bucket at the current moment;

[0012] S7. If the bucket is currently in an obstructed working condition, obtain a preset trajectory planning sequence including the target spatiotemporal position of the bucket at the current moment, and adjust the preset trajectory planning sequence according to the obstructed working condition type of the bucket at the current moment.

[0013] Optionally, S2 includes:

[0014] S21, extract the first bucket region of interest from the shortwave infrared image based on the reflectance of each pixel in the shortwave infrared image, and calculate the average reflectance of the first bucket region of interest.

[0015] S22, extract the region of interest of the second bucket from the thermal infrared image based on the gray value of each pixel, and calculate the average gray value of the region of interest of the second bucket.

[0016] S23, when the average reflectance of the region of interest of the first bucket is greater than the preset reflectance threshold, the region of interest of the first bucket is taken as the region of interest of the bucket; when the average gray value of the region of interest of the second bucket does not exceed the preset gray value threshold, the region of interest of the second bucket is taken as the region of interest of the bucket; when the average reflectance of the region of interest of the first bucket is not greater than the preset reflectance threshold, and the average gray value of the region of interest of the second bucket exceeds the preset gray value threshold, the intersection of the region of interest of the first bucket and the region of interest of the second bucket is taken as the region of interest of the bucket.

[0017] Optionally, the inherent feature is three horizontally arranged rectangles, each with a preset height and a preset spacing between adjacent rectangles. S4 includes:

[0018] S41, extract the top and bottom horizontal edges of the rectangle from the valid event stream;

[0019] S42, detect the six vertical edges of three rectangles in the valid event stream within the coverage area of ​​the upper and lower horizontal edges;

[0020] S43, perform rectangle verification based on the upper horizontal edge, lower horizontal edge, six vertical edges, preset width and preset height, and when the verification is successful, use the features of the three rectangles as inherent features of the bucket back.

[0021] Optionally, S41 includes:

[0022] S411, bucket the ordinate of each pixel in the valid event stream in the event camera coordinate system.

[0023] S412, count the total number of events with positive and negative polarities within each bucket on the vertical axis;

[0024] S413, if the total number of events with positive and negative polarities within any vertical axis bucket is greater than a preset high-density peak value, then mark the vertical axis bucket as a candidate horizontal edge;

[0025] S414, Pair all candidate horizontal edges according to the preset height to obtain the upper and lower horizontal edges in the inherent features of the bucket back.

[0026] Optionally, S42 includes:

[0027] S421, perform horizontal coordinate bucketing on the horizontal coordinates of each pixel in the effective event stream within the coverage area of ​​the upper and lower horizontal edges, in the event coordinates of the event camera event coordinate system.

[0028] S422, count the total number of events with positive and negative polarities within each horizontal axis bucket;

[0029] S423, select the six horizontal axis buckets with the largest total number of events with positive and negative polarities as the six vertical edges in the inherent features of the bucket back.

[0030] Optionally, S5 includes:

[0031] S51. Based on the relative positional relationship between the shortwave infrared camera and the event camera, as well as the event coordinates of the inherent features of the bucket back in the event camera coordinate system and the shortwave infrared coordinates in the shortwave infrared camera coordinate system, calculate the parallax between the event camera and the shortwave infrared camera.

[0032] S52, calculate the depth of the inherent feature on the back of the bucket based on the parallax between the event camera and the shortwave infrared camera and the relative positional relationship between the shortwave infrared camera and the event camera, and determine the three-dimensional coordinates of the inherent feature on the back of the bucket based on the event coordinates of the inherent feature on the back of the bucket in the event camera coordinate system.

[0033] S53, based on the three-dimensional coordinates of the inherent features of the bucket's back and the bucket's prior three-dimensional rigid body model, estimate the bucket's actual spatiotemporal position at the current moment.

[0034] Optionally, S6 includes:

[0035] S61, based on the torque encoders pre-installed on the push motor shaft and the lifting motor shaft of the equipment where the bucket is located, obtain the push torque and lifting torque of the equipment at the current moment;

[0036] S62, calculate the horizontal digging resistance and vertical lifting resistance of the bucket at the current moment based on the pushing torque and lifting torque respectively;

[0037] S63 determines whether the bucket is currently in an obstructed working condition based on the bucket's actual spatial and temporal position, horizontal digging resistance, and vertical lifting resistance.

[0038] Optionally, the obstructed working conditions include hard rock obstruction conditions, large rock obstruction conditions, and overload obstruction conditions, and S63 includes:

[0039] S631, if the horizontal digging resistance of the bucket at the current moment is greater than the empirical threshold for hard rock digging resistance, and the displacement change rate of the bucket in the upward and forward directions is less than the first displacement change rate threshold, then it is determined that the bucket is currently in a hard rock obstruction condition.

[0040] S632, if the variance of the horizontal digging resistance of the bucket at the current moment is greater than the preset variance threshold, and the horizontal digging resistance of the bucket at the current moment is greater than the preset hard rock digging resistance threshold, then it is determined that the bucket is currently in the condition of being obstructed by large rocks.

[0041] S633, if the vertical lifting resistance of the bucket at the current moment is greater than the preset overload resistance upper limit threshold, and the displacement change rate of the bucket in the upward direction is less than the second displacement change rate threshold, then it is determined that the bucket is currently in an overloaded and obstructed working condition.

[0042] Optionally, S7 includes:

[0043] S71, if the bucket is currently in an obstructed working condition, a preset trajectory planning sequence including the target spatiotemporal position of the bucket at the current moment is obtained through a pre-trained trajectory planning neural network. Each time the trajectory planning neural network performs bucket trajectory planning, it outputs a trajectory planning sequence, which includes a preset number of target spatiotemporal positions.

[0044] S72, take the target spatiotemporal position of the bucket at the current moment as the obstruction point, and determine the position of the obstruction point in the preset trajectory planning sequence;

[0045] S73 adjusts the target spatiotemporal position starting from the obstruction point in the preset trajectory planning sequence according to the obstruction condition type of the bucket at the current moment and the position of the obstruction point in the preset trajectory planning sequence.

[0046] Optionally, S73 includes:

[0047] If the current obstruction condition of the bucket is hard rock, then the first th ... k The spatiotemporal location of each target is adjusted:

[0048] (1);

[0049] In formula (1), Indicates the first in the preset trajectory planning sequence k The spatiotemporal location of the target Indicates the adjusted number k The spatiotemporal location of the target Indicates the lateral offset coefficient. Indicates the lifting coefficient. Indicates the adjustment factor. This indicates the current horizontal digging resistance of the bucket. This represents the empirical threshold for hard rock excavation resistance. K This indicates the position number of the obstructed point in the preset trajectory planning sequence. P The preset number indicates the number of target spatiotemporal locations included in the preset trajectory planning sequence. x The right side of the bucket. y The direction directly above the bucket. z The direction directly in front of the bucket;

[0050] If the current obstruction condition of the bucket is large rock obstruction, then the first step in the preset trajectory planning sequence is calculated using formula (2). k The spatiotemporal location of each target is adjusted:

[0051] (2);

[0052] In formula (2), This indicates the displacement of the boom retraction. Indicates the oscillation frequency. Indicates the amplitude of the swing. exp ( ) represents an exponential function. This represents the variance of horizontal excavation resistance under normal operating conditions;

[0053] If the current obstruction condition of the bucket is an overload obstruction condition, then the first th ... k The spatiotemporal location of each target is adjusted:

[0054] (3);

[0055] In formula (3), This indicates the upper limit threshold of overload resistance. This indicates the vertical lifting resistance of the bucket at the current moment. This indicates the overload factor.

[0056] All of the above-mentioned optional technical solutions can be combined arbitrarily, and the present invention will not provide a detailed description of the structure after each combination.

[0057] By means of the above solution, the beneficial effects of the present invention are as follows:

[0058] By installing a temporally and spatially aligned broadband infrared camera and an event camera below the boom of the equipment housing the bucket, and estimating the bucket's actual spatiotemporal position based on data collected by these three cameras and their relative positions, the influence of harsh environments such as changes in lighting or dusty conditions on the determination of the bucket's actual spatiotemporal position can be overcome, resulting in a more accurate estimated bucket position. Furthermore, by determining the bucket's current obstruction condition based on its actual spatiotemporal position and adjusting the preset trajectory planning sequence according to the type of obstruction, the pre-planned bucket trajectory can be adjusted in real time based on the bucket's digging obstruction. This significantly improves the bucket's adaptability and robustness under obstruction conditions such as hard rock, large rocks, and overload, making the adjusted trajectory planning sequence more accurate.

[0059] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, the preferred embodiments of the present invention are described in detail below with reference to the accompanying drawings. Attached Figure Description

[0060] Figure 1 This invention provides a method for planning the obstructed trajectory of a bucket by fusing broadband infrared and event camera information.

[0061] Figure 2 This is a schematic diagram of the equipment, including the area behind the bucket.

[0062] Figure 3 This is a schematic diagram of the inherent features of the back of the bucket in an embodiment of the present invention. Detailed Implementation

[0063] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.

[0064] The bucket obstruction trajectory planning method based on the fusion of broadband infrared and event camera information provided in this invention can be implemented using any electronic device with computing capabilities, such as a PC, mobile terminal, or server. Before executing the method provided in this invention, by combining the complementary advantages of short-wave infrared cameras, thermal infrared cameras, and event cameras, this invention constructs an anti-obstruction, highly robust bucket obstruction trajectory planning system. Specifically, this invention pre-installs broadband infrared cameras and event cameras under the boom of the equipment where the bucket is located, such as an excavator or loader. The broadband infrared cameras include short-wave infrared cameras and thermal infrared cameras, and the event cameras, short-wave infrared cameras, and thermal infrared cameras are aligned in time and space.

[0065] Temporal alignment refers to ensuring that the image acquisition frequency and time of the event camera, shortwave infrared camera, and thermal infrared camera are the same. Spatial alignment refers to maintaining the relative positional relationship of the event camera, shortwave infrared camera, and thermal infrared camera, and calculating the intrinsic and extrinsic parameters of the calibrated cameras using a checkerboard calibration plate.

[0066] For ease of implementation, in this embodiment of the invention, when installing a broadband infrared camera and an event camera below the crane boom, the short-wave infrared camera and the thermal infrared camera can be installed on the same horizontal plane, and the event camera and the short-wave infrared camera can be installed on the same vertical plane.

[0067] Based on the above, such as Figure 1 As shown, the bucket obstruction trajectory planning method based on broadband infrared and event camera information fusion provided in this embodiment of the invention can be implemented through the following steps S1 to S7:

[0068] S1: Acquire the short-wave infrared image and thermal infrared image captured by the short-wave infrared camera and thermal infrared camera at the current moment, respectively, and acquire the spatiotemporal event stream captured by the event camera at the current moment.

[0069] Specifically, after acquiring short-wave infrared images, thermal infrared images, and spatiotemporal event streams, the short-wave infrared camera, thermal infrared camera, and event camera transmit them to the electronic device in real time. The electronic device obtains the short-wave infrared images, thermal infrared images, and spatiotemporal event streams by receiving the data they transmit.

[0070] S2, Extract the region of interest of the bucket from shortwave infrared images and / or thermal infrared images.

[0071] The region of interest (ROI) for the bucket is the area including the bucket extracted from shortwave infrared (SIR) images and / or thermal infrared (TII) images. The "and / or" here includes three cases: extracting the ROI from shortwave infrared (SIR) images, extracting the ROI from thermal infrared (TII) images, and extracting the ROI from both SIR and TII images. Since regions including the bucket can be extracted from both SIR and TII images, determining which of these three cases to use for ROI extraction depends on the imaging characteristics of the SIR and TII images. Details are as follows:

[0072] In one specific embodiment, S2 includes the following steps S21 to S23:

[0073] S21, extract the first bucket region of interest from the shortwave infrared image based on the reflectance of each pixel in the shortwave infrared image, and calculate the average reflectance of the first bucket region of interest.

[0074] The imaging advantages of short-wave infrared cameras include: under daytime or well-lit conditions, in the 1.5–2.5 μm band of the short-wave infrared camera, there are significant differences in optical properties between the reflectance of metal buckets (reflectance ≥60%) and materials (carbon-based materials reflectance <15%, quartz reflectance >40%, mica reflectance <20%). Even if the bucket surface is oxidized or coated with a wear-resistant coating (reflectance reduced to 40–50%), the bucket area and non-bucket area can still be quickly distinguished by setting a priori threshold reflectance. Therefore, this embodiment of the invention extracts the first bucket region of interest from the short-wave infrared image based on the different reflectance characteristics of different materials in the short-wave infrared band. To ensure that the first bucket region of interest, which includes the entire bucket area, can be extracted from the short-wave infrared image, this embodiment of the invention sets a priori threshold reflectance of 50% based on the reflectance of each material.

[0075] Specifically, based on the short-wave infrared image output by the short-wave infrared camera, the short-wave infrared coordinates and reflectance of each pixel in the short-wave infrared camera coordinate system can be obtained. Therefore, in this embodiment of the invention, during S21, pixels with a reflectance greater than 50% are extracted from the short-wave infrared image, and these pixels are connected to form a region, thus forming a first bucket region of interest. Further, when calculating the average reflectance of the first bucket region of interest, the reflectances of all pixels included in the first bucket region of interest are added together and then divided by the number of pixels included in the first bucket region of interest to obtain the average reflectance of the first bucket region of interest.

[0076] S22, extract the region of interest of the second bucket from the thermal infrared image based on the gray values ​​of each pixel in the thermal infrared image, and calculate the average gray value of the region of interest of the second bucket.

[0077] The imaging advantages of thermal infrared cameras include: under limited illumination conditions at night, thermal infrared cameras (8–14 μm) generate thermal infrared images by capturing the thermal radiation of the object itself (characterized by radiance). Specifically, after receiving the radiance of the object, the thermal infrared camera converts it into the grayscale value of the pixels. In order to clarify the relationship between the grayscale value of each pixel and the radiance and emissivity of the material, this embodiment of the invention constructs a grayscale-emissivity-temperature joint model as shown in the following formula (4) to segment the bucket and other materials in the thermal infrared image.

[0078] (4);

[0079] In formula (4), D Represents the grayscale value of a pixel, ranging from 0 to 255; k L It is the radiance gain coefficient, which can be obtained by calibration measurement under a standard radiation source and is used to accurately convert the radiance of a pixel into a grayscale value. λ Indicates wavelength; It represents the radiance of an object and is related to the wavelength λ. Represents the emissivity of a material, in relation to wavelength. λ Related; Indicates wavelength λ Planck radiance at that location; h Denotes Planck's constant; c Represents the speed of light; r Represents the Boltzmann constant; T This represents the absolute temperature of a blackbody.

[0080] Metallic materials typically have low emissivity in the thermal infrared band, usually between 0.1 and 0.3; non-metallic materials have higher emissivity, for example, coal typically has an emissivity between 0.85 and 0.95, and rocks typically have an emissivity between 0.7 and 0.9.

[0081] Based on the above grayscale-emissivity-temperature joint model, at a nighttime ambient temperature of -5℃, in the thermal infrared band with λ=10μm, the radiance of various materials is as follows: Radiance of a metal bucket or anodized metal bucket radiance of coal Radiance of quartz Radiance of mica Therefore, it can be concluded that there is an order-of-magnitude difference in the radiance of the bucket compared to other materials. Based on the relationship between radiance and grayscale value in the above formula (4), it can be seen that the order-of-magnitude difference in radiance between the bucket and other materials can also be reflected in the grayscale value of each pixel in the thermal infrared image. Therefore, based on the order-of-magnitude difference in grayscale value of various materials, the embodiments of the present invention can extract the second bucket region of interest from the thermal infrared image according to the grayscale value of each pixel in the thermal infrared image.

[0082] Specifically, in extracting the region of interest (ROI) of the second bucket, this embodiment of the invention first finds the pixel corresponding to the maximum grayscale value in the thermal infrared image, and uses 30% of the maximum grayscale value as a reference grayscale value. This reference grayscale value is used as the extraction standard for the ROI of the second bucket. Pixels with grayscale values ​​less than the reference grayscale value are extracted from the thermal infrared image, and these pixels are connected to form a region, thus forming the ROI of the second bucket. Further, when calculating the average grayscale value of the ROI of the second bucket, the grayscale values ​​of all pixels included in the ROI of the second bucket are added together and then divided by the number of pixels included in the ROI of the second bucket to obtain the average grayscale value of the ROI of the second bucket.

[0083] S23, when the average reflectance of the region of interest of the first bucket is greater than the preset reflectance threshold, the region of interest of the first bucket is taken as the region of interest of the bucket; when the average gray value of the region of interest of the second bucket does not exceed the preset gray value threshold, the region of interest of the second bucket is taken as the region of interest of the bucket; when the average reflectance of the region of interest of the first bucket is not greater than the preset reflectance threshold, and the average gray value of the region of interest of the second bucket exceeds the preset gray value threshold, the intersection of the region of interest of the first bucket and the region of interest of the second bucket is taken as the region of interest of the bucket.

[0084] Specifically, in order to accurately determine the region of interest in the bucket, this embodiment of the invention sets a preset reflectivity threshold of 1.2 times the prior threshold reflectivity, i.e., a preset reflectivity threshold of 60%. Based on the differences in grayscale values ​​of various materials, this embodiment of the invention sets a preset grayscale threshold of 20% of the maximum grayscale value of the thermal infrared image, i.e., a preset grayscale threshold of two-thirds of the reference grayscale value.

[0085] In other words, the average reflectivity of the bucket is highest in short-wave infrared images. When the average reflectivity of the first bucket region of interest is greater than 1.2 times the prior threshold reflectivity, this embodiment of the invention prioritizes the first bucket region of interest extracted from the short-wave infrared image as the final bucket region of interest. The average gray value of the bucket in thermal infrared images is low. When the average gray value of the second bucket region of interest does not exceed 20% of the maximum gray value of the thermal infrared image, this embodiment of the invention prioritizes the second bucket region of interest extracted from the thermal infrared image as the final bucket region of interest. If neither of the above two conditions is met, the intersection of the first bucket region of interest and the second bucket region of interest is taken as the final bucket region of interest.

[0086] S3: Filter the event streams corresponding to the region of interest of the bucket from the spatiotemporal event streams as valid event streams;

[0087] Specifically, the output of the event camera is a sequence of quadruples: ,in, It is a pixel m Event coordinates in the event camera coordinate system x m It is the x-coordinate in the event coordinate system. y m It is the ordinate in the event coordinate system. t m It is the event trigger timestamp; p m Indicates polarity, and can take values ​​of +1 or -1, where +1 and -1 represent an increase or decrease in brightness, respectively; N This refers to the total number of events. A valid event stream refers to the event stream corresponding to the region of interest of the bucket within the spatiotemporal event stream.

[0088] S4 extracts the inherent features of the bucket back from the valid event stream.

[0089] Specifically, based on the characteristic that the event camera is highly sensitive to changes in brightness of moving edge features, and combined with the huge difference between the regular geometric structure of the bucket back and the irregular shape of the material, this embodiment of the invention extracts the inherent features of the bucket back from the effective event stream.

[0090] like Figure 2 The diagram shown is a schematic of the equipment including the back area of ​​the bucket, with the red box highlighting the back area of ​​the bucket. Analysis Figure 2 As can be seen from the actual working conditions of the bucket, the inherent features of the bucket back include three horizontally arranged rectangles, such as... Figure 3 As shown. The geometric characteristics of the three rectangles satisfy: the height is the same, all are h; the widths are w1, w2, and w3 respectively; the horizontal spacing is uniform, and the spacing between adjacent rectangles (preset width) is d; the overall width is w1+w2+w3+2d.

[0091] In light of the inherent characteristics of the bucket's back, step S4, in its specific implementation, includes the following steps S41 to S43:

[0092] S41, extract the top and bottom horizontal edges of the rectangle from the valid event stream.

[0093] Based on the geometric features of the rectangular back of the bucket described above, the horizontal edges of the three rectangles are located on the same horizontal straight line. Therefore, there are two horizontal edges in the inherent features of the bucket back: the upper horizontal edge and the lower horizontal edge.

[0094] In one specific embodiment, S41 includes:

[0095] S411 performs y-coordinate binning on the y-coordinate of each pixel in the valid event stream within the event camera coordinate system.

[0096] Specifically, when performing vertical coordinate binning, the vertical coordinate interval in the event coordinates of adjacent vertical coordinate bins is set to be the same.

[0097] S412, count the total number of events with positive and negative polarities within each bucket on the vertical axis.

[0098] Specifically, since the edge brightness of the inherent features on the back of the bucket changes from dark to bright or from bright to dark, this embodiment of the invention counts the total number of events with positive (+1) and negative (-1) polarities within each vertical axis bin in order to accurately extract the inherent features on the back of the bucket.

[0099] S413, if the total number of events with positive and negative polarities within any vertical coordinate bin is greater than a preset high-density peak value, then the vertical coordinate bin is marked as a candidate horizontal edge.

[0100] The specific value of the preset high-density peak value can be set as needed. In this embodiment of the invention, the preset high-density peak value is set to... N / 15.

[0101] S414, Pair all candidate horizontal edges according to the preset height to obtain the upper and lower horizontal edges in the inherent features of the bucket back.

[0102] Specifically, since the height of the three rectangles in the inherent features of the bucket back is a preset height h, this embodiment of the invention selects two candidate horizontal edges with a vertical coordinate difference of h±h1 from all candidate horizontal edges as the upper horizontal edges. y top and lower horizontal edge y bottom Where h1 is the recognition error height.

[0103] S42, detects the six vertical edges of three rectangles in the valid event stream within the coverage area of ​​the upper and lower horizontal edges.

[0104] Specifically, in combination Figure 3 It can be seen that the inherent features of the bucket back include three rectangles, and therefore, a total of six vertical edges.

[0105] In one specific embodiment, S42 includes:

[0106] S421 performs horizontal coordinate bucketing on the horizontal coordinates of each pixel in the effective event stream within the coverage area of ​​the upper and lower horizontal edges, based on the horizontal coordinates of the event coordinates in the event camera's event coordinate system.

[0107] Specifically, when performing horizontal coordinate binning, the horizontal coordinate interval in the event coordinates between adjacent horizontal coordinate bins is set to be the same.

[0108] S422, count the total number of events with positive and negative polarities within each horizontal axis bucket.

[0109] The principle of this step is the same as that of S412, and will not be repeated here.

[0110] S423, select the six horizontal axis buckets with the largest total number of events with positive and negative polarities as the six vertical edges in the inherent features of the bucket back.

[0111] Specifically, based on the characteristics of the three rectangles on the back of the bucket, it can be seen that there are a total of six vertical edges. Furthermore, based on the characteristics of edge polarity change, the number of events with positive and negative polarities should be the largest among the vertical edges. Therefore, in this embodiment of the invention, the six horizontal coordinate bins with the largest total number of events with positive and negative polarities are selected as the six vertical edges in the inherent features of the back of the bucket.

[0112] Furthermore, after selecting six vertical edges, to ensure the accuracy of the selected vertical edges, this embodiment of the invention can further verify the six vertical edges based on the widths of the three rectangles. If the difference in the horizontal coordinates between the first and second vertical edges (representing the width w1 of the first rectangle), the difference in the horizontal coordinates between the third and fourth vertical edges (representing the width w2 of the second rectangle), and the difference in the horizontal coordinates between the fifth and sixth vertical edges (representing the width w3 of the third rectangle) are all within a preset difference range, then the verification is considered successful. Otherwise, the six vertical edges are selected again using the above method until the widths of the first, second, and third rectangles are all within the preset difference range. The preset difference range can be empirically set to 20-30 event camera pixels.

[0113] S43, perform rectangle verification based on the upper horizontal edge, lower horizontal edge, six vertical edges, preset width and preset height, and when the verification is successful, use the features of the three rectangles as inherent features of the bucket back.

[0114] Specifically, during verification, it is determined whether the height of the three rectangles is within the range of h±h1, whether the spacing between adjacent rectangles is within the range of d±d1, and whether the overall width of the three rectangles is within the range of w1+w2+w3+2d±w. If so, the verification passes; otherwise, the upper horizontal edge, lower horizontal edge, and six vertical edges are redefined using the above method. The value of h1 can be empirically set to 10 event camera pixels, the value of d1 can be empirically set to 10 event camera pixels, and the value of w can be empirically set to 20-30 event camera pixels.

[0115] S5 estimates the actual spatiotemporal position of the bucket at the current moment based on the inherent characteristics of the bucket's back and the relative positional relationship between the shortwave infrared camera and the event camera.

[0116] In one specific embodiment, S5 includes:

[0117] S51, based on the relative positional relationship between the shortwave infrared camera and the event camera, as well as the event coordinates of the inherent features of the bucket back in the event camera coordinate system and the shortwave infrared coordinates in the shortwave infrared camera coordinate system, calculate the parallax between the event camera and the shortwave infrared camera.

[0118] Specifically, the event coordinates are the coordinates of the inherent features of the bucket's back in the event camera coordinate system; the shortwave infrared coordinates are the coordinates of the inherent features of the bucket's back in the shortwave infrared camera coordinate system. The parallax between the event camera and the shortwave infrared camera... g This represents the difference between the x-coordinates of the same point imaged by the shortwave infrared camera and the event camera.

[0119] S52, calculate the depth of the inherent feature on the back of the bucket based on the parallax between the event camera and the shortwave infrared camera and the relative positional relationship between the shortwave infrared camera and the event camera, and determine the three-dimensional coordinates of the inherent feature on the back of the bucket by combining the event coordinates of the inherent feature on the back of the bucket in the event camera coordinate system.

[0120] Specifically, in calculating the depth of a certain point in the inherent features of the bucket back. z This is achieved through formula (5):

[0121] (5);

[0122] In formula (5), b This represents the baseline distance between the optical centers of the event camera and the shortwave infrared camera. This indicates the unified focal length of the event camera and the shortwave infrared camera after calibration. g The parallax is between the event camera and the shortwave infrared camera.

[0123] S53, based on the three-dimensional coordinates of the inherent features of the bucket's back and the bucket's prior three-dimensional rigid body model, estimate the bucket's actual spatiotemporal position at the current moment.

[0124] In one specific embodiment, based on the three-dimensional coordinates of the inherent features of the bucket back Combining the prior three-dimensional rigid body model of the bucket M A correspondence is established between the prior three-dimensional rigid body model of the bucket and the three-dimensional coordinates of the inherent features on the back of the bucket. Then, the actual spatiotemporal position of the bucket at the current moment is obtained based on the correspondence between the two. .

[0125] S6: Obtain the current pushing torque and lifting torque of the pushing motor shaft and lifting motor shaft of the equipment where the bucket is located, and determine whether the bucket is in an obstructed working condition based on the actual spatiotemporal position, pushing torque and lifting torque of the bucket at the current moment.

[0126] In this embodiment of the invention, the obstructed working conditions include hard rock obstruction, large rock obstruction, and overload obstruction. This step involves determining whether the bucket is currently in a hard rock obstruction, large rock obstruction, or overload obstruction condition. If the bucket is not currently in any of these conditions, it is determined that the bucket is currently in a normal working condition.

[0127] In one specific embodiment, S6 includes:

[0128] S61: Based on the torque encoders pre-installed on the push motor shaft and the lifting motor shaft of the equipment where the bucket is located, the push torque and lifting torque of the equipment at the current moment are obtained respectively.

[0129] In one specific embodiment, a torque encoder is installed on the push motor shaft of the bucket-mounted device, which can detect the push torque of the bucket-mounted device in real time; a torque encoder is installed on the lifting motor shaft of the bucket-mounted device, which can detect the lifting torque of the bucket-mounted device in real time; after detecting the push torque and lifting torque, the torque encoder transmits the push torque and lifting torque to the electronic device in real time, and the electronic device obtains the push torque and lifting torque of the bucket-mounted device at the current moment by receiving the push torque and lifting torque.

[0130] S62 calculates the horizontal digging resistance and vertical lifting resistance of the bucket at the current moment based on the pushing torque and lifting torque, respectively.

[0131] Among them, the current horizontal digging resistance of the bucket The calculation formula is: ; Indicates the pushing torque; Indicating the energy conversion efficiency of the push-pressure method, the embodiments of the present invention are set as follows: The value range is [0.9-1], preferably 0.95; This indicates the radius (or effective working radius) of the push mechanism gear.

[0132] The current vertical lifting resistance of the bucket The calculation formula is: ;in, This indicates an increase in torque; To improve conversion efficiency, the embodiments of the present invention are configured as follows: The value range is [0.9-1], preferably 0.95; This indicates the radius of the lifting mechanism drum (or the radius of the drum on which the rope is wound).

[0133] S63 determines whether the bucket is currently in an obstructed working condition based on the bucket's actual spatial and temporal position, horizontal digging resistance, and vertical lifting resistance.

[0134] Among these, the bucket operating characteristics under hard rock obstruction conditions are continuously high resistance and low rates of change in forward and upward displacement. The bucket operating characteristics under large rock obstruction conditions are relatively high resistance with significant fluctuations (large variance). The bucket operating characteristics under overload obstruction conditions are continuously high average lifting resistance and low rates of change in upward displacement.

[0135] Based on the above, the determination of whether the bucket is currently in one of these three obstructed working conditions is achieved through steps S631 to S633 respectively:

[0136] S631, if the horizontal digging resistance of the bucket at the current moment is greater than the empirical threshold for hard rock digging resistance, and the displacement change rate of the bucket in the upward and forward directions is less than the first displacement change rate threshold, then it is determined that the bucket is currently in a hard rock obstruction condition.

[0137] Specifically, the hard rock obstruction condition is expressed by formula (6):

[0138] (6);

[0139] In formula (6), This represents the empirical threshold for hard rock excavation resistance, which is obtained through statistical analysis of historical data. , This represents the average horizontal excavation resistance under normal working conditions, and is taken as the average horizontal excavation resistance under the most recent 100 normal working conditions. This represents the variance of horizontal excavation resistance under normal operating conditions, and its value is the mean variance of horizontal excavation resistance under the most recent 100 operating conditions. a This represents the threshold value for the first displacement change rate, and its specific value is determined empirically. In this embodiment of the invention, it is preferably set as follows: a The value is 0.05; y The direction is directly above the bucket. z The direction is directly in front of the bucket.

[0140] S632, if the variance of the horizontal digging resistance of the bucket at the current moment is greater than the preset variance threshold, and the horizontal digging resistance of the bucket at the current moment is greater than the preset hard rock digging resistance threshold, then it is determined that the bucket is currently in a working condition where it is obstructed by large rocks.

[0141] Specifically, the hard rock obstruction condition is expressed by formula (7):

[0142] (7);

[0143] In formula (7), Var () represents the variance function; This represents a preset variance threshold, which is set in this embodiment of the invention. ; This indicates the preset hard rock excavation resistance threshold, which is set empirically to 0.85 times the empirical hard rock excavation resistance threshold.

[0144] S633, if the vertical lifting resistance of the bucket at the current moment is greater than the preset overload resistance upper limit threshold, and the displacement change rate of the bucket in the upward direction is less than the second displacement change rate threshold, then it is determined that the bucket is currently in an overloaded and obstructed working condition.

[0145] Specifically, the overload and obstructed operating condition is expressed by formula (8):

[0146] (8);

[0147] In formula (8), This indicates the preset overload resistance upper limit threshold, and , This is the maximum allowable torque for the lifting motor; f This represents the threshold value for the second displacement change rate, which is set empirically. Preferably, in this embodiment of the invention, it is set as follows: f The value is 0.1.

[0148] S7. If the bucket is currently in an obstructed working condition, obtain a preset trajectory planning sequence including the target spatiotemporal position of the bucket at the current moment, and adjust the preset trajectory planning sequence according to the obstructed working condition type of the bucket at the current moment.

[0149] It should be noted that before planning the obstructed bucket trajectory according to the method provided in this embodiment of the invention, the trajectory planning neural network obtained through pre-training is used as the bucket planning trajectory. Each time the trajectory planning neural network plans the bucket trajectory, it outputs a trajectory planning sequence. This sequence includes a preset number of target spatiotemporal locations and their timestamps. The target spatiotemporal locations refer to the ideal spatiotemporal locations pre-planned for the bucket. The trajectory planning neural network is typically trained by a convolutional neural network using massive amounts of bucket trajectory data during normal excavation. The bucket trajectory (including several actual spatiotemporal locations) for a period of time prior to the current moment is input into the trajectory planning neural network, which then outputs a trajectory planning sequence for a future period.

[0150] Based on the above, in a specific embodiment, step S7 includes:

[0151] S71, if the bucket is currently in an obstructed working condition, a preset trajectory planning sequence including the target spatiotemporal position of the bucket at the current moment is obtained through a pre-trained trajectory planning neural network.

[0152] Specifically, the preset trajectory planning sequence includes target spatiotemporal positions that the bucket has already traveled before the current moment and target spatiotemporal positions that are in the future moment at the current moment, for a total of a preset number of target spatiotemporal positions.

[0153] It should be noted that establishing the relationship between the bucket's current actual spatiotemporal position and the target spatiotemporal position can be achieved by matching timestamps. For example, the current moment can be matched with the timestamp of each target spatiotemporal position in the preset trajectory planning sequence, and the target spatiotemporal position with the smallest time difference can be selected as the bucket's current target spatiotemporal position. Alternatively, it can be achieved by distance matching. For example, the distance between the bucket's current actual spatiotemporal position and each target spatiotemporal position in the preset trajectory planning sequence can be calculated, and the target spatiotemporal position with the smallest distance can be selected as the bucket's current target spatiotemporal position.

[0154] S72, takes the target spatiotemporal position of the bucket at the current moment as the obstruction point, and determines the position of the obstruction point in the preset trajectory planning sequence.

[0155] The location of the obstructed point in the preset trajectory planning sequence is indicated by its number. The preset number of target spatiotemporal locations in the preset trajectory planning sequence are numbered 1, 2, ... P .

[0156] S73 adjusts the target spatiotemporal position starting from the obstruction point in the preset trajectory planning sequence according to the obstruction condition type of the bucket at the current moment and the position of the obstruction point in the preset trajectory planning sequence.

[0157] In one specific embodiment, depending on the type of obstructed operating condition, S73 may include the following three cases in its implementation:

[0158] The first scenario: If the current obstruction condition of the bucket is hard rock, then the first th ... k The spatiotemporal location of each target is adjusted:

[0159] (1);

[0160] In formula (1), Indicates the first in the preset trajectory planning sequence kThe spatiotemporal location of the target Indicates the adjusted number k The spatiotemporal location of the target; This represents the lateral offset coefficient, and This indicates that the lateral offset of the bucket is generated according to the change in the horizontal digging resistance of the bucket at the current moment; This indicates the current horizontal digging resistance of the bucket. This represents the empirical threshold for hard rock excavation resistance. This indicates the amount by which the horizontal digging resistance of the bucket exceeds the empirical threshold for hard rock digging resistance at the current moment; This represents the lift coefficient, with a default value of 0.02 m / kN; The physical meaning is the lift height caused by the excess drag; Indicates the adjustment factor; K This indicates the position number of the obstructed point in the preset trajectory planning sequence. P This indicates the number of target spatiotemporal locations included in the preset trajectory planning sequence, i.e., the preset number. x The right side of the bucket. y The direction directly above the bucket. z The direction directly in front of the bucket.

[0161] and The physical meaning is: (1) the bucket is at the point of obstruction ( k = K (1) No sudden changes, to avoid causing the bucket to move instantaneously and generate impact. (2) The bucket is at the target point (the last target spatiotemporal position of the preset trajectory planning sequence), that is , Setting the value to 0 ensures that the bucket returns to the target spatial and temporal position defined by the preset trajectory planning sequence at the target point, thereby guaranteeing the accuracy of the excavation operation.

[0162] The adjustment method achieves the following effect: a smooth shift from the obstructed point to the original planned target spatiotemporal position at the target point. The time offset is the largest. The overall effect is that the bucket swings laterally in the hard rock area and gradually rises to get out of the hard rock area. This method can ensure that the trajectory is continuous and smooth at the obstruction point and the target point, avoiding sudden speed changes.

[0163] It should be noted that when K equal P When the obstruction point is the last target spatiotemporal position in the preset trajectory planning sequence, the present invention embodiment does not adjust the target spatiotemporal position by default.

[0164] The second scenario: If the current obstruction condition of the bucket is a large rock obstruction condition, then the first step in the preset trajectory planning sequence is calculated using formula (2). k The spatiotemporal location of each target is adjusted:

[0165] (2);

[0166] In formula (2), This indicates the displacement of the boom retraction, and ; Indicates the oscillation frequency. Indicates the amplitude of the swing. exp ( ) represents an exponential function. This represents the variance of horizontal excavation resistance under normal operating conditions. f_swing The default value is 1.5Hz. p_swing The default value is 0.25m.

[0167] The physical meaning is: when The maximum amount of stick retraction is when the stick is pulled back, its value is ;when At that time, the stick retraction was approximately 0 (natural attenuation).

[0168] The adjustment methods can achieve the following effects: quickly get out of the stuck area of ​​a large rock by retreating; try to loosen or go around the large rock by swinging laterally.

[0169] The third scenario: If the current obstruction condition of the bucket is an overload obstruction condition, then the first digit in the preset trajectory planning sequence is calculated using formula (3). k The spatiotemporal location of each target is adjusted:

[0170] (3);

[0171] In formula (3), This indicates the upper limit threshold of overload resistance. This indicates the vertical lifting resistance of the bucket at the current moment. This represents the overload factor. Wherein, The properties of the tanh function are: . The physical meaning is: the lift of the obstructed point is 0; when At that time, the lifting amount of the bucket is ;when At that time, the elevation of the target point is .

[0172] The adjustment method achieves the following effect: the bucket is raised gradually, that is, the bucket is not raised initially, and the bucket is raised more as the digging process progresses. Moreover, the greater the bucket overload, the greater the lifting range.

[0173] In summary, the bucket obstruction trajectory planning method based on the fusion of broadband infrared and event camera information provided in this embodiment of the invention has the following beneficial effects:

[0174] (1) By fusing the information collected by the broadband infrared camera and the event camera, the influence of light changes or dust environment on the judgment of the actual spatiotemporal position of the bucket is overcome based on the complementarity of multi-sensor information; the region of interest of the bucket is obtained by the short-wave infrared camera and the thermal infrared camera and the inherent features of the back of the bucket are extracted by the event camera, providing an accurate data basis for the subsequent estimation of the actual spatiotemporal position of the bucket.

[0175] (2) By combining the relative positional relationship of the shortwave infrared camera, thermal infrared camera and event camera with the coordinates of the inherent features of the bucket back in different camera coordinate systems, the actual spatiotemporal position of the bucket at the current moment is estimated, and a robust and occlusion-resistant method for estimating the real-time actual spatiotemporal position of the bucket is constructed, thereby improving the accuracy of the estimation of the actual spatiotemporal position of the bucket.

[0176] (3) Based on the torque encoders on the push motor shaft and the lifting motor shaft, real-time, direct, and comprehensive push torque and lifting torque are provided, overcoming the delay and inaccuracy problems of relying on external static sensor scanning. Differentiated trajectory adjustment strategies are proposed for different types of obstructed working conditions, realizing the formulation of different bucket trajectory adjustment strategies according to the type of obstructed working conditions. This can significantly improve the adaptability of the bucket under complex and non-homogeneous obstructed working conditions, realize the operation of executing corresponding strategies according to different types of obstructed working conditions, and improve the accuracy of bucket trajectory planning under obstructed working conditions.

[0177] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for planning the obstructed trajectory of a bucket by fusing broadband infrared and event camera information, characterized in that, A broadband infrared camera and an event camera are installed below the boom of the equipment where the bucket is located. The broadband infrared camera includes a short-wave infrared camera and a thermal infrared camera, and the event camera, the short-wave infrared camera and the thermal infrared camera are temporally and spatially aligned. S1, acquire the short-wave infrared image and thermal infrared image collected by the short-wave infrared camera and thermal infrared camera at the current moment, respectively, and acquire the spatiotemporal event stream collected by the event camera at the current moment; S2, Extract the region of interest of the bucket from shortwave infrared images and / or thermal infrared images; S3: Filter the event streams corresponding to the region of interest of the bucket from the spatiotemporal event streams as valid event streams; S4, extract the inherent features of the bucket back from the valid event stream; S5. Based on the inherent features of the bucket's back and the relative positional relationship between the shortwave infrared camera and the event camera, estimate the actual spatiotemporal position of the bucket at the current moment. S6, obtain the current pushing torque and lifting torque of the pushing motor shaft and lifting motor shaft of the equipment where the bucket is located, and determine whether the bucket is in an obstructed working condition based on the actual spatiotemporal position, pushing torque and lifting torque of the bucket at the current moment; S7. If the bucket is currently in an obstructed working condition, obtain a preset trajectory planning sequence including the target spatiotemporal position of the bucket at the current moment, and adjust the preset trajectory planning sequence according to the obstructed working condition type of the bucket at the current moment. Wherein, S2 includes: S21, extract the first bucket region of interest from the shortwave infrared image based on the reflectance of each pixel in the shortwave infrared image, and calculate the average reflectance of the first bucket region of interest. S22, extract the region of interest of the second bucket from the thermal infrared image based on the gray value of each pixel, and calculate the average gray value of the region of interest of the second bucket. S23, when the average reflectance of the region of interest of the first bucket is greater than the preset reflectance threshold, the region of interest of the first bucket is taken as the region of interest of the bucket; when the average gray value of the region of interest of the second bucket does not exceed the preset gray value threshold, the region of interest of the second bucket is taken as the region of interest of the bucket; when the average reflectance of the region of interest of the first bucket is not greater than the preset reflectance threshold, and the average gray value of the region of interest of the second bucket exceeds the preset gray value threshold, the intersection of the region of interest of the first bucket and the region of interest of the second bucket is taken as the region of interest of the bucket.

2. The bucket obstruction trajectory planning method based on broadband infrared and event camera information fusion according to claim 1, characterized in that, The inherent feature consists of three horizontally arranged rectangles, each with a preset height and a preset spacing between adjacent rectangles. S4 includes: S41, extract the top and bottom horizontal edges of the rectangle from the valid event stream; S42, detect the six vertical edges of three rectangles in the valid event stream within the coverage area of ​​the upper and lower horizontal edges; S43, perform rectangle verification based on the upper horizontal edge, lower horizontal edge, six vertical edges, preset width and preset height, and when the verification is successful, use the features of the three rectangles as inherent features of the bucket back.

3. The bucket obstruction trajectory planning method based on broadband infrared and event camera information fusion according to claim 2, characterized in that, S41 includes: S411, bucket the ordinate of each pixel in the valid event stream in the event camera coordinate system. S412, count the total number of events with positive and negative polarities within each bucket on the vertical axis; S413, if the total number of events with positive and negative polarities within any vertical axis bucket is greater than a preset high-density peak value, then mark the vertical axis bucket as a candidate horizontal edge; S414, Pair all candidate horizontal edges according to the preset height to obtain the upper and lower horizontal edges in the inherent features of the bucket back.

4. The bucket obstruction trajectory planning method based on broadband infrared and event camera information fusion according to claim 2 or 3, characterized in that, S42 includes: S421, perform horizontal coordinate bucketing on the horizontal coordinates of each pixel in the effective event stream within the coverage area of ​​the upper and lower horizontal edges, in the event coordinates of the event camera event coordinate system. S422, count the total number of events with positive and negative polarities within each horizontal axis bucket; S423, select the six horizontal axis buckets with the largest total number of events with positive and negative polarities as the six vertical edges in the inherent features of the bucket back.

5. The bucket obstruction trajectory planning method based on broadband infrared and event camera information fusion according to claim 1, characterized in that, S5 includes: S51. Based on the relative positional relationship between the shortwave infrared camera and the event camera, as well as the event coordinates of the inherent features of the bucket back in the event camera coordinate system and the shortwave infrared coordinates in the shortwave infrared camera coordinate system, calculate the parallax between the event camera and the shortwave infrared camera. S52, calculate the depth of the inherent feature on the back of the bucket based on the parallax between the event camera and the shortwave infrared camera and the relative positional relationship between the shortwave infrared camera and the event camera, and determine the three-dimensional coordinates of the inherent feature on the back of the bucket based on the event coordinates of the inherent feature on the back of the bucket in the event camera coordinate system. S53, based on the three-dimensional coordinates of the inherent features of the bucket's back and the bucket's prior three-dimensional rigid body model, estimate the bucket's actual spatiotemporal position at the current moment.

6. The bucket obstruction trajectory planning method based on broadband infrared and event camera information fusion according to claim 1, characterized in that, S6 includes: S61, based on the torque encoders pre-installed on the push motor shaft and the lifting motor shaft of the equipment where the bucket is located, obtain the push torque and lifting torque of the equipment at the current moment; S62, calculate the horizontal digging resistance and vertical lifting resistance of the bucket at the current moment based on the pushing torque and lifting torque respectively; S63 determines whether the bucket is currently in an obstructed working condition based on the bucket's actual spatial and temporal position, horizontal digging resistance, and vertical lifting resistance.

7. The bucket obstruction trajectory planning method based on broadband infrared and event camera information fusion according to claim 6, characterized in that, The obstructed working conditions include hard rock obstruction conditions, large rock obstruction conditions, and overload obstruction conditions. S63 includes: S631, if the horizontal digging resistance of the bucket at the current moment is greater than the empirical threshold for hard rock digging resistance, and the displacement change rate of the bucket in the upward and forward directions is less than the first displacement change rate threshold, then it is determined that the bucket is currently in a hard rock obstruction condition. S632, if the variance of the horizontal digging resistance of the bucket at the current moment is greater than the preset variance threshold, and the horizontal digging resistance of the bucket at the current moment is greater than the preset hard rock digging resistance threshold, then it is determined that the bucket is currently in the condition of being obstructed by large rocks. S633, if the vertical lifting resistance of the bucket at the current moment is greater than the preset overload resistance upper limit threshold, and the displacement change rate of the bucket in the upward direction is less than the second displacement change rate threshold, then it is determined that the bucket is currently in an overloaded and obstructed working condition.

8. The bucket obstruction trajectory planning method based on broadband infrared and event camera information fusion according to claim 1, characterized in that, S7 includes: S71, if the bucket is currently in an obstructed working condition, a preset trajectory planning sequence including the target spatiotemporal position of the bucket at the current moment is obtained through a pre-trained trajectory planning neural network. Each time the trajectory planning neural network performs bucket trajectory planning, it outputs a trajectory planning sequence, which includes a preset number of target spatiotemporal positions. S72, take the target spatiotemporal position of the bucket at the current moment as the obstruction point, and determine the position of the obstruction point in the preset trajectory planning sequence; S73 adjusts the target spatiotemporal position starting from the obstruction point in the preset trajectory planning sequence according to the obstruction condition type of the bucket at the current moment and the position of the obstruction point in the preset trajectory planning sequence.

9. The bucket obstruction trajectory planning method based on broadband infrared and event camera information fusion according to claim 8, characterized in that, S73 includes: If the current obstruction condition of the bucket is hard rock, then the first th ... k The spatiotemporal location of each target is adjusted: （1）； In formula (1), Indicates the first in the preset trajectory planning sequence k The spatiotemporal location of the target Indicates the adjusted number k The spatiotemporal location of the target Indicates the lateral offset coefficient. Indicates the lifting coefficient. Indicates the adjustment factor. This indicates the current horizontal digging resistance of the bucket. This represents the empirical threshold for hard rock excavation resistance. K This indicates the position number of the obstructed point in the preset trajectory planning sequence. P The preset number indicates the number of target spatiotemporal locations included in the preset trajectory planning sequence. x The right side of the bucket. y The direction directly above the bucket. z The direction directly in front of the bucket; If the current obstruction condition of the bucket is large rock obstruction, then the first step in the preset trajectory planning sequence is calculated using formula (2). k The spatiotemporal location of each target is adjusted: （2）； In formula (2), This indicates the displacement of the boom retraction. Indicates the oscillation frequency. Indicates the amplitude of the swing. exp ( ) represents an exponential function. This represents the variance of horizontal excavation resistance under normal operating conditions; If the current obstruction condition of the bucket is an overload obstruction condition, then the first th ... k The spatiotemporal location of each target is adjusted: （3）； In formula (3), This indicates the upper limit threshold of overload resistance. This indicates the vertical lifting resistance of the bucket at the current moment. This indicates the overload factor.

Images