Environment sensing method for remote spading assistance of loader

Abstract

The invention discloses an environment sensing method for remote spading assistance of a loading machine. The method comprises the following steps that sensitive obstacles around the loading machine are recognized based on image information of the surrounding environment of the loading machine; the method comprises the following steps: constructing point cloud data of a to-be-shoveled stack based on loader radar feedback data, preprocessing the point cloud data, and calculating a stack contour line array sequence and a section line array sequence of the to-be-shoveled stack based on the preprocessed point cloud data; the information of the sensitive obstacle and the contour information are packaged and transmitted back to a remote control center, and the remote control center displays the contour of the to-be-shoveled stack at multiple angles based on the transmitted-back information; sensitive obstacles around the loader and the outline of the stack can be sensed, the side view and the top view of the loader and the stack are generated at the far end, a remote operator is helped to know comprehensive information around the loader more visually, and the remote operation efficiency, the operation accuracy and the operation safety are improved.

Description

Technical Field

[0001] The present invention relates to the field of environmental perception technology. Specifically, the present invention is applied to the field of loader remote control, and in particular to an environmental perception method for loader remote excavation assistance. Background Art

[0002] At present, loaders are widely used in construction, mining, road construction and other fields. They are important engineering machinery equipment. The core functions of loaders include: loading, transporting and unloading materials.

[0003] The traditional loader control solution is on-site manual driving or control solution, but with the continuous development of remote communication technology, remote control loaders are gradually being applied to work scenarios in harsh environments to avoid direct exposure of workers to dangerous environments. Specifically, when performing shoveling operations, the loader operator needs to judge the subsequent control operations of the loader based on the current state of the pile of materials to be shoveled. At the same time, the loader operator also needs to observe whether there are sensitive obstacles such as people or vehicles around the loader to ensure the safety of the shoveling operation, prevent dangerous situations such as collisions, and ensure the efficiency of the shoveling operation.

[0004] Therefore, in the above context, a loader remote control solution based on environmental perception came into being. Such a solution can effectively feedback and display the environmental information around the material stack and the loader, reduce the difficulty of remote operations, and improve the efficiency and safety of remote operations.

[0005] However, the existing remote control solutions for loaders mainly focus on remote vehicle motion control or bucket control. The operator observes the environment by transmitting images back through cameras around the loader. This solution has significant limitations. In the remote control mode of this solution, the operator can only rely on the original images transmitted back, which makes it difficult for the operator to accurately judge the actual situation of the stack and the working environment. There may be errors in the judgment of the stack distance, which will lead to the wrong selection of the excavation point and accidents such as stack collapse, affecting work efficiency and posing corresponding safety risks. In addition, there is a delay in image transmission, and it cannot be updated with real-time changing environmental factors. It is easily affected by dust in the environment and blocked, which will also affect the operator's judgment. For example, even if the clarity of the image transmitted back by the camera is extremely high, a large amount of dust in the environment may seriously block the camera's field of view, increasing the risk of misjudgment and the probability of misoperation of the remote operator.

[0006] More specifically, the existing remote control solutions for loaders have the following typical cases:

[0007] 1. A remote-controlled loader steering device disclosed in Chinese patent CN202222481556.X, and a remote-controlled cockpit for a loader disclosed in Chinese patent CN202321299286.9. Such solutions focus on the performance or design of the remote operating device itself, and do not consider how to accurately handle environmental information such as material stacking.

[0008] 2. Chinese patent CN202211561938.1 discloses a wireless communication-connected loader device and operator, which is equipped with an image collector connected to a wireless signal transmitter, which can meet the user's needs for remote operation of the loader; Chinese patent CN202211609915.3 discloses a loader operation guidance system for non-line-of-sight remote control driving, which collects vehicle data and vehicle-mounted sensor data in real time through a vehicle-mounted terminal and a vehicle-mounted environmental sensing module, and uploads data to a human-computer interaction module through a wireless communication module to build a guidance model; this type of solution focuses on the research of wireless transmission and surrounding environment image acquisition, and does not pay attention to auxiliary technologies such as the outer contour perception of material stacking.

[0009] 3. Chinese patent CN117344819A discloses an intelligent sensing system and method for a loader. This solution moves the robotic arm so that the sensing device at the end of the robotic arm can reach any position within the maximum radius of the robotic arm, thereby expanding the sensing range. However, there may be delays in the movement of the robotic arm and the adjustment of the sensing device, which will also affect the operating efficiency and accuracy, and the driver cannot better understand the current operating conditions to assist in excavation. Although this type of solution improves the sensing range, it does not involve specific methods for sensing the outer contour of the material stack.

[0010] In summary, the existing remote control solutions for loaders have not conducted research on the contour perception of material stacks and the refined processing of environmental information. The existing solutions lack a certain degree of accuracy and safety and cannot stably guarantee the operating efficiency and accuracy of the loader under remote control. Summary of the invention

[0011] The object of the present invention is to provide an environmental perception method for remote excavation assistance of a loader, thereby solving all or one of the above-mentioned problems existing in the prior art.

[0012] In order to solve the above technical problems, the specific technical solutions of the present invention are as follows:

[0013] The present invention provides an environment perception method for remote excavation assistance of a loader, comprising the following steps:

[0014] Steps for initial environmental perception:

[0015] Identify sensitive obstacles around the loader based on the image information of the loader's surrounding environment;

[0016] Stacking perception steps:

[0017] constructing point cloud data for the pile to be excavated based on the radar feedback data of the loader, preprocessing the point cloud data, and calculating the outline information of the pile to be excavated based on the preprocessed point cloud data;

[0018] The outline information includes: a stacked contour line array sequence and a profile line array sequence;

[0019] Data return steps:

[0020] The information of the sensitive obstacle and the outline information are packaged and transmitted back to a remote control center, and the remote control center displays the outline of the pile to be excavated from multiple angles based on the transmitted information.

[0021] In one embodiment, the preprocessing of the point cloud data and calculating the outline information of the pile to be excavated based on the preprocessed point cloud data includes:

[0022] Performing filtering processing on the point cloud data;

[0023] Eliminating bucket data from the filtered point cloud data to obtain stacking point cloud data;

[0024] A greedy projection algorithm is used to reconstruct the surface of the stacked point cloud data to obtain a triangulated mesh;

[0025] Based on the triangulated grid and the linear interpolation algorithm, determine the intersection of the contour plane of each sub-stack in the to-be-excavated stack and each edge in the triangulated grid, and connect a number of the intersections in order to obtain a contour array sequence of the to-be-excavated stack;

[0026] Based on the acquisition logic of the contour line array sequence, the section line array sequence is acquired.

[0027] In one embodiment, the surface reconstruction of the stacked point cloud data using a greedy projection algorithm to obtain a triangulated mesh includes:

[0028] For each group of sub-stacked point cloud data in the stacked point cloud data, initializing an empty grid;

[0029] For each group of sub-stacked point cloud data in the stacked point cloud data, sequentially performing optimal point selection, valid triangle determination, and valid triangle construction cycles based on the empty grid;

[0030] The optimal point selection includes: using a greedy projection algorithm to determine three optimal points in the point cloud data of the sub-stack, and judging whether the three optimal points satisfy the formation of a valid triangle;

[0031] The valid triangle determination includes: in response to the three optimal points being able to form a valid triangle, adding the formed valid triangle as a newly created triangle to the empty mesh;

[0032] The effective triangle construction loop includes: in response to the newly created triangle being added to the empty mesh, removing the three best points of the stacked point cloud data that have been used to construct the effective triangle, continuing to select new best points from the stacked point cloud data to continue to construct the effective triangle, and looping; in response to the three best points failing to satisfy the formation of the effective triangle, jumping out of the loop; based on the loop, expanding into the stacked point cloud data set until the stacked point cloud data is empty or fails to form a valid triangle, ending the loop to obtain the connected triangulated mesh;

[0033] The triangulated mesh is used to simulate the outer surface of the pile to be excavated.

[0034] In one embodiment, based on the triangulated grid and a linear interpolation algorithm, the intersection of the contour plane of each sub-stack in the to-be-excavated stack and each edge in the triangulated grid is determined, and a plurality of the intersections are sequentially connected to obtain a contour array sequence of the to-be-excavated stack, including:

[0035] Setting a number of reference elevation values, and calculating the elevation value of the contour line of each sub-stack in the stack to be excavated based on the reference elevation values;

[0036] For each sub-stack, the plane where the contour line is located and each triangle in the triangulated mesh are respectively judged to intersect based on the corresponding elevation value, and the intersection point is calculated when they intersect;

[0037] Calculate the angle information corresponding to the intersection vector based on the intersection point;

[0038] sorting the plurality of intersection points based on the angle information;

[0039] The contour line array sequence is constructed based on the sorted intersection points.

[0040] In one embodiment, the method further comprises: determining the intersection of the contour plane of each sub-stack in the to-be-excavated stack with each edge in the triangulated grid based on the triangulated grid and the linear interpolation algorithm, and sequentially connecting a plurality of the intersections to obtain a contour array sequence of the to-be-excavated stack, and further comprising:

[0041] Traversing each triangle in the triangulated mesh in turn, checking the elevation values ​​of three vertices of each triangle, and determining the elevation value at a specific contour level;

[0042] Determine the intersection points of the plane where the contour line is located and the sides of the triangles across different contour line levels by interpolation calculation, calculate the intersection points, and record a number of the intersection points constituting a contour line;

[0043] Create an intersection storage list, and use the intersection storage list to store the intersections at the heights corresponding to the reference elevation values; determine the intersection vectors of the intersections, calculate the angle formed by the intersection vector and the positive semi-axis of the Y-axis of the vehicle coordinate system, store the angle in the information of the corresponding intersection, and determine whether the intersection is located on the right side, left side, or center of the loader according to the positive value, negative value, or zero value of the angle;

[0044] According to the angle information respectively corresponding to the intersections in the intersection storage list, the intersections are sorted in a directed ascending order with respect to the angle information, and the sorted intersections are used as the contour line array sequence.

[0045] In one implementation manner, the intersection judgment strategy includes:

[0046] Define the contour line of the sub-stack as H i ;

[0047] The coordinates of the two vertices of the selected side of the selected triangle are (x A ,y A ,z A )、(x B ,y B ,z B );

[0048] The elevation values ​​corresponding to the two vertices of the selected side of the selected triangle are z A 、z B ;

[0049] When (z A -H i )(z B -H i )>0, determining that the plane where the contour line of the sub-stack is located does not intersect with the selected triangle;

[0050] When (z A -H i )(z B -H i )<0, determining that the plane where the contour line of the sub-stack is located intersects with the selected triangle;

[0051] The intersection calculation strategy includes:

[0052] In response to the plane where the contour lines of the sub-stack are located intersecting with the selected triangle, a linear interpolation algorithm is used to calculate the intersection of the plane where the contour lines of the sub-stack are located and the selected side of the selected triangle, and the intersection calculation formula is:

[0053]

[0054] Wherein, x is the horizontal coordinate of the intersection point, y is the vertical coordinate of the intersection point, H i as the ordinate of the intersection point.

[0055] In one embodiment, the acquisition logic based on the contour line array sequence is used to acquire the profile line array sequence, including:

[0056] Determining a section that passes through the pile to be excavated based on the triangulated mesh and the intersection storage list;

[0057] Traversing each triangle in the triangulated mesh, determining whether the edge of each triangle intersects with the section; in response to the intersection, using a linear interpolation algorithm to calculate the section intersection and the position information of the section intersection to obtain a plurality of section intersections;

[0058] According to the values ​​of the ordinates of the section intersections, the section intersections are ordered from small to large, and the ordered section intersections are used as the section line array sequence.

[0059] In one embodiment, the information of the sensitive obstacle and the outline information are packaged and transmitted back to a remote control center, and the outline of the pile to be excavated is displayed at multiple angles in the remote control center based on the transmitted information, including:

[0060] Packaging the sensitive obstacle information and the outline information in a standard data format;

[0061] Send the packaged data packets to the remote control center server of the loader;

[0062] The received data packet is unpacked by the remote control center server, and the unpacked data is converted and processed and visualized.

[0063] In one embodiment, after sending the packaged data packet to the remote control center server of the loader, the method further includes:

[0064] In response to the remote control center server receiving the data packet, sending a reception confirmation signal to the loader;

[0065] In response to the loader not receiving the confirmation signal within the time period, the data packet is resent to the remote control center server until the loader receives the confirmation signal.

[0066] In one embodiment, the converting and visualizing the unpacked data includes:

[0067] Create top-view and side-view images of the loader's surroundings;

[0068] Projecting the three-dimensional points stored in the unpacked contour array sequence to the top view image; projecting the three-dimensional points stored in the unpacked profile array sequence to the side view image; in response to the projection of the three-dimensional points of the contour array and the three-dimensional points of the profile array, connecting lines in the top view image and the side view image in the existing order of a number of the three-dimensional points to obtain stacked contour lines and stacked profile lines in the top view image and the side view image;

[0069] A loader bucket simulation image is constructed in the top view image and the side view image.

[0070] The beneficial effects of the technical solution of the present invention are:

[0071] 1. The environmental perception method for remote excavation assistance of a loader described in the present invention can accurately process point cloud data, and then effectively and efficiently extract the contour lines and profile lines of the stacking surface, and express the outline information of the material stack in a concise, efficient and intuitive manner, providing accurate and powerful environmental perception support for remote operation.

[0072] 2. The environmental perception method for remote excavation assistance of a loader described in the present invention uses local calculation to process contour lines and profile lines, which can not only clearly display the material outline information, but also greatly reduce the size of remote return data, facilitate real-time data transmission, and ensure that the image of the remote control center can be displayed smoothly in real time, greatly improving work efficiency, work safety and work stability. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0074] Figure 1 It is a flow chart of the environment perception method for remote excavation assistance of a loader according to Embodiment 1 of the present invention;

[0075] Figure 2 It is a logical flow diagram of the environment perception method for loader remote excavation assistance according to Embodiment 1 of the present invention;

[0076] Figure 3 It is a schematic diagram of reconstruction of the triangulated mesh in the environment perception method for remote excavation assistance of a loader according to Example 1 of the present invention;

[0077] Figure 4 It is a schematic diagram of the intersection of contour lines in the environment perception method for remote excavation assistance of a loader described in Example 1 of the present invention;

[0078] Figure 5 It is the schematic diagram of the intersection of the section lines in the environment perception method for remote excavation assistance of a loader described in Example 1 of the present invention;

[0079] Figure 6 is a schematic diagram of the image coordinate system in the environment perception method for remote excavation assistance of a loader described in Example 1 of the present invention;

[0080] Figure 7 It is a schematic diagram of a real operation scene of a loader in the environment perception method for remote excavation assistance of a loader according to Embodiment 1 of the present invention;

[0081] Figure 8 It is a schematic diagram of a simulated loader operation scene in a display image of the remote control center server in the environment perception method for remote excavation assistance of a loader according to Example 1 of the present invention;

[0082] Fig. 9 It is a schematic diagram of the architecture of the environment perception system for remote excavation assistance of a loader according to Embodiment 2 of the present invention;

[0083] Fig.10 This is a schematic diagram of the arrangement of sensors in the environment perception system for remote excavation assistance of a loader according to Embodiment 2 of the present invention;

[0084] Fig.11 It is a schematic diagram of the environment perception system for remote excavation assistance of a loader according to Embodiment 2 of the present invention, which is fixed to the loader body coordinate system;

[0085] The markings in the accompanying drawings are as follows:

[0086] 1. Radar; 2. Main camera; 3. Surround view camera. DETAILED DESCRIPTION

[0087] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby making a clearer and more definite definition of the protection scope of the present invention.

[0088] In the description of the present invention, it should be noted that the embodiments described in the present invention are only part of the embodiments of the present invention, rather than all of the embodiments; based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present invention.

[0089] The terms "first", "second", etc. in the specification and claims of this article and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of this article described here can be implemented in an order other than those illustrated or described here. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, device, product or equipment that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0090] Example 1

[0091] This embodiment provides an environment perception method for remote excavation assistance of a loader, such as Figures 1 to 8 As shown, the following steps are included:

[0092] The core function of this method is: Figure 2 As shown, a frequency modulated continuous wave radar and a camera are used to sense the working environment around the loader, identify the relative position between the loader and the pile of excavated materials and the outer contour information of the pile, and generate a schematic diagram of the pile contour line; a side view and a top view of the loader and the pile are generated through three-dimensional projection technology, and the synchronously sensed sensitive obstacles such as people and vehicles around the loader and the pile information are transmitted back to the remote control center and rendered in the side view and top view for display, so as to help the remote operator to more intuitively understand the comprehensive information around the loader, greatly reduce the difficulty of remote operation of the loader, and improve the remote operation efficiency, operation accuracy and operation safety. The specific steps are as follows;

[0093] S100, initialization step:

[0094] In this step, each module implementing the method is powered on and initialized, as follows:

[0095] S101, after the system is powered on, start the FMCW radar, camera and other modules, initialize and set the started modules respectively, put the equipment in normal working state, and wait for subsequent remote control work;

[0096] Specifically, the initialization modules include but are not limited to: radar, main camera, surround view cameras (4), perception computing module, communication module and display module.

[0097] S200, initial environmental perception steps:

[0098] In this step, the sensitive obstacles in the environment around the loader are identified as follows:

[0099] S201, obtaining image information of the surrounding environment of the loader fed back by the camera;

[0100] In step S201, the collected image information includes: image information D of the surrounding environment collected by the main camera, and image information M of the surrounding environment collected by four surround view cameras (Camera1, Camera2, Camera3 and Camera4) around the loader body;

[0101] S202, identifying the image information and determining sensitive obstacles such as people, cars, and stones;

[0102] In step S202, an image recognition algorithm (such as YOLOv5) is used to identify obstacles in the image information M, and key information such as the position coordinates and confidence of the corresponding obstacles is recorded, and the recorded information is stored in a corresponding list;

[0103] In step S202, the obstacle recognition process is as follows:

[0104] In response to the recognition result being "human" and the confidence level being greater than a preset confidence level threshold, the key information corresponding to the result is stored in the ObstacleList_humans{} list;

[0105] In response to the recognition result being "car" or "stone", and the confidence being greater than a preset confidence threshold, key information corresponding to the result is stored in the ObstacleList_obstacles{} list.

[0106] S300, stacking perception steps:

[0107] In this step, point cloud data about the pile to be excavated is constructed based on the data fed back by the radar, and the point cloud data is processed accordingly to calculate the pile outline information (pile contour lines and profile lines), as follows:

[0108] S301, obtaining original point cloud data of the surrounding environment of the loader fed back by radar;

[0109] In step S301, the original point cloud data is defined as P, each point in P is defined as p, and the data of p includes its three-dimensional coordinates (x, y, z) in space and the speed v at the current moment; wherein P is processed by voxel filtering to reduce the point cloud density;

[0110] S302, filtering the point cloud data P;

[0111] In step S302, the speed threshold v is set according to the working environment. threshold , traverse each point p of P, when the velocity v≤v threshold When , the point p is added to P'; this process filters the point cloud of P based on the speed threshold, and its pseudo code is as follows:

[0112]

[0113] S303, removing the bucket data in the processed point cloud P' and segmenting the stacking point cloud data;

[0114] In step S303, cluster P' to obtain bucket angle information F, and calculate the loader bucket point cloud data S based on the above information; then remove S from P', and the processed P is used as the stacking point cloud data set T;

[0115] S304, triangulating the stacking point cloud T, and calculating the stacking contour lines and profile lines based on the triangulation results;

[0116] In step S304, as an optional method, the algorithm for calculating the stacking contour is as follows:

[0117] S3041, firstly, using a greedy projection algorithm to reconstruct the surface of the stacking point cloud T, generating a connected triangulated mesh, and using the triangulated mesh to simulate the outer surface of the stacking;

[0118] Specifically, for each set of stacked point cloud data in T, an empty mesh is initialized; for each set of stacked point cloud data in T, the optimal point selection, valid triangle determination, and valid triangle construction loop are performed in turn, as follows:

[0119] (1) Use the projection strategy, greedy strategy and breadth-first strategy in the greedy projection algorithm to find the best point (p1, p2, p3) and check whether the three best points can form a valid triangle;

[0120] (2) When the three best points can form a valid triangle, the valid triangle is added to the mesh as a newly created triangle;

[0121] (3) Then update the stacked point cloud data T, remove the three best points (p1, p2, p3) that have been used to construct a valid triangle, and continue to select new best points from T to continue to construct a triangle, and repeat this cycle; when the three best points cannot form a valid triangle, exit the cycle;

[0122] (4) Based on the above strategy, the set T is gradually expanded until T is empty or no valid triangles can be formed. The whole cycle ends and a connected triangulated mesh is finally generated, such as Figure 3 As shown; the pseudo code of the above process is as follows:

[0123]

[0124] S3042, then based on the triangulated grid and the linear interpolation algorithm, determine the intersection of the contour plane of each stack and each edge in the triangulated grid, connect the above intersections in order, and obtain the contour coordinate sequence of the entire stack;

[0125] Specifically, the elevation value of each stack contour line is calculated as follows:

[0126] Set several benchmark contour elevation values; according to formula H i =H0+n×d h Calculate the elevation value H corresponding to each contour line i , i∈[1,2,3……n]; in the formula, H0 is the set base contour height (elevation value), d h is the set contour interval, n is the number of intervals between the set reference contour and the calculated contour, and the set elevation value H i for Fig.10 The z-coordinate value in the coordinate system.

[0127] Specifically, based on the plane where the contour line is located and each triangle of the triangulated mesh, an intersection judgment is performed, and the intersection point is calculated when the intersection occurs; the angle information corresponding to the intersection vector is calculated based on the intersection point; the intersection points are sorted according to the angle information, and a contour line array sequence is constructed based on the sorted intersection points, as follows:

[0128] (1) traverse each triangle in the triangulated mesh in turn, check the elevation values ​​of the three vertices of each triangle, and determine the elevation value at a specific contour level;

[0129] (2) Determine the intersection of the plane where the contour line is located and the triangle edge across different contour line levels through interpolation calculation, calculate the intersection point, and record the intersection point that constitutes a contour line as l(x l ,y l ,z l );

[0130] (3) Since a number of contour elevation values ​​are preset before, a list L (i.e., an intersection point storage list) is created for each preset contour elevation value, and the list L is used to store the intersection point l corresponding to the height of the preset contour elevation value;

[0131] (4) For the elevation value z l The intersection point corresponding to the contour lines of l ,y l ,z l ) and the point (0,0,z l ) The angle θ formed with the positive half axis of the Y axis is stored in the corresponding point information. The calculation formula of the angle θ is: In the above formula, if θ is a negative value, the corresponding intersection point l is judged to be located on the left side of the loader; if θ is a 0 value, the corresponding intersection point l is judged to be located in the center of the loader; if θ is a positive value, the corresponding intersection point l is judged to be located on the right side of the loader;

[0132] (5) After traversing each point in the list L of each contour line, select the intersection point l with the smallest angle θ as the first point, and sort all the intersection points l in the list L in the order of θ from small to large, so as to ensure that the generated contour line is directed and continuous; finally, put the sorted points in the list L into the stacked contour array sequence contour_list{};

[0133] (6) When making intersection judgment, the specific intersection judgment strategy is: select an elevation value H i The plane where the contour lines are located is parallel to the plane formed by the XY axis; define the selected triangle as ABC, select any side AB of the triangle, and define the coordinates of the two vertices of AB as (x A ,y A ,z A )、(x B ,y B ,z B ), define the elevation values ​​corresponding to points A and B as z A 、z B ; Based on the above parameters, determine whether the plane intersects with the triangle, that is, when (z A -H i )(z B -H i )>0, the plane is judged to be non-intersecting with the triangle. A -H i )(z B -H i )<0, it is determined that the plane intersects the triangle;

[0134] (7) When determining whether a plane intersects a triangle, the intersection point calculation strategy is as follows: If the plane intersects the triangle, a linear interpolation method is used to calculate the intersection point C (x, y, H) between the plane and the AB side of the triangle. i ), the calculation formula of the intersection point C coordinate is as follows:

[0135] It should be noted that since the lidar cannot measure the obscured stacks, sorting by angle will not cause data confusion.

[0136] In step S304, as an optional method, the logic for calculating the stacking profile line is the same as the logic for calculating the contour line, and the algorithm is as follows:

[0137] S3043, determine the section position: based on the stacking triangulated mesh constructed in S3031, according to the calculation logic of the contour line, traverse the list L corresponding to each contour line, and select the plane where the point l with θ=0° in the list L is located as the section passing through the stacking;

[0138] S3044, judging the intersection between the section and the triangulated mesh: traversing each triangle in the triangulated mesh to judge whether the edge of each triangle intersects with the aforementioned section;

[0139] S3045, cross-section intersection calculation: if they intersect, the position information of the corresponding intersection point W is calculated by linear interpolation method, and then a number of points constituting the stacking cross section are obtained;

[0140] S3046, sorting and arranging intersection points: sorting a number of intersection points W in order from small to large according to their Y-axis coordinate values, and putting the sorted points constituting the stacking profile line into the profile_list{} list, thus completing the calculation of the profile line.

[0141] S400, data transmission steps:

[0142] In this step, the effective information of the obstacles and stacks identified and calculated above is packaged and wirelessly transmitted back to the remote control center. The remote control center displays the stack contours and profiles in real time based on the above information, and performs corresponding safety monitoring and alarm functions, as follows:

[0143] S401, all valid information is packaged and integrated into a data packet using a standard data format;

[0144] In step S401, the standard data format includes but is not limited to JSON;

[0145] In step S401, all valid information includes but is not limited to: stacking contour array sequence contour_list{}, stacking profile array sequence profile_list{}, personnel list ObstacleList_humans{}, obstacle list ObstacleList_obstacles{}, bucket angle information F, surrounding environment image information D collected by the main camera, and surrounding environment original point cloud P collected by the radar, etc.

[0146] S402, using the wireless communication module on the loader to send the packaged data packet to a designated or preset loader remote control center server;

[0147] In step S402, the wireless communication module includes but is not limited to a 4G / 5G module or a Wi-Fi module;

[0148] In step S402, after receiving the data packet, the remote control center server sends a reception confirmation signal to the loader, indicating that the data has been successfully received; when the loader does not receive the confirmation signal within the preset specified time, the loader resends the data packet to the remote control center server until the loader receives the confirmation signal; this process is carried out in real time to ensure the real-time and stability of data return;

[0149] In step S402, the surrounding environment image information D collected by the main camera and the surrounding environment original point cloud P collected by the radar are used for the remote control center to directly display the original signal under special circumstances.

[0150] S403, the remote control center server unpacks the received data packet, performs corresponding conversion processing and visual display for the remote operator to view;

[0151] In step S403, the remote control center server uses a top view and a side view to respectively display the material pile and sensitive obstacles around the loader;

[0152] Specifically, the top view uses a horizontal plane as the projection plane, and the side view uses a plane perpendicular to the ground as the projection plane. Finally, the side view and the top view are displayed in an image coordinate system with the U axis as the horizontal axis and the V axis as the vertical axis. The image coordinate system is as follows: Figure 6 As shown:

[0153] Specifically, the coordinate transformation algorithm of the side view is as follows: Ignore the change of the X axis, project the three-dimensional coordinates to the YZ plane, then map the X axis to the U axis, and the Y and Z axes to the V axis. The P of the projection matrix is side The transformation formula is: Among them, P side Map the X coordinate to 0, the Y coordinate to U, and the Z coordinate to V;

[0154] Specifically, the coordinate transformation algorithm of the top view is as follows: Ignore the Z-axis change, project the three-dimensional coordinates onto the YX plane, then map the Y axis to the U axis, and the X and Z axes to the V axis. The P of the projection matrix is top The transformation formula is: Among them, P top Map the Z coordinate to 0, leaving the X and Y coordinates unchanged.

[0155] In step S403, the remote control center server projects the three-dimensional points stored in the unpacked contour array sequence contour_list{} to the top view image, projects the three-dimensional points stored in the unpacked stacked profile array sequence profile_list{} to the side view image, and highlights the positions of sensitive obstacles in the personnel list ObstacleList_humans{} and the obstacle list ObstacleList_obstacles{} in the display images of the top view and the side view;

[0156] Specifically, for any of the above three-dimensional points, the projection algorithm is: multiply the coordinate p of the three-dimensional point in the original loader consolidation coordinate system by the corresponding transformation matrix and the scaling ratio k in sequence to obtain the new coordinates of the three-dimensional point in the corresponding display image. The transformation matrix is:

[0157] In step S403, after the contour array contour_list{} and the profile array profile_list{} are projected, they are connected in the display image according to the existing order of the three-dimensional points, thereby obtaining the material stacking contour lines and profile lines in the display image; in addition, interpolation points and Bezier curves can be used for corresponding smoothing to improve the display effect;

[0158] In step S403, according to the bucket angle information F and the size parameters of the loader, a graphic representing the loader can be displayed in the image so that the displayed bucket position matches the actual bucket position, thereby helping the remote operator to more accurately determine the working position and simulate the real scene. Figure 7 As shown;

[0159] In summary, based on the above steps S100 to S400, the remote operation center displays an image such as Figure 8 shown.

[0160] It should be noted that all the above examples are only for explaining the present invention and cannot limit the protection scope of the present invention.

[0161] Example 2

[0162] This embodiment is based on the same inventive concept as the environmental perception method for remote shoveling assistance of a loader described in Embodiment 1, and provides an environmental perception system for remote shoveling assistance of a loader, such as Figures 9 to 11 As shown, it includes: a radar 1, a main camera 2, a surround-view camera 3, a perception and computing module, a communication module and a display module.

[0163] Radar 1 and main camera 2 are used to collect raw data of the stack. It should be noted that radar 1 can use frequency-modulated continuous wave radar, and adjust parameters such as the carrier frequency, bandwidth and sweep time of the transmitted signal to maximize the filtering of dust interference. It is conceivable that ordinary radar can be used as a replacement in order to control costs. The effect of ordinary radar will be relatively poor, but it can still provide environmental perception assistance.

[0164] The surround-view camera 3 is used to collect raw data of obstacles such as people and vehicles around the loader. In the present embodiment, there are four surround-view cameras. It should be noted that the number of the main camera 2 and the surround-view camera 3 is not uniquely specified and can be adaptively adjusted according to specific needs or actual conditions.

[0165] In this embodiment, the arrangement of the sensor module on the loader is shown as follows: Fig.10 As shown; wherein, the radar 1 and the main camera 2 are installed at the top of the loader body, and the four surround-view cameras 3 are installed at the front, back, left and right positions under the body respectively; after the installation, the above-mentioned sensor modules are calibrated according to the general industry specifications to accurately feedback data; in addition, it should be noted that in this application, the feedback coordinate accuracy of sensitive obstacles such as people and vehicles is not limited to high-precision requirements. This application focuses more on the perception and processing of the outer contour of the material stack, so the coordinate accuracy requirements of the outer contour of the material stack are limited to high-precision requirements; in addition, in the preliminary configuration stage, the system establishes a coordinate system fixed to the vehicle body, the origin of the coordinate system is located on the center line in front of the vehicle body, the X-axis is toward the right side of the vehicle body, the Y-axis is toward the front of the vehicle head, and the Z-axis is set vertically upward. For specific reference Fig.11 shown.

[0166] The perception computing module is used to process raw data, detect and calculate stack contour information such as stack contour lines and profile lines, and detect information on sensitive obstacles such as people and vehicles around.

[0167] The communication module is used to wirelessly transmit the stacking, obstacle information, bucket position and other information sensed or calculated to the remote control center server; it should be noted that the camera raw data transmission operation is a commonly used remote control method in this field. The raw data transmission related operations of this patent are not unique to this patent, nor are they the scope of protection of this patent, so they will not be repeated here.

[0168] The display module is used to display the received stacking, obstacle information and bucket position in a two-dimensional view on the remote control center server.

[0169] Example 3

[0170] This embodiment provides a computer-readable storage medium, including:

[0171] The storage medium is used to store computer software instructions used to implement the environmental perception method for remote excavation assistance of a loader described in the above-mentioned embodiment 1, which includes a program for executing the above-mentioned program set for the environmental perception method for remote excavation assistance of a loader; specifically, the executable program can be built into the environmental perception system for remote excavation assistance of a loader described in embodiment 2, so that the environmental perception system for remote excavation assistance of a loader can implement the environmental perception method for remote excavation assistance of a loader described in embodiment 1 by executing the built-in executable program.

[0172] In addition, the computer-readable storage medium of this embodiment may adopt any combination of one or more computer-readable storage media, wherein the computer-readable storage medium includes electrical, optical, electromagnetic, infrared or semiconductor systems, devices or components, or any combination thereof.

[0173] It should be understood that in the various embodiments of this document, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this document.

[0174] It should also be understood that in the embodiments of this article, the term "and / or" is only a description of the association relationship of the associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0175] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this article.

[0176] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0177] In the several embodiments provided herein, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, or can be electrical, mechanical or other forms of connection.

[0178] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of this article.

[0179] In addition, each functional unit in each embodiment of this invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above integrated unit may be implemented in the form of hardware or in the form of software functional unit.

[0180] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this article is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of this article. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0181] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. An environmental perception method for remote excavation assistance of a loader, characterized in that: The following steps are involved: Environmental perception steps: Identify sensitive obstacles around the loader based on the image information of the loader's surrounding environment; Stacking perception steps: constructing point cloud data for the pile to be excavated based on the radar feedback data of the loader, preprocessing the point cloud data, and calculating the outline information of the pile to be excavated based on the preprocessed point cloud data; The outline information includes: a stacked contour line array sequence and a profile line array sequence; Data return steps: The information of the sensitive obstacle and the outline information are packaged and transmitted back to a remote control center, and the remote control center displays the outline of the pile to be excavated from multiple angles based on the transmitted information.

2. The environmental perception method for remote excavation assistance of a loader according to claim 1, characterized in that: The step of preprocessing the point cloud data and calculating the outline information of the pile to be excavated based on the preprocessed point cloud data includes: Performing filtering processing on the point cloud data; Eliminating bucket data from the filtered point cloud data to obtain stacking point cloud data; A greedy projection algorithm is used to perform surface reconstruction on the stacked point cloud data to obtain a triangulated mesh; Based on the triangulated grid and the linear interpolation algorithm, determine the intersection of the contour plane of each sub-stack in the to-be-excavated stack and each edge in the triangulated grid, and connect a number of the intersections in order to obtain a contour array sequence of the to-be-excavated stack; Based on the acquisition logic of the contour line array sequence, the section line array sequence is acquired.

3. The environmental perception method for remote excavation assistance of a loader according to claim 2, characterized in that: The method of using a greedy projection algorithm to perform surface reconstruction on the stacked point cloud data to obtain a triangulated mesh includes: For each group of sub-stacked point cloud data in the stacked point cloud data, initializing an empty grid; For each group of sub-stacked point cloud data in the stacked point cloud data, sequentially performing optimal point selection, valid triangle determination, and valid triangle construction cycles based on the empty grid; The optimal point selection includes: using a greedy projection algorithm to determine three optimal points in the point cloud data of the sub-stack, and judging whether the three optimal points satisfy the formation of a valid triangle; The valid triangle determination includes: in response to the three optimal points being able to form a valid triangle, adding the formed valid triangle as a newly created triangle to the empty mesh; The effective triangle construction loop includes: in response to the newly created triangle being added to the empty mesh, removing the three best points in the stacked point cloud data that have been used to construct the effective triangle, selecting new best points from the stacked point cloud data to construct a new effective triangle, and looping; in response to the three best points failing to satisfy the formation of an effective triangle, jumping out of the loop; based on the loop, expanding into the stacked point cloud data set until the stacked point cloud data is empty or fails to form a valid triangle, ending the loop to obtain the connected triangulated mesh; The triangulated mesh is used to simulate the outer surface of the pile to be excavated.

4. The environmental perception method for remote excavation assistance of a loader according to claim 2, characterized in that: Based on the triangulated grid and the linear interpolation algorithm, the intersection of the contour plane of each sub-stack in the to-be-excavated stack and each edge in the triangulated grid is determined, and a plurality of the intersections are sequentially connected to obtain a contour array sequence of the to-be-excavated stack, including: Setting a number of reference elevation values, and calculating the elevation value of the contour line of each sub-stack in the stack to be excavated based on the reference elevation values; For each sub-stack, the plane where the contour line is located and each triangle in the triangulated mesh are respectively judged to intersect based on the corresponding elevation value, and the intersection point is calculated when they intersect; Calculate the angle information corresponding to the intersection vector based on the intersection point; sorting the plurality of intersection points based on the angle information; The contour line array sequence is constructed based on the sorted intersection points.

5. The environmental perception method for remote excavation assistance of a loader according to claim 4, characterized in that: The method further comprises: determining the intersection of the contour plane of each sub-stack in the to-be-excavated stack and each edge in the triangulated grid based on the triangulated grid and sequentially connecting a plurality of the intersections to obtain a contour array sequence of the to-be-excavated stack, and: Traversing each triangle in the triangulated mesh in turn, checking the elevation values ​​of three vertices of each triangle, and determining the elevation value at a specific contour level; Determine the intersection points of the plane where the contour line is located and the sides of the triangles across different contour line levels by interpolation calculation, calculate the intersection points, and record a number of the intersection points constituting a contour line; Create an intersection storage list, use the intersection storage list to store a number of the intersections; determine the intersection vector of the intersection, calculate the angle between the intersection vector and the positive semi-axis of the Y axis of the vehicle coordinate system, store the angle in the information of the corresponding intersection, and judge whether the intersection is located on the right side, left side or center of the loader according to the positive value, negative value or zero value of the angle; According to the angle information respectively corresponding to the intersections in the intersection storage list, the intersections are sorted in a directed ascending order with respect to the angle information, and the sorted intersections are used as the contour line array sequence.

6. The environmental perception method for remote excavation assistance of a loader according to claim 5, characterized in that: The intersection judgment strategy includes: Define the contour line of the sub-stack as H i ; The coordinates of the two vertices of the selected side of the selected triangle are (x A ,y A ,z A ) and (x B ,y B ,z B ); The elevation values ​​corresponding to the two vertices of the selected side of the selected triangle are z A and z B ; When (z A -H i )(z B -H i )>0, determining that the plane where the contour line of the sub-stack is located does not intersect with the selected triangle; When (z A -H i )(z B -H i )<0, determining that the plane where the contour line of the sub-stack is located intersects with the selected triangle; The intersection calculation strategy includes: In response to the plane where the contour lines of the sub-stack are located intersecting with the selected triangle, a linear interpolation algorithm is used to calculate the intersection of the plane where the contour lines of the sub-stack are located and the selected side of the selected triangle, and the intersection calculation formula is: Wherein, x is the horizontal coordinate of the intersection point, y is the vertical coordinate of the intersection point, H i as the ordinate of the intersection point.

7. The environment perception method for remote excavation assistance of a loader according to claim 5, characterized in that: The acquiring logic based on the contour line array sequence, acquiring the profile line array sequence, comprises: Determining a section that passes through the pile to be excavated based on the triangulated mesh and the intersection storage list; Traversing each triangle in the triangulated mesh, and determining whether an edge of each triangle intersects with the cross section; In response to the intersection, a linear interpolation algorithm is used to calculate the section intersection points and the position information of the section intersection points to obtain a plurality of section intersection points; According to the values ​​of the ordinates of the section intersections, the section intersections are ordered from small to large, and the ordered section intersections are used as the section line array sequence.

8. The environment perception method for remote excavation assistance of a loader according to claim 1, characterized in that: The packaging and transmission of the sensitive obstacle information and the outer contour information to the remote control center, and the remote control center performing a multi-angle display of the outer contour of the pile to be excavated based on the transmitted information, includes: Packaging the sensitive obstacle information and the outline information in a standard data format; Sending the packaged data packets to the remote control center server of the loader; The received data packet is unpacked by the remote control center server, and the unpacked data is converted and processed and visualized.

9. The environment perception method for remote excavation assistance of a loader according to claim 8, characterized in that: After the packaged data packet is sent to the remote control center server of the loader, the process further includes: In response to the remote control center server receiving the data packet, sending a reception confirmation signal to the loader; In response to the loader not receiving the confirmation signal within the time period, the data packet is resent to the remote control center server until the loader receives the confirmation signal.

10. The environment perception method for remote excavation assistance of a loader according to claim 8, characterized in that: The conversion and visualization of the unpacked data includes: Create top-view and side-view images of the loader's surroundings; Projecting the three-dimensional points stored in the unpacked contour array sequence to the top view image; projecting the three-dimensional points stored in the unpacked profile array sequence to the side view image; in response to the projection of the three-dimensional points of the contour array and the three-dimensional points of the profile array, connecting lines in the top view image and the side view image in the existing order of a number of the three-dimensional points to obtain stacked contour lines and stacked profile lines in the top view image and the side view image; A simulated image of a loader bucket is constructed in the top view image and the side view image.

Images