A method and device for generating a travel area of a construction machine, a storage medium, and a terminal
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING BUILDER INTELLIGENT TECHNOLOGY CO LTD
- Filing Date
- 2022-07-04
- Publication Date
- 2026-06-30
Smart Images
Figure CN115376096B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of engineering machinery technology, and in particular to a method, device, storage medium and terminal for generating the driving area of engineering machinery. Background Technology
[0002] With the vigorous development of infrastructure, more and more construction machinery (such as single-bucket excavators, loaders, drilling rigs, etc.) need to operate in scenarios without obvious road markings, and need to continuously travel to complete work tasks in different locations during the operation.
[0003] Currently, when controlling construction machinery, drivers typically look around to find a suitable road surface before taking control of the machinery. However, due to the lack of clear road markings and the potential presence of numerous obstacles at work sites, collisions between the machinery and these obstacles can occur if the driver fails to observe them promptly, thus reducing the safety of the machinery. Summary of the Invention
[0004] This application provides a method, apparatus, storage medium, and terminal for generating the driving area of engineering machinery. To provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. This summary is not intended as a general description, nor is it intended to identify key / important components or describe the scope of protection of these embodiments. Its sole purpose is to present some concepts in a simple form as a prelude to the detailed description that follows.
[0005] In a first aspect, embodiments of this application provide a method for generating the operating area of engineering machinery, the method comprising:
[0006] It receives in real time initial images, initial scene point clouds, and relative angles of mechanical components from sensors pre-installed on the engineering machinery;
[0007] Preprocess the initial image and initial scene point cloud based on the relative angles of the mechanical parts to generate the target image and target scene point cloud;
[0008] Based on the target image and the target scene point cloud, the horizontal ground region and the object region are divided;
[0009] The driving area is drawn as a 2D mesh based on the horizontal ground area and the object area.
[0010] Optionally, the initial image and initial scene point cloud are preprocessed based on the relative angles of the mechanical components to generate the target image and target scene point cloud, including:
[0011] The spatial position of each mechanical component is determined based on the relative angles of the mechanical parts.
[0012] Based on the spatial location of each mechanical component, and combined with image semantic segmentation technology, the region image of each mechanical component is segmented and removed from the initial image to obtain the target image;
[0013] Based on the spatial location of each mechanical component, and combined with the pre-built mechanical 3D model, the point cloud of each mechanical component is segmented and removed from the initial scene point cloud to obtain the target scene point cloud.
[0014] Optionally, the spatial position of each mechanical component is determined based on the relative angles between the mechanical components, including:
[0015] Based on the relative angles of the mechanical components and combined with the pre-built 3D mechanical model, the spatial position of the mechanical components is inferred, and the spatial position of each mechanical component is output.
[0016] Optionally, the driving area is drawn as a 2D mesh based on the horizontal ground area and object area, including:
[0017] Record all point clouds and pixels belonging to the object region to obtain the object outline;
[0018] The object outline is enlarged based on a pre-built mechanical 3D model to obtain the enlarged object outline.
[0019] By removing the magnified object outline from the horizontal ground area, the area where the construction machinery can travel is obtained.
[0020] A 2D mesh is drawn for the drivable area of the construction machinery to obtain the drivable area in 2D mesh state.
[0021] Optionally, after drawing the driving area as a 2D mesh based on the horizontal ground area and object area, it also includes:
[0022] Draw the top view of the pre-built mechanical 3D model;
[0023] The driving area in the 2D mesh state is merged into the corresponding position around the top view of the model, and the driving area in the 2D mesh state is colored according to the preset color rendering parameters to obtain the target top view.
[0024] Display a top-down view of the target.
[0025] Optionally, the horizontal ground region and object region are divided based on the target image and the target scene point cloud, including:
[0026] Extract the feasible region of the target image to obtain a flat region in the image space;
[0027] Extract the feasible region of the point cloud of the target scene to obtain the flat area under the spatial point cloud;
[0028] By using the mapping relationship between point clouds and images, the flat regions in the image space and the flat regions in the spatial point cloud are fused to obtain the overlapping and non-overlapping parts;
[0029] The overlapping areas are defined as horizontal ground regions, and the non-overlapping areas are defined as object regions.
[0030] Optionally, the horizontal ground region and object region are divided based on the target image and the target scene point cloud, including:
[0031] The target image is input into a preset image feature extraction network to obtain image features;
[0032] The point cloud of the target scene is input into a preset point cloud feature extraction network to obtain point cloud features;
[0033] Image features and point cloud features are fused to obtain hybrid features;
[0034] The mixed features are input into the semantic segmentation network, which outputs horizontal ground regions and object regions.
[0035] Secondly, embodiments of this application provide a device for generating the driving area of engineering machinery, the device comprising:
[0036] The sensor parameter receiving module is used to receive in real time the initial images, initial scene point clouds, and relative angles of mechanical components sensed by sensors pre-installed on the engineering machinery.
[0037] The parameter preprocessing module is used to preprocess the initial image and initial scene point cloud according to the relative angle of the mechanical parts, and generate the target image and target scene point cloud.
[0038] The region segmentation module is used to divide the horizontal ground region and object region based on the target image and the target scene point cloud;
[0039] The driving area drawing module is used to draw a 2D mesh driving area based on the horizontal ground area and the object area.
[0040] Thirdly, embodiments of this application provide a computer storage medium storing multiple instructions adapted for loading and execution of the above-described method steps by a processor.
[0041] Fourthly, embodiments of this application provide a terminal that may include: a processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and executed by the above-described method steps.
[0042] The technical solutions provided in this application embodiment may include the following beneficial effects:
[0043] In this embodiment, the driving area generation device for construction machinery first receives in real-time initial images, initial scene point clouds, and relative angles of mechanical components from sensors pre-installed on the construction machinery. Then, it preprocesses the initial images and initial scene point clouds based on the relative angles of the mechanical components to generate target images and target scene point clouds. Next, it divides the horizontal ground area and object area based on the target images and target scene point clouds. Finally, it draws a 2D mesh driving area based on the horizontal ground area and object area. Because this application uses sensors to perceive environmental data around the construction machinery and divides the horizontal ground area and object area based on the environmental data, and then draws the driving area based on the horizontal ground area and object area, it can provide a reference for the driving path of the unmanned construction machinery, avoiding collisions with obstacles in the object area and improving the safety factor of the construction machinery.
[0044] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Attached Figure Description
[0045] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.
[0046] Figure 1 This is a flowchart illustrating a method for generating the driving area of engineering machinery according to an embodiment of this application;
[0047] Figure 2 This is a schematic diagram of the installation of an environmental perception sensor for the travel direction of an excavator, provided in an embodiment of this application.
[0048] Figure 3 This is a schematic diagram of an excavator robotic arm status measurement sensor provided in an embodiment of this application;
[0049] Figure 4 This is a schematic diagram of the installation of an environmental perception sensor for the driving direction of a loader, provided in an embodiment of this application.
[0050] Figure 5 This is a schematic diagram of a loader front frame and bucket status measurement sensor provided in an embodiment of this application;
[0051] Figure 6 This is a schematic block diagram of the process of generating the driving area of an engineering machinery provided in this application;
[0052] Figure 7 This is a flowchart illustrating another method for generating the driving area of engineering machinery provided in an embodiment of this application;
[0053] Figure 8This is a schematic diagram of the structure of a device for generating the driving area of engineering machinery provided in an embodiment of this application;
[0054] Figure 9 This is a schematic diagram of the structure of a terminal provided in an embodiment of this application. Detailed Implementation
[0055] The following description and accompanying drawings fully illustrate specific embodiments of the invention to enable those skilled in the art to practice them.
[0056] It should be understood that the described embodiments are merely some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.
[0057] In the following description, when referring to the accompanying drawings, the same numbers in different drawings denote the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention as detailed in the appended claims.
[0058] In the description of this invention, it should be understood that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Those skilled in the art can understand the specific meaning of these terms in this invention based on the specific circumstances. Furthermore, in the description of this invention, unless otherwise stated, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.
[0059] This application provides a method, apparatus, storage medium, and terminal for generating the driving area of construction machinery to solve the problems existing in the aforementioned related technologies. In the technical solution provided by this application, environmental data surrounding the construction machinery is perceived by sensors, and horizontal ground areas and object areas are divided based on the environmental data. The driving area is then drawn based on the horizontal ground areas and object areas, thereby providing a reference for the driving path of unmanned construction machinery, avoiding collisions with obstacles in the object areas, and improving the safety factor of the construction machinery. The following is a detailed description using exemplary embodiments.
[0060] The following will be combined with the appendix Figure 1 -Appendix Figure 7This application provides a detailed description of the method for generating the driving area of construction machinery according to embodiments. This method can be implemented using a computer program and can run on a driving area generation device for construction machinery based on the von Neumann architecture. This computer program can be integrated into an application or run as a standalone utility application.
[0061] Please see Figure 1 This is a flowchart illustrating a method for generating the driving area of engineering machinery, as provided in an embodiment of this application. Figure 1 As shown, the method in this application embodiment may include the following steps:
[0062] S101 receives in real time initial images, initial scene point clouds, and relative angles of mechanical components from sensors pre-installed on the engineering machinery;
[0063] Construction machinery can include excavators, loaders, drilling rigs, etc. Sensors are electronic components used to acquire images, scene point clouds, and relative angles of mechanical parts. Sensors for acquiring images can be multiple color cameras or thermal imaging cameras. Sensors for acquiring scene point clouds can be multiple binocular cameras / LiDAR / 4D millimeter-wave radar. Sensors for acquiring relative angles of mechanical parts can be tilt sensors.
[0064] Typically, there is some overlap between images and point clouds acquired by sensors, enabling full coverage of the 90-degree range to the left and right of the construction machinery's forward direction. Since construction machinery often has operational devices mounted along its forward direction, the installation location of the tilt sensor needs to be selected based on the characteristics of each type of machinery to observe the surrounding environment and measure the status of its own mechanical devices.
[0065] In this application embodiment, the installation of sensors on construction machinery is specifically implemented in the following ways. When installing sensors on an excavator, two sets of sensors can be installed on each side of the excavator, facing forward and to the left / right sides respectively. The installed sensors include several types of sensors such as color cameras, thermal imaging cameras, binocular cameras, LiDAR, and 4D millimeter-wave radar, which can acquire scene images and point clouds, for example... Figure 2 As shown. Tilt sensors are installed on the bucket, boom, and arm to obtain the relative angles between the bucket and the boom, the boom and arm, and the arm and chassis. For example... Figure 3 As shown. When installing sensors on the loader, a bracket is added to the front half-shaft of the loader, and three sets of sensors are installed on the bracket, facing forward and to the left / right sides respectively. Another set of sensors is installed behind the bucket to observe the front when the bucket is raised. The installed sensors include several types of cameras such as color cameras, thermal imaging cameras, binocular cameras, LiDAR, and 4D millimeter-wave radar, which can acquire scene images and point clouds, for example... Figure 4As shown. A pivot angle sensor is installed at the hinge point of the front and rear frames to measure the included angle between the front and rear frames. A pivot angle sensor is installed at the connection point between the boom and the frame to measure the relative angle between the boom and the frame. A pivot angle sensor is installed at the connection point between the bucket and the boom to measure the included angle between the bucket and the boom, for example... Figure 5 As shown.
[0066] After the sensors are installed on the construction machinery, the driving area of the construction machinery can be specifically planned based on the initial images, initial scene point clouds, and relative angles of the mechanical components sensed by the sensors.
[0067] In one possible implementation, after the construction machinery is started, the pre-installed sensors begin to perceive data of the surrounding environment in real time, obtaining initial images, initial scene point clouds, and relative angles of mechanical parts. Then, the sensors send the perceived data to the vehicle terminal or server. The vehicle terminal or server receives the initial images, initial scene point clouds, and relative angles of mechanical parts perceived by the sensors pre-installed on the construction machinery in real time.
[0068] S102, preprocess the initial image and initial scene point cloud according to the relative angle of the mechanical parts to generate the target image and target scene point cloud;
[0069] Among them, mechanical components, taking an excavator as an example, can include the bucket, boom, and arm.
[0070] In this embodiment of the application, when preprocessing the initial image and the initial scene point cloud according to the relative angle of the mechanical components, the spatial position of each mechanical component is first determined according to the relative angle of the mechanical components. Then, based on the spatial position of each mechanical component and combined with image semantic segmentation technology, the region image of each mechanical component is segmented and removed from the initial image to obtain the target image. Finally, based on the spatial position of each mechanical component and combined with the pre-constructed mechanical 3D model, the point cloud of each mechanical component is segmented and removed from the initial scene point cloud to obtain the target scene point cloud.
[0071] Specifically, when determining the spatial position of each mechanical component based on the relative angles of the mechanical parts, the spatial position of the mechanical components is inferred based on the relative angles of the mechanical parts and in conjunction with the pre-built 3D mechanical model, and the spatial position of each mechanical component is output.
[0072] Furthermore, when generating the pre-built mechanical 3D model, the dimensional parameters of each component of the engineering machinery are first determined, and then the mechanical 3D model of the engineering machinery is constructed based on the dimensional parameters of each component, thus obtaining the pre-built mechanical 3D model.
[0073] S103, divide the horizontal ground region and object region according to the target image and the target scene point cloud;
[0074] In one possible implementation, the feasible region of the target image is first extracted to obtain the flat region in the image space. Then, the feasible region of the point cloud of the target scene is extracted to obtain the flat region in the spatial point cloud. Next, the flat region in the image space and the flat region in the spatial point cloud are fused through the mapping relationship between the point cloud and the image to obtain the overlapping part and the non-overlapping part. Finally, the overlapping part is determined as the horizontal ground region, and the non-overlapping part is determined as the object region.
[0075] Specifically, when extracting the feasible region of a target image, traditional image processing methods such as color extraction and image gradient extraction can be used. Alternatively, deep learning methods can be employed to segment the image, train a model capable of finding flat terrain by labeling flat regions in the image space, and extract the feasible region. Deep learning methods include Mask-R-CNN and DeepLabv3+.
[0076] Specifically, when extracting the feasible region of the point cloud of a target scene, traditional methods can be used to extract horizontal regions. For example, the RANSAC algorithm can be used to extract the horizontal plane, and asymptotic morphological filtering methods can be used to extract the lowest ground level. Alternatively, deep learning methods can be employed, training a model capable of finding flat ground by annotating flat regions in the point cloud space. Deep learning methods include RANDLA-Net, etc.
[0077] In another possible implementation, the target image is first input into a preset image feature extraction network to obtain image features, and then the target scene point cloud is input into a preset point cloud feature extraction network to obtain point cloud features. Next, the image features and point cloud features are fused to obtain mixed features. Finally, the mixed features are input into a semantic segmentation network to output the horizontal ground region and object region.
[0078] Furthermore, flat regions can be extracted based on continuous data. Specifically, mapping methods such as rtabmap-slam and orb-slam can be used to build a scene map, and then horizontal regions can be extracted using the horizontal region extraction methods mentioned in the image and point cloud backend fusion.
[0079] S104, draws a 2D mesh of the driving area based on the horizontal ground area and the object area.
[0080] In this embodiment of the application, when drawing a 2D mesh state driving area based on the horizontal ground area and the object area, firstly, all point clouds and pixels belonging to the object area are recorded to obtain the object outline. Then, the object outline is enlarged according to the pre-built mechanical 3D model to obtain the enlarged object outline. Next, the enlarged object outline is removed in the horizontal ground area to obtain the driving area of the construction machinery. Finally, a 2D mesh is drawn for the driving area of the construction machinery to obtain the driving area in the 2D mesh state.
[0081] Furthermore, after obtaining the driving area in 2D mesh state, firstly, draw the top view of the model corresponding to the pre-built mechanical 3D model, then merge the driving area in 2D mesh state into the corresponding position around the top view of the model, and perform color rendering on the driving area in 2D mesh state according to the pre-set color rendering parameters to obtain the target top view, and finally display the target top view.
[0082] For example Figure 6 As shown, Figure 6 This application provides a schematic flowchart of the process for generating the driving area of construction machinery. First, corresponding sensors are installed on the construction machinery. Using the sensor data measuring the machinery itself, and combining it with a pre-constructed 3D model of the machinery, the spatial positions of each component are inferred, such as the position of the excavator bucket relative to the excavator base. Then, based on the spatial positions of each component, image data and point cloud data sensed by the sensors are preprocessed. For example, for image data, the projection of the 3D model of the machinery onto the image can be used to remove parts belonging to the machinery itself. Image semantic segmentation technology can also be used to segment the image belonging to the machinery. For point cloud data, the corresponding point clouds are removed using the 3D model of the machinery. Next, based on the preprocessed data, flat ground and objects outside the flat ground are divided. For each object outside the flat ground, all point clouds and pixels belonging to that object are recorded to obtain 2D / 3D contours. Based on the construction machinery's own model, the 2D / 3D contours of objects outside the flat ground are expanded, and the expanded portion is removed from the 2D / 3D representation of the flat ground. The remaining flat ground is the area where the construction machinery can drive. Finally, draw a top view of the construction machinery, with a 3D model of the machinery in the center. Around the model, draw a 2D mesh, using different shades of color to indicate whether a certain area is drivable.
[0083] In this embodiment, the driving area generation device for construction machinery first receives in real-time initial images, initial scene point clouds, and relative angles of mechanical components from sensors pre-installed on the construction machinery. Then, it preprocesses the initial images and initial scene point clouds based on the relative angles of the mechanical components to generate target images and target scene point clouds. Next, it divides the horizontal ground area and object area based on the target images and target scene point clouds. Finally, it draws a 2D mesh driving area based on the horizontal ground area and object area. Because this application uses sensors to perceive environmental data around the construction machinery and divides the horizontal ground area and object area based on the environmental data, and then draws the driving area based on the horizontal ground area and object area, it can provide a reference for the driving path of the unmanned construction machinery, avoiding collisions with obstacles in the object area and improving the safety factor of the construction machinery.
[0084] Please see Figure 7 This is a flowchart illustrating a method for generating the driving area of engineering machinery, as provided in an embodiment of this application. Figure 7 As shown, the method in this application embodiment may include the following steps:
[0085] S201 receives in real time initial images, initial scene point clouds, and relative angles of mechanical components from sensors pre-installed on the engineering machinery;
[0086] S202, determine the spatial position of each mechanical component based on the relative angle of the mechanical components;
[0087] S203, based on the spatial location of each mechanical component and combined with image semantic segmentation technology, segment and remove the region image of each mechanical component from the initial image to obtain the target image;
[0088] S204. Based on the spatial position of each mechanical component and combined with the pre-built mechanical 3D model, the point cloud of each mechanical component is segmented and removed from the initial scene point cloud to obtain the target scene point cloud.
[0089] S205, divide the horizontal ground region and object region according to the target image and the target scene point cloud;
[0090] S206, record all point clouds and pixels belonging to the object region to obtain the object outline;
[0091] S207, The outline of the object is enlarged based on the pre-built mechanical 3D model to obtain the enlarged outline of the object;
[0092] S208, remove the enlarged object outline from the horizontal ground area to obtain the area where the construction machinery can travel;
[0093] S209, Draw a 2D mesh for the drivable area of the construction machinery to obtain the drivable area in 2D mesh state.
[0094] In this embodiment, the driving area generation device for construction machinery first receives in real-time initial images, initial scene point clouds, and relative angles of mechanical components from sensors pre-installed on the construction machinery. Then, it preprocesses the initial images and initial scene point clouds based on the relative angles of the mechanical components to generate target images and target scene point clouds. Next, it divides the horizontal ground area and object area based on the target images and target scene point clouds. Finally, it draws a 2D mesh driving area based on the horizontal ground area and object area. Because this application uses sensors to perceive environmental data around the construction machinery and divides the horizontal ground area and object area based on the environmental data, and then draws the driving area based on the horizontal ground area and object area, it can provide a reference for the driving path of the unmanned construction machinery, avoiding collisions with obstacles in the object area and improving the safety factor of the construction machinery.
[0095] The following are embodiments of the apparatus of the present invention, which can be used to execute embodiments of the method of the present invention. For details not disclosed in the embodiments of the apparatus of the present invention, please refer to the embodiments of the method of the present invention.
[0096] Please see Figure 8 This illustration shows a schematic diagram of a driving area generation device for construction machinery provided in an exemplary embodiment of the present invention. This driving area generation device for construction machinery can be implemented as all or part of a terminal through software, hardware, or a combination of both. The device 1 includes a sensor parameter receiving module 10, a parameter preprocessing module 20, a region division module 30, and a driving area drawing module 40.
[0097] The sensor parameter receiving module 10 is used to receive in real time the initial image, initial scene point cloud and relative angle of mechanical parts sensed by the sensors pre-installed on the engineering machinery;
[0098] The parameter preprocessing module 20 is used to preprocess the initial image and initial scene point cloud according to the relative angle of the mechanical parts, and generate the target image and target scene point cloud.
[0099] The region segmentation module 30 is used to segment the horizontal ground region and the object region based on the target image and the target scene point cloud.
[0100] The driving area drawing module 40 is used to draw a 2D mesh state driving area based on the horizontal ground area and the object area.
[0101] It should be noted that the above embodiments of the construction machinery driving area generation device are only illustrative examples of the division of the above functional modules when executing the construction machinery driving area generation method. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the equipment can be divided into different functional modules to complete all or part of the functions described above. In addition, the construction machinery driving area generation device and the construction machinery driving area generation method embodiments provided in the above embodiments belong to the same concept, and the implementation process is detailed in the method embodiments, which will not be repeated here.
[0102] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.
[0103] In this embodiment, the driving area generation device for construction machinery first receives in real-time initial images, initial scene point clouds, and relative angles of mechanical components from sensors pre-installed on the construction machinery. Then, it preprocesses the initial images and initial scene point clouds based on the relative angles of the mechanical components to generate target images and target scene point clouds. Next, it divides the horizontal ground area and object area based on the target images and target scene point clouds. Finally, it draws a 2D mesh driving area based on the horizontal ground area and object area. Because this application uses sensors to perceive environmental data around the construction machinery and divides the horizontal ground area and object area based on the environmental data, and then draws the driving area based on the horizontal ground area and object area, it can provide a reference for the driving path of the unmanned construction machinery, avoiding collisions with obstacles in the object area and improving the safety factor of the construction machinery.
[0104] The present invention also provides a computer-readable medium having program instructions stored thereon, which, when executed by a processor, implement the method for generating the driving area of engineering machinery provided in the above-described method embodiments.
[0105] The present invention also provides a computer program product containing instructions that, when run on a computer, causes the computer to execute the method for generating the driving area of engineering machinery described in the above-described method embodiments.
[0106] Please see Figure 9 This is a schematic diagram of the structure of a terminal provided in an embodiment of this application. Figure 9 As shown, terminal 1000 may include: at least one processor 1001, at least one network interface 1004, user interface 1003, memory 1005, and at least one communication bus 1002.
[0107] The communication bus 1002 is used to realize the connection and communication between these components.
[0108] The user interface 1003 may include a display screen and a camera. Optionally, the user interface 1003 may also include a standard wired interface and a wireless interface.
[0109] The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface).
[0110] The processor 1001 may include one or more processing cores. The processor 1001 connects to various parts within the electronic device 1000 using various interfaces and lines. It executes various functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory 1005, and by calling data stored in the memory 1005. Optionally, the processor 1001 may be implemented using at least one hardware form selected from Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). The processor 1001 may integrate one or more of the following: a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), and a modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content to be displayed on the screen; and the modem handles wireless communication. It is understood that the modem may also be implemented as a separate chip, without being integrated into the processor 1001.
[0111] The memory 1005 may include random access memory (RAM) or read-only memory. Optionally, the memory 1005 may include a non-transitory computer-readable storage medium. The memory 1005 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 1005 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the above-described method embodiments, etc.; the data storage area may store data involved in the above-described method embodiments, etc. Optionally, the memory 1005 may also be at least one storage device located remotely from the aforementioned processor 1001. Figure 9 As shown, the memory 1005, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and an application for generating the driving area of engineering machinery.
[0112] exist Figure 9 In the terminal 1000 shown, the user interface 1003 is mainly used to provide an input interface for the user and to obtain the user's input data; while the processor 1001 can be used to call the application program for generating the driving area of the construction machinery stored in the memory 1005, and specifically perform the following operations:
[0113] It receives in real time initial images, initial scene point clouds, and relative angles of mechanical components from sensors pre-installed on the engineering machinery;
[0114] Preprocess the initial image and initial scene point cloud based on the relative angles of the mechanical parts to generate the target image and target scene point cloud;
[0115] Based on the target image and the target scene point cloud, the horizontal ground region and the object region are divided;
[0116] The driving area is drawn as a 2D mesh based on the horizontal ground area and the object area.
[0117] In one embodiment, when the processor 1001 performs preprocessing on the initial image and initial scene point cloud based on the relative angle of the mechanical components to generate the target image and target scene point cloud, it specifically performs the following operations:
[0118] The spatial position of each mechanical component is determined based on the relative angles of the mechanical parts.
[0119] Based on the spatial location of each mechanical component, and combined with image semantic segmentation technology, the region image of each mechanical component is segmented and removed from the initial image to obtain the target image;
[0120] Based on the spatial location of each mechanical component, and combined with the pre-built mechanical 3D model, the point cloud of each mechanical component is segmented and removed from the initial scene point cloud to obtain the target scene point cloud.
[0121] In one embodiment, when the processor 1001 determines the spatial position of each mechanical component based on the relative angle of the mechanical components, it specifically performs the following operations:
[0122] Based on the relative angles of the mechanical components and combined with the pre-built 3D mechanical model, the spatial position of the mechanical components is inferred, and the spatial position of each mechanical component is output.
[0123] In one embodiment, when the processor 1001 executes the operation of drawing a 2D mesh state of the driving area based on the horizontal ground area and the object area, it specifically performs the following operations:
[0124] Record all point clouds and pixels belonging to the object region to obtain the object outline;
[0125] The object outline is enlarged based on a pre-built mechanical 3D model to obtain the enlarged object outline.
[0126] By removing the magnified object outline from the horizontal ground area, the area where the construction machinery can travel is obtained.
[0127] A 2D mesh is drawn for the drivable area of the construction machinery to obtain the drivable area in 2D mesh state.
[0128] In one embodiment, after the processor 1001 performs the following operations when drawing a 2D mesh state of the driving area based on the horizontal ground area and object area:
[0129] Draw the top view of the pre-built mechanical 3D model;
[0130] The driving area in the 2D mesh state is merged into the corresponding position around the top view of the model, and the driving area in the 2D mesh state is colored according to the preset color rendering parameters to obtain the target top view.
[0131] Display a top-down view of the target.
[0132] In one embodiment, when the processor 1001 performs the operation of dividing the horizontal ground region and object region based on the target image and the target scene point cloud, it specifically performs the following operations:
[0133] Extract the feasible region of the target image to obtain a flat region in the image space;
[0134] Extract the feasible region of the target scene point cloud to obtain the flat area under the spatial point cloud;
[0135] By using the mapping relationship between point cloud and image, the flat area in the image space is fused with the flat area in the spatial point cloud to obtain the overlapping part and the non-overlapping part;
[0136] The overlapping portion is defined as the horizontal ground region, and the non-overlapping portion is defined as the object region.
[0137] In one embodiment, when the processor 1001 performs the operation of dividing the horizontal ground region and object region based on the target image and the target scene point cloud, it specifically performs the following operations:
[0138] The target image is input into a preset image feature extraction network to obtain image features;
[0139] The target scene point cloud is input into a preset point cloud feature extraction network to obtain point cloud features;
[0140] The image features and the point cloud features are fused to obtain hybrid features;
[0141] The hybrid features are input into the semantic segmentation network, which outputs horizontal ground regions and object regions.
[0142] In this embodiment, the driving area generation device for construction machinery first receives in real-time initial images, initial scene point clouds, and relative angles of mechanical components from sensors pre-installed on the construction machinery. Then, it preprocesses the initial images and initial scene point clouds based on the relative angles of the mechanical components to generate target images and target scene point clouds. Next, it divides the horizontal ground area and object area based on the target images and target scene point clouds. Finally, it draws a 2D mesh driving area based on the horizontal ground area and object area. Because this application uses sensors to perceive environmental data around the construction machinery and divides the horizontal ground area and object area based on the environmental data, and then draws the driving area based on the horizontal ground area and object area, it can provide a reference for the driving path of the unmanned construction machinery, avoiding collisions with obstacles in the object area and improving the safety factor of the construction machinery.
[0143] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program generated in the driving area of the construction machinery can be stored in a computer-readable storage medium. When executed, the program can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory, or random access memory, etc.
[0144] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall still fall within the scope of this application.
Claims
1. A method for generating the operating area of engineering machinery, characterized in that, The method includes: It receives initial images, initial scene point clouds, and relative angles of mechanical parts from sensors pre-installed on the engineering machinery in real time; the sensors for acquiring images are multiple sets of color cameras and thermal imaging cameras; the sensors for acquiring scene point clouds are multiple sets of binocular cameras, LiDAR, or 4D millimeter-wave radar; and the sensors for acquiring relative angles of mechanical parts are tilt sensors. Preprocessing the initial image and initial scene point cloud based on the relative angles of the mechanical components to generate target images and target scene point clouds includes: inferring the spatial position of the mechanical components based on the relative angles of the mechanical components and combining them with a pre-built 3D mechanical model, and outputting the spatial position of each mechanical component; segmenting and removing the region image of each mechanical component from the initial image based on the spatial position of each mechanical component and combining it with image semantic segmentation technology to obtain the target image; segmenting and removing the point cloud of each mechanical component from the initial scene point cloud based on the spatial position of each mechanical component and combining it with the pre-built 3D mechanical model to obtain the target scene point cloud; generating a pre-built 3D mechanical model includes: determining the size parameters of each component of the engineering machinery, and constructing a 3D mechanical model of the engineering machinery based on the size parameters of each component to obtain the pre-built 3D mechanical model. Based on the target image and the target scene point cloud, the horizontal ground region and object region are divided, including: Extract the feasible region of the target image to obtain a flat region in the image space; extract the feasible region of the target scene point cloud to obtain a flat region in the spatial point cloud; through the mapping relationship between the point cloud and the image, fuse the flat region in the image space and the flat region in the spatial point cloud to obtain overlapping and non-overlapping parts; determine the overlapping part as the horizontal ground region and the non-overlapping part as the object region; or, input the target image into a preset image feature extraction network to obtain image features; input the target scene point cloud into a preset point cloud feature extraction network to obtain point cloud features; fuse the image features and the point cloud features to obtain hybrid features; input the hybrid features into a semantic segmentation network to output the horizontal ground region and the object region; The driving area is drawn as a 2D mesh based on the horizontal ground area and the object area, providing a reference for the driving path of unmanned construction machinery.
2. The method according to claim 1, characterized in that, The driving area, which is drawn as a 2D mesh based on the horizontal ground area and the object area, includes: Record all point clouds and pixels belonging to the object region to obtain the object outline; The object outline is enlarged based on a pre-constructed mechanical 3D model to obtain the enlarged object outline. The enlarged object outline is removed from the horizontal ground area to obtain the area where the construction machinery can travel. A 2D mesh is drawn for the drivable area of the engineering machinery to obtain the drivable area in 2D mesh state.
3. The method according to claim 1, characterized in that, After drawing the driving area into a 2D mesh state based on the horizontal ground area and the object area, the method further includes: Draw the top view of the pre-built mechanical 3D model; The driving area in the 2D mesh state is merged into the corresponding position around the top view of the model, and the driving area in the 2D mesh state is colored according to the preset color rendering parameters to obtain the target top view. A top-down view of the target is displayed.
4. A device for generating the driving area of engineering machinery using the method described in any one of claims 1-3, characterized in that, The device includes: The sensor parameter receiving module is used to receive in real time the initial images, initial scene point clouds, and relative angles of mechanical components sensed by sensors pre-installed on the engineering machinery. The parameter preprocessing module is used to preprocess the initial image and initial scene point cloud according to the relative angle of the mechanical parts, and generate the target image and target scene point cloud. The region segmentation module is used to segment the horizontal ground region and the object region based on the target image and the target scene point cloud; The driving area drawing module is used to draw a 2D mesh driving area based on the horizontal ground area and the object area.
5. A computer storage medium, characterized in that, The computer storage medium stores a plurality of instructions adapted for loading by a processor and executing the method as described in any one of claims 1-3.
6. A terminal, characterized in that, include: A processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and executed as described in any one of claims 1-3.