Charge level identification method and device, electronic equipment and computer readable storage medium

Abstract

The invention provides a charge level identification method and device, electronic equipment and a computer readable storage medium, and the method comprises the steps: aiming at each grid in an elevation map containing a shoveling operation site, according to the height information of a position corresponding to the grid and a height variance value, calculating the height variance value of the position corresponding to the grid; calculating the first height category confidence of the position corresponding to the grid belonging to the ground and the second height category confidence of the position corresponding to the grid belonging to the material pile; inputting an image containing a shoveling operation site into the semantic segmentation model to determine a third semantic category confidence coefficient of the position corresponding to each grid belonging to the ground and a fourth semantic category confidence coefficient of the position corresponding to each grid belonging to a material pile; and according to the first height category confidence and the third semantic category confidence of the position corresponding to the grid belonging to the ground, and the second height category confidence and the fourth semantic category confidence of the position corresponding to the grid belonging to the material pile, determining that the category of the position corresponding to the grid is the ground or the material pile. Through the method, the accuracy of a charge level identification result is improved.

Description

Technical Field

[0001] The present application relates to the field of intelligent detection technologies, and in particular, to a method and device for identifying a material surface, an electronic device, and a computer-readable storage medium. Background Art

[0002] During the autonomous loading operation of an unmanned loader, material surface identification is a key step. Among them, in the loading operation site, the material pile is usually placed on the ground. Material surface identification refers to identifying the material pile area and the ground area in the loading operation site, so as to separate the material pile to be loaded from the ground. Currently, the methods for material surface identification mainly include identifying the material surface based on pure height information and identifying the material pile according to a semantic segmentation model.

[0003] Among them, during the process of identifying the material surface based on pure height information, a ground height threshold is set for identification. What is lower than the threshold is the ground, and what is higher than the threshold is the material pile. In this material surface identification method, when the set ground height threshold is inaccurate and there are height perception errors and other reasons, it may lead to the failure to identify thin material piles.

[0004] During the process of identifying the material pile according to the semantic segmentation model, since the semantic information is obtained based on the image captured by the camera, the identification accuracy is easily affected by the ambient light. Under the interference of the ambient light, incorrect material surface identification results are likely to occur. Summary of the Invention

[0005] In view of this, the purpose of the present application is to provide a method and device for identifying a material surface, an electronic device, and a computer-readable storage medium to improve the accuracy of the material surface identification result.

[0006] In a first aspect, an embodiment of the present application provides a method for identifying a material surface, including:

[0007] For each grid in the elevation map including the loading operation site, calculate a first height category confidence that the position corresponding to the grid belongs to the ground and a second height category confidence that the position corresponding to the grid belongs to the material pile according to the height information of the position corresponding to the grid and the height variance value used to characterize the confidence of the height information; wherein, each grid in the elevation map corresponds to each position in the loading operation site one by one;

[0008] Input the image including the loading operation site into the semantic segmentation model, and based on the first semantic category confidence that each pixel point in the image output by the semantic segmentation model belongs to the ground and the second semantic category confidence that each pixel point in the image belongs to the material pile, determine a third semantic category confidence that the position corresponding to each grid in the elevation map belongs to the ground and a fourth semantic category confidence that the position corresponding to each grid in the elevation map belongs to the material pile, wherein each pixel point in the image corresponds to each position in the loading operation site one by one;

[0009] For each of the grids, based on the confidence of the first height category indicating that the position corresponding to the grid belongs to the ground, the confidence of the third semantic category, the confidence of the second height category indicating that the position belongs to the material pile, and the confidence of the fourth semantic category, determine that the category to which the position corresponding to the grid belongs is the ground or the material pile.

[0010] In a second aspect, an embodiment of the present application further provides a material surface recognition device, including:

[0011] A calculation module, configured to calculate, for each grid in the elevation map including the shoveling operation site, the confidence of the first height category indicating that the position corresponding to the grid belongs to the ground and the confidence of the second height category indicating that the position belongs to the material pile according to the height information of the position corresponding to the grid and the height variance value used to characterize the confidence of the height information; wherein, each grid in the elevation map corresponds one-to-one to each position in the shoveling operation site;

[0012] A first determination module, configured to input an image including the shoveling operation site into a semantic segmentation model, and determine the confidence of the third semantic category indicating that the position corresponding to each grid in the elevation map belongs to the ground and the confidence of the fourth semantic category indicating that the position belongs to the material pile based on the confidence of the first semantic category indicating that each pixel point in the image output by the semantic segmentation model belongs to the ground and the confidence of the second semantic category indicating that the position belongs to the material pile; wherein, each pixel point in the image corresponds one-to-one to each position in the shoveling operation site;

[0013] A second determination module, configured to determine, for each of the grids, that the category to which the position corresponding to the grid belongs is the ground or the material pile according to the confidence of the first height category indicating that the position corresponding to the grid belongs to the ground, the confidence of the third semantic category, the confidence of the second height category indicating that the position belongs to the material pile, and the confidence of the fourth semantic category.

[0014] In a third aspect, an embodiment of the present application further provides an electronic device, including: a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device runs, the processor communicates with the memory through the bus. When the machine-readable instructions are executed by the processor, the steps in any possible implementation manner of the first aspect described above are executed.

[0015] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is run by a processor, the steps in any possible implementation manner of the first aspect described above are executed.

[0016] The material surface recognition method, device, electronic device, and computer-readable storage medium provided by the embodiments of the present application, when recognizing the ground and the material pile in the shoveling operation site, calculate the first height category confidence that the position corresponding to each grid in the elevation map belongs to the ground and the second height category confidence that belongs to the material pile through the height information of the position corresponding to each grid in the elevation map and the height variance value representing the confidence of the height information. Moreover, by inputting the image containing the shoveling operation site into the semantic segmentation model, based on the first semantic category confidence that each pixel point in the image output by the semantic segmentation model belongs to the ground and the second semantic category confidence that belongs to the material pile, the third semantic category confidence that the position corresponding to each grid in the elevation map belongs to the ground and the fourth semantic category confidence that belongs to the material pile are determined. Then, for each grid, according to the first height category confidence and the third semantic category confidence that the position corresponding to the grid belongs to the ground, and the second height category confidence and the fourth semantic category confidence that belong to the material pile, it is determined that the category to which the position corresponding to the grid belongs is the ground or the material pile. It can be seen that in this embodiment, the height information and the semantic information are fused, and the height category confidence and the semantic category confidence are comprehensively considered, so that low material piles (i.e., thin material piles) can be effectively recognized. At the same time, it can also resist the interference of environmental light on the semantic segmentation accuracy and improve the accuracy of material surface recognition.

[0017] To make the above objects, features, and advantages of the present application more obvious and understandable, the following specifically enumerates preferred embodiments and, in conjunction with the accompanying drawings, makes a detailed description as follows. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] To more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required to be used in the embodiments. It should be understood that the following drawings only show some embodiments of the present application and should not be regarded as limiting the scope. For those of ordinary skill in the art, other related drawings can be obtained based on these drawings without creative efforts.

[0019] Figure 1 Shows a flowchart of a material surface recognition method provided by an embodiment of the present application;

[0020] Figure 2 Shows a schematic diagram of a grid of an elevation map provided by an embodiment of the present application;

[0021] Figure 3 Shows a schematic structural diagram of a material surface recognition device provided by an embodiment of the present application;

[0022] Figure 4 Shows a schematic structural diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0023] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the following will clearly and completely describe the technical solutions in the embodiments of this application with reference to the accompanying drawings in the embodiments of this application. It should be understood that the accompanying drawings in this application only serve the purposes of illustration and description, and are not used to limit the protection scope of this application. Additionally, it should be understood that the schematic drawings are not drawn to the actual scale. The flowcharts used in this application illustrate the operations implemented according to some embodiments of this application. It should be understood that the operations in the flowchart may not be implemented in sequence, and steps without a logical context relationship may be reversed in order or implemented simultaneously. In addition, those skilled in the art may add one or more other operations to the flowchart or remove one or more operations from the flowchart under the guidance of the content of this application.

[0024] In addition, the described embodiments are only a part of the embodiments of this application, rather than all of the embodiments. The components of the embodiments of this application usually described and illustrated in the accompanying drawings here may be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents the selected embodiments of this application. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative efforts fall within the protection scope of this application.

[0025] It should be noted that the term "including" will be used in the embodiments of this application to indicate the existence of the subsequently stated features, but does not exclude the addition of other features.

[0026] During the process of identifying the material surface based on pure height information, a ground height threshold is set for identification. What is lower than the threshold is the ground, and what is higher than the threshold is the material pile. In this way of material surface identification, when the set ground height threshold is inaccurate and there are height perception errors and other reasons, it may lead to the failure to identify a thin material pile.

[0027] During the process of identifying the material pile according to the semantic segmentation model, since the semantic information is obtained based on the images captured by the camera, the recognition accuracy is easily affected by the ambient light. Under the interference of the ambient light, it is very easy to obtain incorrect material surface recognition results.

[0028] Based on this, the embodiments of this application provide a method, device, electronic device, and computer-readable storage medium for material surface identification. By fusing height information and semantic information, and comprehensively considering the height category confidence and semantic category confidence, it is possible to effectively identify low-lying material piles (i.e., thin material piles). At the same time, it can also resist the interference of ambient light on the semantic segmentation accuracy and improve the accuracy of the material surface recognition result.

[0029] A material surface recognition method in one of the embodiments of the present application can run on a terminal device or a server. Among them, the terminal device can be a local terminal device. When the material surface recognition method runs on the server, the material surface recognition method can be implemented and executed based on a cloud interaction system, where the cloud interaction system includes a server and a client device (i.e., the terminal device).

[0030] To facilitate the understanding of this embodiment, a material surface recognition method, device, electronic device, and computer-readable storage medium disclosed in the embodiments of the present application will be introduced in detail below.

[0031] As Figure 1 shown Figure 1 shows a flowchart of a material surface recognition method provided by an embodiment of the present application. Among them, the material surface recognition method includes the following steps S101-S103:

[0032] S101: For each grid in the elevation map including the material shoveling operation site, calculate the first height category confidence that the position corresponding to the grid belongs to the ground and the second height category confidence that belongs to the material pile according to the height information of the position corresponding to the grid and the height variance value used to characterize the confidence of the height information; where each grid in the elevation map corresponds to each position in the material shoveling operation site one by one.

[0033] S102: Input the image including the material shoveling operation site into the semantic segmentation model, and based on the first semantic category confidence that each pixel point in the image output by the semantic segmentation model belongs to the ground and the second semantic category confidence that belongs to the material pile, determine the third semantic category confidence that the position corresponding to each grid in the elevation map belongs to the ground and the fourth semantic category confidence that belongs to the material pile, where each pixel point in the image corresponds to each position in the material shoveling operation site one by one.

[0034] S103: For each grid, determine that the category to which the position corresponding to the grid belongs is the ground or the material pile according to the first height category confidence and the third semantic category confidence that the position corresponding to the grid belongs to the ground, and the second height category confidence and the fourth semantic category confidence that belong to the material pile.

[0035] In step S101, the material shoveling operation site includes the ground and the material pile placed on the ground. The purpose of material surface recognition is to identify the material pile and the ground in the material shoveling operation site, so as to control the unmanned loader to shovel up the material pile in the material shoveling operation site.

[0036] Figure 2 shows a schematic diagram of a grid of an elevation map provided by an embodiment of the present application, as Figure 2As shown, the elevation map of the material shoveling operation site is a planar grid map, and the perspective of this planar grid map is a top-down view of the ground. The elevation map contains multiple grids, each grid corresponding to a position on the plane corresponding to the material shoveling operation site, and each grid contains the height information of the position on the plane corresponding to the material shoveling operation site corresponding to this grid, as well as the height variance value used to characterize the confidence of this height information.

[0037] Among them, the height information of the position on the plane corresponding to the grid corresponding to the material shoveling operation site may refer to the vertical height of the position corresponding to this grid relative to the ground. For example, if the position corresponding to the grid is the ground, then the height information of the position corresponding to this grid may be 0; if the position corresponding to the grid is a material pile, then the height information of the position corresponding to this grid may be 0.2 meters. The height variance value corresponding to the grid is used to characterize the confidence (i.e., the credibility) of the height information of the position corresponding to this grid, and each height information corresponds to its own height variance value.

[0038] In this embodiment, the number of grids in the elevation map is usually relatively large. The more grids there are, the finer the material shoveling operation site is divided. In a possible implementation manner, the number of grids in the elevation map can be determined by the size of the material shoveling operation site and a preset resolution. Among them, the number of grids is positively correlated with the size of the material shoveling operation site and positively correlated with the preset resolution.

[0039] In this embodiment, the confidence of the first height category that the position corresponding to the grid belongs to the ground refers to: the probability that the position is determined to be the ground through the height information of the position corresponding to the grid. The confidence of the second height category that the position corresponding to the grid belongs to the material pile refers to: the probability that the position is determined to be the material pile through the height information of the position corresponding to the grid.

[0040] In step S102, an image containing the material shoveling operation site is collected by an image acquisition device, and the collected image is input into a pre-trained semantic segmentation model. Through the semantic segmentation model, the confidence of the first semantic category that each pixel point in the image belongs to the ground and the confidence of the second semantic category that belongs to the material pile are output. Among them, each pixel point in the image corresponds one-to-one to each position in the material shoveling operation site. The confidence of the first semantic category that the pixel point belongs to the ground (i.e., the position corresponding to the pixel point belongs to the ground) refers to: the probability that the position corresponding to this pixel point is determined to be the ground through the semantic segmentation model. The confidence of the second semantic category that the pixel point belongs to the material pile (i.e., the position corresponding to the pixel point belongs to the material pile) refers to: the probability that the position corresponding to this pixel point is determined to be the material pile through the semantic segmentation model.

[0041] In this embodiment, since each pixel point in the image has a corresponding relationship with each position in the material shoveling operation site, and each grid in the elevation map also has a corresponding relationship with each position in the material shoveling operation site, therefore, each pixel point in the image can be projected onto each grid in the elevation map. Thus, according to the confidence of the first semantic category belonging to the ground and the confidence of the second semantic category belonging to the material pile for each pixel point in the image, the confidence of the third semantic category belonging to the ground and the confidence of the fourth semantic category belonging to the material pile corresponding to each position in the elevation map can be determined. Among them, the confidence of the third semantic category that the position corresponding to the grid belongs to the ground refers to: the probability that the position corresponding to the grid belongs to the ground determined by the recognition result of the semantic segmentation model. The confidence of the fourth semantic category that the position corresponding to the grid belongs to the material pile refers to: the probability that the position corresponding to the grid belongs to the material pile determined by the recognition result of the semantic segmentation model.

[0042] It should be noted that the execution order of steps S101 and S102 is not fixed. Step S101 can be executed first and then step S102, or step S102 can be executed first and then step S101, or steps S101 and S102 can be executed simultaneously.

[0043] In step S103, for each grid, calculate the sum of the confidence of the first height category belonging to the ground and the confidence of the third semantic category corresponding to the position of this grid, to obtain the first total confidence that the position corresponding to this grid belongs to the ground, and calculate the sum of the confidence of the second height category belonging to the material pile and the confidence of the fourth semantic category corresponding to the position of this grid, to obtain the second total confidence that the position corresponding to this grid belongs to the material pile. Among them, the first total confidence that the position corresponding to the grid belongs to the ground refers to: the probability that this position is the ground determined by comprehensively considering height information and semantic information. The second total confidence that the position corresponding to the grid belongs to the material pile refers to: the probability that this position is the material pile determined by comprehensively considering height information and semantic information.

[0044] If the first total confidence is greater than the second total confidence, it indicates that the category to which the position corresponding to this grid belongs is the ground; if the first total confidence is less than the second total confidence, it indicates that the category to which the position corresponding to this grid belongs is the material pile. In this way, it is possible to determine whether the category to which each grid in the elevation map belongs is the material pile or the ground.

[0045] In a possible implementation manner, before executing step S102 and step S102, the elevation map of the material shoveling operation site can also be established according to the following steps S1001 - S1003:

[0046] S1001: Collect the laser point cloud data of the material shoveling operation site at each acquisition moment respectively;

[0047] S1002: When the lidar point cloud data is the lidar point cloud data collected at the first acquisition moment, based on the lidar point cloud data collected at the first acquisition moment, establish a topographic map of the material shoveling operation site corresponding to the lidar point cloud data;

[0048] S1003: When the lidar point cloud data is the lidar point cloud data collected at a non-first acquisition moment, use the lidar point cloud data collected at the non-first acquisition moment to dynamically update the topographic map of the material shoveling operation site corresponding to the lidar point cloud data collected at the previous acquisition moment, and obtain the topographic map of the material shoveling operation site corresponding to the lidar point cloud data.

[0049] In this embodiment, the unmanned loader drives the 3D lidar to move within the material shoveling operation site, and during the movement, the 3D lidar is used to collect the lidar point cloud data of the material shoveling operation site at each acquisition moment. First, based on the lidar point cloud data collected at the first acquisition moment, establish a topographic map of the material shoveling operation site corresponding to the lidar point cloud data. Then, in the order of the lidar point cloud data collected from the front to the back, sequentially use the lidar point cloud data collected at each non-first acquisition moment to dynamically update the height information and height variance value of the position corresponding to each grid in the topographic map of the material shoveling operation site corresponding to the lidar point cloud data collected at the previous acquisition moment, so as to optimize the height information and height variance value of the position corresponding to each grid in the topographic map of the material shoveling operation site.

[0050] In a possible implementation manner, when executing step S101, it can be specifically executed according to the following steps S1011 - S1014:

[0051] S1011: According to the preset ground height threshold and the height information of the position corresponding to each grid, extract the candidate grids with height information less than or equal to the preset ground height threshold from each grid included in the topographic map.

[0052] S1012: Filter out the candidate grids with abnormal height information according to the height information of each candidate grid to obtain the target grids.

[0053] S1013: Use each target grid as a ground grid for plane fitting to determine the plane equation of the ground, and calculate the variance of the ground fitting according to the height information of each target grid.

[0054] S1014: For each grid in the topographic map of the material shoveling operation site, calculate the first height category confidence that the position corresponding to the grid belongs to the ground and the second height category confidence that it belongs to the material pile according to the height information of the position corresponding to the grid, the height variance value representing the confidence of the height information, the plane equation, and the variance of the ground fitting.

[0055] In step S1011, the preset ground height threshold is a value determined based on experience. For example, the preset ground height threshold is set to 0.5 meters. The grids in the elevation map with height information less than or equal to the preset ground height threshold are extracted as candidate grids.

[0056] In step S1012, the RANSAC method (Random Sample Consensus algorithm) is used to filter out the candidate grids with abnormal height information, and the remaining candidate grids are used as target grids.

[0057] In step S1013, each target grid is used as a ground grid for plane fitting to determine the plane equation of the ground: Z ′ = Z ′ (x, y); and according to the height information of each target grid, the variance of the ground fitting is calculated

[0058] In step S1014, for each grid in the elevation map of the material shoveling operation site, the confidence p of the first height category that the position corresponding to the grid belongs to the ground 1 and the confidence p of the second height category that belongs to the material pile 2 .

[0059]

[0060] p 2 = 1 - p 1

[0061] where h represents the height information of the position corresponding to the grid; the height variance value used to characterize the confidence of the height information; Z ′ represents the plane equation; is used to characterize the variance of the ground fitting.

[0062] In a possible implementation manner, when executing step S102, it can be specifically executed according to the following steps S1021 - S1025:

[0063] S1021: Use an image acquisition device to acquire an image containing the material shoveling operation site.

[0064] In this embodiment, the image acquisition device can be a camera, and an image containing the material shoveling operation site is captured by the image acquisition device.

[0065] S1022: Input the image into the semantic segmentation model, and through the semantic segmentation model, determine the confidence of the first semantic category that each pixel point in the image belongs to the ground and the confidence of the second semantic category that belongs to the material pile.

[0066] S1023: Project each pixel point in the image onto the elevation map according to the external parameters of the image acquisition device, so as to establish the correspondence between each pixel point in the image and each grid in the elevation map; among them, the pixel point and the grid with a correspondence relationship correspond to the same position in the material shoveling operation site, and the same position can be a point or a range of areas.

[0067] S1024: For each grid in the elevation map, if the grid corresponds to a pixel point, then use the confidence of the first semantic category that the pixel point belongs to the ground as the confidence of the third semantic category that the position corresponding to the grid belongs to the ground, and use the confidence of the second semantic category that the pixel point belongs to the material pile as the confidence of the fourth semantic category that the position corresponding to the grid belongs to the material pile.

[0068] S1025: If the grid corresponds to multiple pixel points, then determine the confidence of the third semantic category that the position corresponding to the grid belongs to the ground according to the confidence of the first semantic category that the multiple pixel points belong to the ground, and determine the confidence of the fourth semantic category that the position corresponding to the grid belongs to the material pile according to the confidence of the second semantic category that the multiple pixel points belong to the material pile.

[0069] In this embodiment, when executing step S1025, the confidence of the third semantic category that the position corresponding to the grid belongs to the ground and the confidence of the fourth semantic category that the position corresponding to the grid belongs to the material pile can be determined in the following manner:

[0070] If the grid corresponds to multiple pixel points, then according to the arrangement order of the multiple pixel points in the image (for example, the arrangement order of the multiple pixel points in the image can be arranged in the order from top to bottom and from left to right), determine the target pixel point with the last arrangement order from the multiple pixel points (for example, the target pixel point with the last arrangement order determined from the multiple pixel points is the pixel point located in the lower right corner of the multiple pixel points), use the confidence of the first semantic category that the target pixel point belongs to the ground as the confidence of the third semantic category that the position corresponding to the grid belongs to the ground, and use the confidence of the second semantic category that the target pixel point belongs to the material pile as the confidence of the fourth semantic category that the position corresponding to the grid belongs to the material pile.

[0071] Or, if the grid corresponds to multiple pixel points, then calculate the average value of the confidence of the first semantic category that the multiple pixel points belong to the ground to obtain a first value, use the first value as the confidence of the third semantic category that the position corresponding to the grid belongs to the ground, and calculate the average value of the confidence of the second semantic category that the multiple pixel points belong to the material pile to obtain a second value, use the second value as the confidence of the fourth semantic category that the position corresponding to the grid belongs to the material pile.

[0072] Alternatively, if the grid corresponds to multiple pixel points, randomly select a first semantic class confidence that each pixel point among the multiple pixel points belongs to the ground as the third semantic class confidence that the position corresponding to the grid belongs to the ground, and randomly select a second semantic class confidence that each pixel point among the multiple pixel points belongs to the material pile as the fourth semantic class confidence that the position corresponding to the grid belongs to the material pile.

[0073] In a possible implementation, considering that there may be obstacles in the material shoveling operation site, if the driverless loader hits an obstacle during the movement in the material shoveling operation site, it will affect the driving safety of the driverless loader. Based on this, after performing step S102 to input the image including the material shoveling operation site into the semantic segmentation model, the following steps can also be performed:

[0074] Based on the fifth semantic class confidence that each pixel point in the image output by the semantic segmentation model belongs to the obstacle, determine the sixth semantic class confidence that the position corresponding to each grid in the elevation map belongs to the obstacle.

[0075] In this embodiment, after inputting the image into the semantic segmentation model, the fifth semantic class confidence that each pixel point in the image belongs to the obstacle can also be determined by the semantic segmentation model. Among them, the fifth semantic class confidence that a pixel point belongs to the obstacle (that is, the position corresponding to the pixel point belongs to the obstacle) refers to the probability that the position corresponding to the pixel point identified by the semantic segmentation model is the position where the obstacle is located.

[0076] After projecting each pixel point in the image onto the elevation map according to the external parameters of the image acquisition device to establish the correspondence between each pixel point in the image and each grid in the elevation map, for each grid in the elevation map, if the grid corresponds to a single pixel point, use the fifth semantic class confidence that the pixel point belongs to the obstacle as the sixth semantic class confidence that the position corresponding to the grid belongs to the obstacle. If the grid corresponds to multiple pixel points, determine the target pixel point with the last arrangement order from the multiple pixel points according to the arrangement order of the multiple pixel points in the image, and use the fifth semantic class confidence that the target pixel point belongs to the obstacle as the sixth semantic class confidence that the position corresponding to the grid belongs to the obstacle.

[0077] At this time, when performing step S103, the following steps S1031 - S1032 can be specifically performed:

[0078] S1031: For each grid, determine whether the category to which the position corresponding to the grid belongs is an obstacle according to the sixth semantic class confidence that the position corresponding to the grid belongs to the obstacle.

[0079] In this embodiment, for each grid, when the confidence of the sixth semantic category that the position corresponding to the grid belongs to an obstacle is greater than the confidence of the third semantic category that the position corresponding to the grid belongs to the ground and greater than the confidence of the fourth semantic category that the position corresponding to the grid belongs to a material pile, the category to which the position corresponding to the grid belongs is determined as an obstacle.

[0080] When the confidence of the sixth semantic category that the position corresponding to the grid belongs to an obstacle is less than or equal to the confidence of the third semantic category that the position corresponding to the grid belongs to the ground, and / or less than or equal to the confidence of the fourth semantic category that the position corresponding to the grid belongs to a material pile, it is determined that the category to which the position corresponding to the grid belongs is not an obstacle.

[0081] S1032: If the category to which the position corresponding to the grid belongs is not an obstacle, then based on the confidence of the first height category and the third semantic category that the position corresponding to the grid belongs to the ground, and the confidence of the second height category and the fourth semantic category that the position corresponding to the grid belongs to a material pile, determine that the category to which the position corresponding to the grid belongs is the ground or a material pile.

[0082] In this embodiment, if the category to which the position corresponding to the grid belongs is not an obstacle, then calculate the sum of the confidence of the first height category and the third semantic category that the position corresponding to the grid belongs to the ground to obtain the first total confidence that the position corresponding to the grid belongs to the ground, and calculate the sum of the confidence of the second height category and the fourth semantic category that the position corresponding to the grid belongs to a material pile to obtain the second total confidence that the position corresponding to the grid belongs to a material pile.

[0083] When the first total confidence is greater than or equal to the second total confidence, determine that the category to which the position corresponding to the grid belongs is the ground.

[0084] When the first total confidence is less than the second total confidence, determine that the category to which the position corresponding to the grid belongs is a material pile.

[0085] In a possible implementation manner, after determining the category to which the position corresponding to each grid in the elevation map belongs, the following steps S1041 - S1043 can also be executed:

[0086] S1041: Based on the category to which the position corresponding to each grid belongs, determine the material pile area, ground area, and obstacle area in the material shoveling operation site.

[0087] In this embodiment, the material pile area in the material shoveling operation site is determined according to the positions corresponding to the grids whose categories are material piles. The ground area in the material shoveling operation site is determined according to the positions corresponding to the grids whose categories are the ground. The obstacle area in the material shoveling operation site is determined according to the positions corresponding to the grids whose categories are obstacles.

[0088] S1042: Control the unmanned loader to avoid the obstacle area and drive through the ground area to a preset distance from the material pile area.

[0089] Exemplarily, the preset distance can be 0.5 meters, 1 meter, etc., which is convenient for the unmanned loader to shovel materials.

[0090] S1043: Control the bucket of the unmanned loader to be pushed into the material pile in the material pile area and shovel up the materials in the material pile area, so that the unmanned loader can perform the material shoveling operation.

[0091] Based on the same inventive concept, the present application also provides a material surface recognition device corresponding to the above-mentioned material surface recognition method. Since the principle of solving problems by the material surface recognition device in the embodiments of the present application is similar to that of the above-mentioned material surface recognition method in the embodiments of the present application, the implementation of the material surface recognition device can refer to the implementation of the above-mentioned material surface recognition method, and the repeated parts will not be described again.

[0092] Refer to Figure 3 as shown in Figure 3 The structural schematic diagram of a material surface recognition device provided by an embodiment of the present application is shown. Among them, the material surface recognition device includes:

[0093] A calculation module 301, configured to calculate, for each grid in the elevation map including the material shoveling operation site, a first height category confidence that the position corresponding to the grid belongs to the ground and a second height category confidence that the position corresponding to the grid belongs to the material pile according to the height information of the position corresponding to the grid and the height variance value for characterizing the confidence of the height information; wherein, each grid in the elevation map corresponds one-to-one to each position in the material shoveling operation site;

[0094] A first determination module 302, configured to input the image including the material shoveling operation site into the semantic segmentation model, and determine a third semantic category confidence that the position corresponding to each grid in the elevation map belongs to the ground and a fourth semantic category confidence that the position corresponding to each grid in the elevation map belongs to the material pile based on the first semantic category confidence that each pixel point in the image belongs to the ground and the second semantic category confidence that each pixel point in the image belongs to the material pile output by the semantic segmentation model, wherein, each pixel point in the image corresponds one-to-one to each position in the material shoveling operation site;

[0095] A second determination module 303, configured to, for each of the grids, determine that the position corresponding to the grid belongs to the ground or the material pile according to the confidence of the first height category of the position corresponding to the grid belonging to the ground, the confidence of the third semantic category, the confidence of the second height category of the position belonging to the material pile, and the confidence of the fourth semantic category.

[0096] In an alternative embodiment, the material surface recognition device further includes:

[0097] An acquisition module, configured to acquire the laser point cloud data of the material shoveling operation site at each acquisition moment respectively;

[0098] A building module, configured to, when the laser point cloud data is the laser point cloud data acquired at the first acquisition moment, build the elevation map of the material shoveling operation site corresponding to the laser point cloud data based on the laser point cloud data acquired at the first acquisition moment;

[0099] An update module, configured to, when the laser point cloud data is the laser point cloud data acquired at a non-first acquisition moment, dynamically update the elevation map of the material shoveling operation site corresponding to the laser point cloud data acquired at the previous acquisition moment by using the laser point cloud data acquired at the non-first acquisition moment, so as to obtain the elevation map of the material shoveling operation site corresponding to the laser point cloud data.

[0100] In an alternative embodiment, when the calculation module 301 is configured to, for each grid in the elevation map of the material shoveling operation site, calculate the confidence of the first height category of the position corresponding to the grid belonging to the ground and the confidence of the second height category of the position belonging to the material pile according to the height information of the position corresponding to the grid and the height variance value of the confidence for characterizing the height information, it is specifically configured to:

[0101] According to a preset ground height threshold and the height information of the position corresponding to each grid, extract candidate grids with height information less than or equal to the preset ground height threshold from each grid included in the elevation map;

[0102] Filter out candidate grids with abnormal height information according to the height information of each of the candidate grids to obtain target grids;

[0103] Perform plane fitting on each of the target grids as ground grids to determine the plane equation of the ground, and calculate the variance of the ground fitting according to the height information of each of the target grids;

[0104] For each grid in the elevation map of the material shoveling operation site, calculate the first height category confidence that the position corresponding to the grid belongs to the ground and the second height category confidence that the position belongs to the material pile according to the height information of the position corresponding to the grid, the height variance value used to characterize the confidence of the height information, and the variance between the plane equation and the ground fitting.

[0105] In an alternative embodiment, when the first determination module 302 is configured to input the image including the material shoveling operation site into the semantic segmentation model, and determine the third semantic category confidence that the position corresponding to each grid in the elevation map belongs to the ground and the fourth semantic category confidence that the position belongs to the material pile based on the first semantic category confidence that each pixel point in the image output by the semantic segmentation model belongs to the ground and the second semantic category confidence that the position belongs to the material pile, it is specifically configured to:

[0106] Collect an image including the material shoveling operation site through an image acquisition device;

[0107] Input the image into the semantic segmentation model, and determine the first semantic category confidence that each pixel point in the image belongs to the ground and the second semantic category confidence that the position belongs to the material pile through the semantic segmentation model;

[0108] Project each pixel point in the image onto the elevation map according to the external parameters of the device of the image acquisition device to establish a correspondence between each pixel point in the image and each grid in the elevation map; wherein, the pixel point and the grid with the correspondence correspond to the same position in the material shoveling operation site;

[0109] For each grid in the elevation map, if the grid corresponds to a pixel point, then use the first semantic category confidence that the pixel point belongs to the ground as the third semantic category confidence that the position corresponding to the grid belongs to the ground, and use the second semantic category confidence that the pixel point belongs to the material pile as the fourth semantic category confidence that the position corresponding to the grid belongs to the material pile;

[0110] If the grid corresponds to multiple pixel points, then determine the third semantic category confidence that the position corresponding to the grid belongs to the ground according to the first semantic category confidence that the multiple pixel points belong to the ground, and determine the fourth semantic category confidence that the position corresponding to the grid belongs to the material pile according to the second semantic category confidence that the multiple pixel points belong to the material pile.

[0111] In an alternative embodiment, the material surface recognition device further includes:

[0112] A third determination module, configured to, after the first determination module 302 inputs an image including the image of the material shoveling operation site into a semantic segmentation model, determine, based on the fifth semantic class confidence that each pixel point in the image output by the semantic segmentation model belongs to an obstacle, the sixth semantic class confidence that the position corresponding to each grid in the elevation map belongs to an obstacle;

[0113] When the second determination module 303 is configured to, for each of the grids, determine that the class to which the position corresponding to the grid belongs is the ground or a material pile according to the first height class confidence and the third semantic class confidence that the position corresponding to the grid belongs to the ground, and the second height class confidence and the fourth semantic class confidence that the position corresponding to the grid belongs to a material pile, it is specifically configured to:

[0114] For each of the grids, determine whether the class to which the position corresponding to the grid belongs is an obstacle according to the sixth semantic class confidence that the position corresponding to the grid belongs to an obstacle;

[0115] If the class to which the position corresponding to the grid belongs is not an obstacle, then determine that the class to which the position corresponding to the grid belongs is the ground or a material pile according to the first height class confidence and the third semantic class confidence that the position corresponding to the grid belongs to the ground, and the second height class confidence and the fourth semantic class confidence that the position corresponding to the grid belongs to a material pile.

[0116] In an alternative embodiment, when the second determination module 303 is configured to determine that the class to which the position corresponding to the grid belongs is the ground or a material pile according to the first height class confidence and the third semantic class confidence that the position corresponding to the grid belongs to the ground, and the second height class confidence and the fourth semantic class confidence that the position corresponding to the grid belongs to a material pile, it is specifically configured to:

[0117] Calculate the sum value of the first height class confidence and the third semantic class confidence that the position corresponding to the grid belongs to the ground, to obtain the first total confidence that the position corresponding to the grid belongs to the ground, and calculate the sum value of the second height class confidence and the fourth semantic class confidence that the position corresponding to the grid belongs to a material pile, to obtain the second total confidence that the position corresponding to the grid belongs to a material pile;

[0118] When the first total confidence is greater than or equal to the second total confidence, determine that the class to which the position corresponding to the grid belongs is the ground;

[0119] When the first total confidence is less than the second total confidence, determine that the class to which the position corresponding to the grid belongs is a material pile.

[0120] In an alternative embodiment, when the second determination module 303 is used to determine, for each of the grids, whether the position corresponding to the grid belongs to the category of obstacle according to the confidence of the sixth semantic category that the position corresponding to the grid belongs to the obstacle, it is specifically configured to:

[0121] For each of the grids, when the confidence of the sixth semantic category that the position corresponding to the grid belongs to the obstacle is greater than the confidence of the third semantic category that the position corresponding to the grid belongs to the ground and greater than the confidence of the fourth semantic category that the position corresponding to the grid belongs to the material pile, determine that the category to which the position corresponding to the grid belongs is an obstacle;

[0122] When the confidence of the sixth semantic category that the position corresponding to the grid belongs to the obstacle is less than or equal to the confidence of the third semantic category that the position corresponding to the grid belongs to the ground, and / or less than or equal to the confidence of the fourth semantic category that the position corresponding to the grid belongs to the material pile, it is determined that the category to which the position corresponding to the grid belongs is not an obstacle.

[0123] In an alternative embodiment, the material surface recognition device further includes:

[0124] A fourth determination module, configured to, after the second determination module determines the category to which the position corresponding to each grid belongs, determine the material pile area, the ground area, and the obstacle area in the material shoveling operation site according to the category to which the position corresponding to each grid belongs;

[0125] A first control module, configured to control the unmanned loader to avoid the obstacle area and drive through the ground area to a preset distance from the material pile area;

[0126] A second control module, configured to control the bucket of the unmanned loader to be pushed into the material pile in the material pile area and shovel up the material in the material pile area, so that the unmanned loader performs the operation of shoveling and loading materials.

[0127] Based on the above-mentioned material surface recognition device provided by the embodiments of the present application, by fusing height information and semantic information and comprehensively considering the height category confidence and the semantic category confidence, it is possible to effectively identify low material piles (i.e., thin material piles). At the same time, it can also resist the interference of environmental light on the accuracy of semantic segmentation and improve the accuracy of the material surface recognition result.

[0128] Based on the same inventive concept, the present application also provides an electronic device corresponding to the above-mentioned material surface recognition method. Since the principle of solving problems by the electronic device in the embodiments of the present application is similar to that of the above-mentioned material surface recognition method in the embodiments of the present application, the implementation of the electronic device can refer to the implementation of the above-mentioned material surface recognition method, and the repeated parts will not be described again.

[0129] Figure 4 The following is a schematic structural diagram of an electronic device 400 provided by an embodiment of the present application, including: a processor 401, a memory 402, and a bus 403. The memory 402 stores machine-readable instructions executable by the processor 401. When the electronic device runs an audio generation method as in an embodiment, communication between the processor 401 and the memory 402 is through the bus 403, and the processor 401 executes the machine-readable instructions. Among them, when the processor 401 executes the machine-readable instructions, the following steps are implemented. Specifically:

[0130] For each grid in the elevation map of the shoveling operation site, according to the height information of the position corresponding to the grid and the height variance value used to characterize the confidence of the height information, calculate the first height category confidence that the position corresponding to the grid belongs to the ground and the second height category confidence that belongs to the material pile; wherein, each grid in the elevation map corresponds one-to-one with each position in the shoveling operation site;

[0131] Input the image including the shoveling operation site into a semantic segmentation model, and based on the first semantic category confidence that each pixel point in the image output by the semantic segmentation model belongs to the ground and the second semantic category confidence that belongs to the material pile, determine the third semantic category confidence that the position corresponding to each grid in the elevation map belongs to the ground and the fourth semantic category confidence that belongs to the material pile, wherein each pixel point in the image corresponds one-to-one with each position in the shoveling operation site;

[0132] For each of the grids, according to the first height category confidence and the third semantic category confidence that the position corresponding to the grid belongs to the ground, and the second height category confidence and the fourth semantic category confidence that belong to the material pile, determine that the category to which the position corresponding to the grid belongs is the ground or the material pile.

[0133] In an alternative embodiment, the processor 401 is further configured to:

[0134] Collect the laser point cloud data of the shoveling operation site at each acquisition moment respectively;

[0135] When the laser point cloud data is the laser point cloud data collected at the first acquisition moment, based on the laser point cloud data collected at the first acquisition moment, establish the elevation map of the shoveling operation site corresponding to the laser point cloud data;

[0136] When the laser point cloud data is the laser point cloud data collected at a non-first acquisition moment, use the laser point cloud data collected at the non-first acquisition moment to dynamically update the elevation map of the shoveling operation site corresponding to the laser point cloud data collected at the previous acquisition moment, and obtain the elevation map of the shoveling operation site corresponding to the laser point cloud data.

[0137] In an alternative embodiment, when the processor 401 is configured to calculate, for each grid in the elevation map of the material shoveling operation site, a first height category confidence that the position corresponding to the grid belongs to the ground and a second height category confidence that the position corresponding to the grid belongs to the material pile according to the height information of the position corresponding to the grid and the height variance value used to characterize the confidence of the height information, the processor 401 is specifically configured to:

[0138] According to a preset ground height threshold and the height information of the position corresponding to each grid, candidate grids with height information less than or equal to the preset ground height threshold are extracted from each grid included in the elevation map;

[0139] According to the height information of each of the candidate grids, the candidate grids with abnormal height information are filtered out to obtain target grids;

[0140] Perform plane fitting on each of the target grids as ground grids to determine the plane equation of the ground, and calculate the variance of the ground fitting according to the height information of each of the target grids;

[0141] For each grid in the elevation map of the material shoveling operation site, calculate a first height category confidence that the position corresponding to the grid belongs to the ground and a second height category confidence that the position corresponding to the grid belongs to the material pile according to the height information of the position corresponding to the grid, the height variance value used to characterize the confidence of the height information, the plane equation, and the variance of the ground fitting.

[0142] In an alternative embodiment, when the processor 401 is configured to input the image including the material shoveling operation site into the semantic segmentation model and determine a third semantic category confidence that the position corresponding to each grid in the elevation map belongs to the ground and a fourth semantic category confidence that the position corresponding to each grid in the elevation map belongs to the material pile based on the first semantic category confidence that each pixel point in the image output by the semantic segmentation model belongs to the ground and the second semantic category confidence that each pixel point in the image output by the semantic segmentation model belongs to the material pile, the processor 401 is specifically configured to:

[0143] Collect an image including the material shoveling operation site through an image acquisition device;

[0144] Input the image into the semantic segmentation model, and determine, through the semantic segmentation model, a first semantic category confidence that each pixel point in the image belongs to the ground and a second semantic category confidence that each pixel point in the image belongs to the material pile;

[0145] Project each pixel point in the image onto the elevation map according to the external parameters of the device of the image acquisition device to establish a correspondence between each pixel point in the image and each grid in the elevation map; wherein, the pixel point and the grid with the correspondence correspond to the same position in the material shoveling operation site.

[0146] For each grid in the elevation map, if the grid corresponds to a pixel, the confidence of the first semantic category that the pixel belongs to the ground is used as the confidence of the third semantic category that the position corresponding to the grid belongs to the ground, and the confidence of the second semantic category that the pixel belongs to the material pile is used as the confidence of the fourth semantic category that the position corresponding to the grid belongs to the material pile;

[0147] If the grid corresponds to multiple pixels, the confidence of the third semantic category that the position corresponding to the grid belongs to the ground is determined according to the confidence of the first semantic category that the multiple pixels belong to the ground, and the confidence of the fourth semantic category that the position corresponding to the grid belongs to the material pile is determined according to the confidence of the second semantic category that the multiple pixels belong to the material pile.

[0148] In an alternative embodiment, after the processor 401 is used to input the image including the material shoveling operation site into the semantic segmentation model, it is further used for:

[0149] Based on the confidence of the fifth semantic category that each pixel in the image output by the semantic segmentation model belongs to an obstacle, determine the confidence of the sixth semantic category that the position corresponding to each grid in the elevation map belongs to an obstacle;

[0150] When the processor 401 is used to determine, for each grid, that the category to which the position corresponding to the grid belongs is the ground or the material pile according to the confidence of the first height category and the confidence of the third semantic category that the position corresponding to the grid belongs to the ground, and the confidence of the second height category and the confidence of the fourth semantic category that the position corresponding to the grid belongs to the material pile, it is specifically used for:

[0151] For each grid, determine whether the category to which the position corresponding to the grid belongs is an obstacle according to the confidence of the sixth semantic category that the position corresponding to the grid belongs to an obstacle;

[0152] If the category to which the position corresponding to the grid belongs is not an obstacle, determine that the category to which the position corresponding to the grid belongs is the ground or the material pile according to the confidence of the first height category and the confidence of the third semantic category that the position corresponding to the grid belongs to the ground, and the confidence of the second height category and the confidence of the fourth semantic category that the position corresponding to the grid belongs to the material pile.

[0153] In an alternative embodiment, when the processor 401 is used to determine that the category to which the position corresponding to the grid belongs is the ground or the material pile according to the confidence of the first height category and the confidence of the third semantic category that the position corresponding to the grid belongs to the ground, and the confidence of the second height category and the confidence of the fourth semantic category that the position corresponding to the grid belongs to the material pile, it is specifically used for:

[0154] Calculate the sum of the confidence of the first height category that the position corresponding to the grid belongs to the ground and the confidence of the third semantic category, to obtain the first total confidence that the position corresponding to the grid belongs to the ground, and calculate the sum of the confidence of the second height category that the position corresponding to the grid belongs to the material pile and the confidence of the fourth semantic category, to obtain the second total confidence that the position corresponding to the grid belongs to the material pile;

[0155] When the first total confidence is greater than or equal to the second total confidence, determine that the category to which the position corresponding to the grid belongs is the ground;

[0156] When the first total confidence is less than the second total confidence, determine that the category to which the position corresponding to the grid belongs is the material pile.

[0157] In an alternative embodiment, when the processor 401 is used to determine whether the category to which the position corresponding to each grid belongs is an obstacle according to the confidence of the sixth semantic category that the position corresponding to the grid belongs to an obstacle, it is specifically used for:

[0158] For each grid, when the confidence of the sixth semantic category that the position corresponding to the grid belongs to an obstacle is greater than the confidence of the third semantic category that the position corresponding to the grid belongs to the ground and greater than the confidence of the fourth semantic category that the position corresponding to the grid belongs to the material pile, determine that the category to which the position corresponding to the grid belongs is an obstacle;

[0159] When the confidence of the sixth semantic category that the position corresponding to the grid belongs to an obstacle is less than or equal to the confidence of the third semantic category that the position corresponding to the grid belongs to the ground, and / or less than or equal to the confidence of the fourth semantic category that the position corresponding to the grid belongs to the material pile, then determine that the category to which the position corresponding to the grid belongs is not an obstacle.

[0160] In an alternative embodiment, after the processor 401 is used to determine the category to which the position corresponding to each grid belongs, it is further used for:

[0161] Determine the material pile area, ground area, and obstacle area in the material shoveling operation site according to the category to which the position corresponding to each grid belongs;

[0162] Control the unmanned loader to avoid the obstacle area and drive through the ground area to a preset distance from the material pile area;

[0163] Control the bucket of the unmanned loader to be pushed into the material pile in the material pile area and shovel up the material in the material pile area, so that the unmanned loader performs the material shoveling operation.

[0164] Through the above electronic device provided by the embodiments of the present application, by fusing height information and semantic information, and comprehensively considering the height category confidence and semantic category confidence, it is possible to effectively identify low material piles (i.e., thin material piles). At the same time, it can also resist the interference of environmental light on the semantic segmentation accuracy and improve the accuracy of the material surface recognition result.

[0165] Based on the same inventive concept, the embodiments of the present application also provide a computer-readable storage medium, on which a computer program is stored. When the computer program is run by a processor, the processor executes the following steps:

[0166] For each grid in the elevation map including the shoveling operation site, according to the height information of the position corresponding to the grid and the height variance value representing the confidence of the height information, calculate the first height category confidence that the position corresponding to the grid belongs to the ground and the second height category confidence that belongs to the material pile; wherein, each grid in the elevation map corresponds one-to-one with each position in the shoveling operation site;

[0167] Input the image including the shoveling operation site into the semantic segmentation model, and based on the first semantic category confidence that each pixel point in the image belongs to the ground and the second semantic category confidence that belongs to the material pile output by the semantic segmentation model, determine the third semantic category confidence that the position corresponding to each grid in the elevation map belongs to the ground and the fourth semantic category confidence that belongs to the material pile, wherein, each pixel point in the image corresponds one-to-one with each position in the shoveling operation site;

[0168] For each of the grids, according to the first height category confidence and the third semantic category confidence that the position corresponding to the grid belongs to the ground, and the second height category confidence and the fourth semantic category confidence that belong to the material pile, determine that the category to which the position corresponding to the grid belongs is the ground or the material pile.

[0169] In an alternative embodiment, the processor is further configured to:

[0170] Collect the laser point cloud data of the shoveling operation site at each acquisition moment respectively;

[0171] When the laser point cloud data is the laser point cloud data collected at the first acquisition moment, based on the laser point cloud data collected at the first acquisition moment, establish the elevation map of the shoveling operation site corresponding to the laser point cloud data;

[0172] When the laser point cloud data is the laser point cloud data collected at a non-first acquisition time, the elevation map of the material shoveling operation site corresponding to the laser point cloud data collected at the previous acquisition time is dynamically updated using the laser point cloud data collected at the non-first acquisition time, to obtain the elevation map of the material shoveling operation site corresponding to this laser point cloud data.

[0173] In an alternative embodiment, when the processor is used to calculate, for each grid in the elevation map of the material shoveling operation site, a first height category confidence that the position corresponding to the grid belongs to the ground and a second height category confidence that the position belongs to the material pile according to the height information of the position corresponding to the grid and the height variance value used to characterize the confidence of the height information, it is specifically used for:

[0174] According to a preset ground height threshold and the height information of the position corresponding to each grid, candidate grids with height information less than or equal to the preset ground height threshold are extracted from each grid included in the elevation map;

[0175] According to the height information of each of the candidate grids, candidate grids with abnormal height information are filtered out to obtain target grids;

[0176] Plane fitting is performed on each of the target grids as ground grids to determine the plane equation of the ground, and the variance of the ground fitting is calculated according to the height information of each of the target grids;

[0177] For each grid in the elevation map of the material shoveling operation site, a first height category confidence that the position corresponding to the grid belongs to the ground and a second height category confidence that the position belongs to the material pile are calculated according to the height information of the position corresponding to the grid, the height variance value used to characterize the confidence of the height information, and the plane equation and the variance of the ground fitting.

[0178] In an alternative embodiment, when the processor is used to input the image including the material shoveling operation site into a semantic segmentation model, and determine a third semantic category confidence that the position corresponding to each grid in the elevation map belongs to the ground and a fourth semantic category confidence that the position belongs to the material pile based on the first semantic category confidence that each pixel point in the image output by the semantic segmentation model belongs to the ground and the second semantic category confidence that the pixel point belongs to the material pile, it is specifically used for:

[0179] An image including the material shoveling operation site is collected by an image acquisition device;

[0180] The image is input into the semantic segmentation model, and the first semantic category confidence that each pixel point in the image belongs to the ground and the second semantic category confidence that the pixel point belongs to the material pile are determined by the semantic segmentation model;

[0181] According to the external parameters of the image acquisition device, project each pixel point in the image onto the elevation map to establish the correspondence between each pixel point in the image and each grid in the elevation map; wherein, the pixel point and the grid with a correspondence relationship correspond to the same position in the material shoveling operation site;

[0182] For each grid in the elevation map, if the grid corresponds to a pixel point, then use the confidence of the first semantic category that the pixel point belongs to the ground as the confidence of the third semantic category that the position corresponding to the grid belongs to the ground, and use the confidence of the second semantic category that the pixel point belongs to the material pile as the confidence of the fourth semantic category that the position corresponding to the grid belongs to the material pile;

[0183] If the grid corresponds to multiple pixel points, then determine the confidence of the third semantic category that the position corresponding to the grid belongs to the ground according to the confidence of the first semantic category that the multiple pixel points belong to the ground, and determine the confidence of the fourth semantic category that the position corresponding to the grid belongs to the material pile according to the confidence of the second semantic category that the multiple pixel points belong to the material pile.

[0184] In an alternative embodiment, after the processor is used to input the image containing the material shoveling operation site into the semantic segmentation model, it is further used for:

[0185] Based on the confidence of the fifth semantic category that each pixel point in the image belongs to an obstacle output by the semantic segmentation model, determine the confidence of the sixth semantic category that the position corresponding to each grid in the elevation map belongs to an obstacle;

[0186] When the processor is used to determine that the category to which the position corresponding to each grid belongs is the ground or the material pile according to the confidence of the first height category that the position corresponding to the grid belongs to the ground and the confidence of the third semantic category, and the confidence of the second height category that the position corresponding to the grid belongs to the material pile and the confidence of the fourth semantic category, it specifically uses:

[0187] For each grid, judge whether the category to which the position corresponding to the grid belongs is an obstacle according to the confidence of the sixth semantic category that the position corresponding to the grid belongs to an obstacle;

[0188] If the category to which the position corresponding to the grid belongs is not an obstacle, then determine that the category to which the position corresponding to the grid belongs is the ground or the material pile according to the confidence of the first height category that the position corresponding to the grid belongs to the ground and the confidence of the third semantic category, and the confidence of the second height category that the position corresponding to the grid belongs to the material pile and the confidence of the fourth semantic category.

[0189] In an alternative embodiment, when the processor is used to determine that the category to which the position corresponding to the grid belongs is the ground or the material pile according to the confidence of the first height category and the confidence of the third semantic category of the position corresponding to the grid belonging to the ground, and the confidence of the second height category and the confidence of the fourth semantic category of the position corresponding to the grid belonging to the material pile, it is specifically used for:

[0190] Calculate the sum of the confidence of the first height category and the confidence of the third semantic category of the position corresponding to the grid belonging to the ground, to obtain the first total confidence of the position corresponding to the grid belonging to the ground, and calculate the sum of the confidence of the second height category and the confidence of the fourth semantic category of the position corresponding to the grid belonging to the material pile, to obtain the second total confidence of the position corresponding to the grid belonging to the material pile;

[0191] When the first total confidence is greater than or equal to the second total confidence, determine that the category to which the position corresponding to the grid belongs is the ground;

[0192] When the first total confidence is less than the second total confidence, determine that the category to which the position corresponding to the grid belongs is the material pile.

[0193] In an alternative embodiment, when the processor is used to determine whether the category to which the position corresponding to each grid belongs is an obstacle according to the confidence of the sixth semantic category of the position corresponding to the grid belonging to the obstacle, it is specifically used for:

[0194] For each grid, when the confidence of the sixth semantic category of the position corresponding to the grid belonging to the obstacle is greater than the confidence of the third semantic category of the position corresponding to the grid belonging to the ground and greater than the confidence of the fourth semantic category of the position corresponding to the grid belonging to the material pile, determine that the category to which the position corresponding to the grid belongs is an obstacle;

[0195] When the confidence of the sixth semantic category of the position corresponding to the grid belonging to the obstacle is less than or equal to the confidence of the third semantic category of the position corresponding to the grid belonging to the ground, and / or less than or equal to the confidence of the fourth semantic category of the position corresponding to the grid belonging to the material pile, then determine that the category to which the position corresponding to the grid belongs is not an obstacle.

[0196] In an alternative embodiment, after the processor determines the category to which the position corresponding to each grid belongs, it is further used for:

[0197] According to the category to which the position corresponding to each grid belongs, determine the material pile area, the ground area and the obstacle area in the material shoveling operation site;

[0198] Control the unmanned loader to avoid the obstacle area, drive through the ground area to a preset distance from the material pile area;

[0199] Control the bucket of the unmanned loader to push into the material pile in the material pile area and scoop up the material in the material pile area, so that the unmanned loader performs the operation of scooping and loading materials.

[0200] Through the computer-readable storage medium provided by the embodiments of the present application, by fusing height information and semantic information, comprehensively considering the height category confidence and semantic category confidence, it is possible to effectively identify low material piles (i.e., thin material piles). At the same time, it can also resist the interference of environmental light on the semantic segmentation accuracy and improve the accuracy of the material surface recognition result.

[0201] In the embodiments of the present application, when the computer-readable storage medium is run by a processor, it can also execute other machine-readable instructions to execute the material surface recognition method described in other parts of the embodiments. For the specific steps and principles of the executed material surface recognition method, refer to the description of the method-side embodiments and will not be elaborated here.

[0202] In the embodiments provided by the present application, it should be understood that the disclosed method, device, electronic device, and computer-readable storage medium can be implemented in other ways. The device embodiments described above are only illustrative. For example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation. For another example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection between each other can be through some communication interfaces. The indirect coupling or communication connection of the system or unit can be in an electrical, mechanical, or other form.

[0203] The unit described as a separate component may or may not be physically separated. The component displayed as a unit may or may not be a physical unit, that is, it may be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0204] In addition, in each embodiment of the present application, the functional units can be integrated in a processing unit, or each unit exists physically alone, or two or more units can be integrated in one unit.

[0205] When the above-mentioned functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium executable by a processor. Based on such understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of this application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROM), random access memories (RAM), magnetic disks, or optical discs.

[0206] Finally, it should be noted that the above-mentioned embodiments are only specific implementation manners of this application, used to illustrate the technical solution of this application, rather than limiting it. The protection scope of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: any person skilled in the art within the technical scope disclosed in this application can still modify the technical solutions recorded in the foregoing embodiments, or can easily think of changes, or perform equivalent replacements on some of the technical features; and these modifications, changes, or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be covered by the protection scope of this application. Therefore, the protection scope of this application should be subject to the protection scope of the claims.

Claims

1. A material surface recognition method, characterized in that: The method comprises: For each grid in the elevation map including the shoveling operation site, according to the height information of the position corresponding to the grid and the height variance value used to characterize the confidence of the height information, calculate the first height category confidence that the position corresponding to the grid belongs to the ground and the second height category confidence that the position belongs to the material pile; wherein each grid in the elevation map corresponds one-to-one to each position in the shoveling operation site; An image including the shoveling operation site is input into a semantic segmentation model, and based on the first semantic category confidence that each pixel in the image belongs to the ground and the second semantic category confidence that each pixel belongs to the material pile output by the semantic segmentation model, a third semantic category confidence that each position corresponding to each grid in the elevation map belongs to the ground and a fourth semantic category confidence that each position belongs to the material pile are determined, wherein each pixel in the image corresponds one-to-one to each position in the shoveling operation site; For each of the grids, determine whether the category to which the position corresponding to the grid belongs is the ground or the material pile based on the first height category confidence and the third semantic category confidence that the position corresponding to the grid belongs to the ground, and the second height category confidence and the fourth semantic category confidence that the position belongs to the material pile.

2. The method according to claim 1, characterized in that: The method further comprises: Collecting laser point cloud data of the shoveling operation site at each collection time; When the laser point cloud data is the laser point cloud data collected at the first collection moment, an elevation map of the shoveling operation site corresponding to the laser point cloud data is established based on the laser point cloud data collected at the first collection moment; When the laser point cloud data is not the laser point cloud data collected at the first collection moment, the laser point cloud data collected at the non-first collection moment is used to dynamically update the elevation map of the shoveling work site corresponding to the laser point cloud data collected at the previous collection moment, so as to obtain the elevation map of the shoveling work site corresponding to the laser point cloud data.

3. The method according to claim 1, characterized in that: The method of calculating, for each grid in the elevation map including the shoveling operation site, the first height category confidence that the position corresponding to the grid belongs to the ground and the second height category confidence that the position corresponding to the grid belongs to the material pile according to the height information of the position corresponding to the grid and the height variance value used to characterize the confidence of the height information, includes: According to a preset ground height threshold and the height information of the position corresponding to each grid, candidate grids having height information less than or equal to the preset ground height threshold are extracted from the grids included in the elevation map; According to the height information of each candidate grid, the candidate grids with abnormal height information are filtered out to obtain the target grid; Performing plane fitting on each of the target grids as a ground grid, determining a plane equation of the ground, and calculating a variance of the ground fitting based on height information of each of the target grids; For each grid in the elevation map that includes the shoveling operation site, the first height category confidence that the position corresponding to the grid belongs to the ground and the second height category confidence that the position corresponding to the grid belongs to the material pile are calculated based on the height information of the position corresponding to the grid, the height variance value used to characterize the confidence of the height information, and the variance of the plane equation and the ground fitting.

4. The method according to claim 1, characterized in that: The step of inputting the image including the shoveling operation site into the semantic segmentation model, and determining the third semantic category confidence that the position corresponding to each grid in the elevation map belongs to the ground and the fourth semantic category confidence that the position belongs to the material pile based on the first semantic category confidence that each pixel in the image output by the semantic segmentation model belongs to the ground and the second semantic category confidence that the position belongs to the material pile, includes: Capturing an image containing the shoveling operation site by an image acquisition device; Inputting the image into a semantic segmentation model, and determining, by the semantic segmentation model, a first semantic category confidence that each pixel in the image belongs to the ground, and a second semantic category confidence that each pixel in the image belongs to a material pile; Projecting each pixel point in the image onto the elevation map according to the device external parameters of the image acquisition device to establish a corresponding relationship between each pixel point in the image and each grid in the elevation map; wherein the pixel points and grids having a corresponding relationship correspond to the same position in the shoveling operation site; For each grid in the elevation map, if the grid corresponds to a pixel point, the first semantic category confidence that the pixel point belongs to the ground is used as the third semantic category confidence that the position corresponding to the grid belongs to the ground, and the second semantic category confidence that the pixel point belongs to the material pile is used as the fourth semantic category confidence that the position corresponding to the grid belongs to the material pile; If the grid corresponds to multiple pixels, the third semantic category confidence that the position corresponding to the grid belongs to the ground is determined based on the first semantic category confidence that the multiple pixels belong to the ground, and the fourth semantic category confidence that the position corresponding to the grid belongs to the material pile is determined based on the second semantic category confidence that the multiple pixels belong to the material pile.

5. The method according to claim 1, characterized in that: After inputting the image containing the shoveling operation site into the semantic segmentation model, the method further includes: Based on the confidence level of the fifth semantic category that each pixel in the image output by the semantic segmentation model belongs to an obstacle, determining the confidence level of the sixth semantic category that the position corresponding to each grid in the elevation map belongs to an obstacle; The determining, for each of the grids, that the category to which the position corresponding to the grid belongs is the ground or the material pile according to the first height category confidence and the third semantic category confidence that the position corresponding to the grid belongs to the ground, and the second height category confidence and the fourth semantic category confidence that the position belongs to the material pile, includes: For each of the grids, judging whether the category to which the position corresponding to the grid belongs is an obstacle according to the confidence that the position corresponding to the grid belongs to the sixth semantic category of an obstacle; If the category to which the position corresponding to the grid belongs is not an obstacle, then based on the first height category confidence and the third semantic category confidence that the position corresponding to the grid belongs to the ground, and the second height category confidence and the fourth semantic category confidence that the position corresponding to the grid belongs to the material pile, it is determined that the category to which the position corresponding to the grid belongs is the ground or the material pile.

6. The method according to claim 1 or 5, characterized in that: The determining that the category to which the position corresponding to the grid belongs is the ground or the material pile according to the first height category confidence and the third semantic category confidence that the position corresponding to the grid belongs to the ground, and the second height category confidence and the fourth semantic category confidence that the position corresponding to the grid belongs to the material pile, includes: Calculate the sum of the first height category confidence and the third semantic category confidence that the position corresponding to the grid belongs to the ground, and obtain a first total confidence that the position corresponding to the grid belongs to the ground, and calculate the sum of the second height category confidence and the fourth semantic category confidence that the position corresponding to the grid belongs to the material pile, and obtain a second total confidence that the position corresponding to the grid belongs to the material pile; When the first total confidence is greater than or equal to the second total confidence, determining that the category to which the position corresponding to the grid belongs is ground; When the first total confidence is less than the second total confidence, it is determined that the category to which the position corresponding to the grid belongs is a material pile.

7. The method according to claim 5, characterized in that: The step of determining, for each of the grids, whether the category to which the position corresponding to the grid belongs is an obstacle according to the sixth semantic category confidence that the position corresponding to the grid belongs to an obstacle includes: For each of the grids, when the sixth semantic category confidence that the position corresponding to the grid belongs to an obstacle is greater than the third semantic category confidence that the position corresponding to the grid belongs to the ground, and greater than the fourth semantic category confidence that the position corresponding to the grid belongs to a material pile, the category to which the position corresponding to the grid belongs is determined to be an obstacle; When the confidence level of the sixth semantic category that the position corresponding to the grid belongs to an obstacle is less than or equal to the confidence level of the third semantic category that the position corresponding to the grid belongs to the ground, and / or is less than or equal to the confidence level of the fourth semantic category that the position corresponding to the grid belongs to a pile of materials, it is determined that the category to which the position corresponding to the grid belongs is not an obstacle.

8. The method according to claim 7, characterized in that: After determining the category to which the position corresponding to each grid belongs, the method further includes: Determine the material pile area, ground area and obstacle area in the shoveling operation site according to the categories to which the positions corresponding to the grids belong; Controlling the unmanned loader to avoid the obstacle area and drive through the ground area to a preset distance from the material pile area; The bucket of the unmanned loader is controlled to be pushed into the material pile in the material pile area, and the material in the material pile area is shoveled up, so that the unmanned loader performs a shoveling material operation.

9. A material surface identification device, characterized in that: include: A calculation module is used to calculate, for each grid in the elevation map including the shoveling operation site, a first height category confidence that the position corresponding to the grid belongs to the ground, and a second height category confidence that the position belongs to the material pile, according to the height information of the position corresponding to the grid and the height variance value used to characterize the confidence of the height information; wherein each grid in the elevation map corresponds to each position in the shoveling operation site one by one; A first determination module is used to input an image containing the shoveling operation site into a semantic segmentation model, and determine a third semantic category confidence that a position corresponding to each grid in the elevation map belongs to the ground, and a fourth semantic category confidence that a position corresponding to each grid in the elevation map belongs to the material pile, based on a first semantic category confidence that each pixel in the image belongs to the ground, and a second semantic category confidence that each pixel belongs to the material pile, output by the semantic segmentation model, wherein each pixel in the image corresponds one-to-one to each position in the shoveling operation site; The second determination module is used to determine, for each of the grids, whether the category to which the position corresponding to the grid belongs is the ground or the material pile based on the first height category confidence and the third semantic category confidence that the position corresponding to the grid belongs to the ground, and the second height category confidence and the fourth semantic category confidence that the position belongs to the material pile.

10. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the method as claimed in any one of claims 1 to 8 are performed.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are executed.

Images