Multi-modal sensing construction site stockpiling management method, equipment and medium

Abstract

The embodiment of the invention discloses a multi-modal sensing construction site stockpile management method and device and a medium, and relates to the technical field of stockpile management, and the method comprises the steps: collecting multi-modal stockpile data corresponding to site stockpiles of a construction site; according to the multi-modal stockpiling data, on-site stockpiling is segmented, a plurality of stockpiling individuals corresponding to the on-site stockpiling are determined, and individual multi-modal stockpiling data corresponding to each stockpiling individual are obtained; identifying the stacking individuals through a pre-constructed multi-modal stacking identification model and the individual multi-modal stacking data so as to determine the stacking type corresponding to each stacking individual; and on-site stockpiling is managed based on the material type corresponding to each stockpiling individual and pre-acquired spatial distribution data corresponding to the construction site. Accurate stockpiling individual segmentation and type identification are realized through multi-modal stockpiling data, stockpiling is managed based on material types and spatial distribution, limitation of material type identification in a material mixing scene is overcome, and management hidden dangers are eliminated.

Description

Technical Field

[0001] The present specification relates to the technical field of stockpile management, and in particular to a multi-modal sensing construction site stockpile management method, equipment and medium. Background Art

[0002] In the process of modern construction, the management of material piles at the construction site is of vital importance. With the continuous expansion of the scale of construction projects and the increasing complexity of construction processes, construction sites are often piled with a wide variety of materials in large quantities. How to efficiently and accurately manage these piles has become a key link in construction management. The traditional method of managing material piles at construction sites mainly relies on manual experience and simple visual inspections. On the one hand, manual identification of material pile types is inefficient and prone to misjudgment, especially when faced with mixed piles of multiple similar materials, it is difficult to accurately distinguish them; on the other hand, there is a lack of scientific basis for the spatial layout and stacking location management of the piles, which often leads to problems such as inconvenient material access and unreasonable space utilization, which in turn affects construction progress and cost control.

[0003] With the development of sensor technology and computer vision technology, some stockpile management methods based on single sensor data have emerged. For example, visible light images are used for stockpile identification and management. However, this method is greatly affected by lighting conditions, and the recognition accuracy drops significantly in complex lighting environments. For another example, although the spatial information of stockpile can be obtained by relying solely on three-dimensional point cloud data, it is difficult to effectively distinguish some materials with similar materials and shapes. In addition, when identifying stockpile in the field of machine learning, the collected images are usually processed and directly input into the model. However, at the construction site, not every type of material exists separately. There may be mixed or stacked materials. In this case, the identification of source data containing multiple materials increases the difficulty of model identification, which in turn affects the efficiency of type identification.

[0004] Therefore, with regard to the scenario where materials may be mixed at the construction site, the traditional method has limitations in the results of material type identification, which in turn leads to hidden dangers in the management of on-site material stacking. Summary of the invention

[0005] One or more embodiments of the present specification provide a multimodal perception construction site material stockpile management method, equipment and medium for solving the following technical problems: For scenarios where materials may be mixed at a construction site, traditional methods have limitations in material type identification results, which in turn leads to hidden dangers in the management of on-site material stockpiles.

[0006] One or more embodiments of this specification adopt the following technical solutions:

[0007] One or more embodiments of the present specification provide a multimodal perception method for managing material piles at a construction site, the method comprising: collecting multimodal material pile data corresponding to material piles at the construction site, wherein the multimodal material pile data comprises a visible light image sequence, infrared sensor data and material pile three-dimensional point cloud data; segmenting the material pile according to the multimodal material pile data, determining a plurality of material pile individuals corresponding to the material pile, so as to obtain individual multimodal material pile data corresponding to each of the material pile individuals; identifying the material pile individuals through a pre-constructed multimodal material pile recognition model and the individual multimodal material pile data, so as to determine the material pile type corresponding to each of the material pile individuals; and managing the material pile on site based on the material type corresponding to each of the material pile individuals and the pre-acquired spatial distribution data corresponding to the construction site.

[0008] One or more embodiments of this specification provide a multi-modal sensing construction site stockpile management device, including:

[0009] at least one processor; and,

[0010] a memory communicatively connected to the at least one processor; wherein,

[0011] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the above method.

[0012] One or more embodiments of the present specification provide a non-volatile computer storage medium storing computer executable instructions, wherein the computer executable instructions are configured to execute the above method.

[0013] At least one of the above technical solutions adopted in the embodiments of this specification can achieve the following beneficial effects: through the technical solutions of the embodiments of this specification, visible light image sequences, infrared sensor data and three-dimensional point cloud data of piles are collected, and multimodal data can describe the piles from different dimensions. Compared with a single data source, multimodal data fusion collection can reduce data missing and errors, comprehensively and accurately obtain pile information, and avoid the problem of incomplete information caused by environmental interference of a certain type of data (such as visible light affected by illumination, point cloud data affected by occlusion); the on-site piles are segmented according to the multimodal pile data, and the advantages of different modal data are comprehensively utilized to improve the accuracy of segmentation. The multimodal collaborative segmentation method can accurately distinguish between piles of irregular shapes that are stacked on each other, determine multiple piles of individuals and obtain their corresponding individual multimodal pile data, so that the analysis of each pile of individuals is more detailed and in-depth; the pre-constructed multimodal pile recognition model and individual multimodal pile data are used to identify the pile individuals, which is different from the method based only on a single image or point cloud data. Compared with the existing recognition models, this model can better cope with the complex and changeable material stacking situation at the construction site and reduce misjudgment caused by similar appearance and similar materials. In addition, when materials are mixed and stacked, the features in the image are extremely complex and interfere with each other. After the material individuals are segmented, the complex mixed features can be separated into simple features of a single material, so that the model only needs to focus on identifying the features of a single material, reducing confusion and interference between features, thereby more accurately extracting and matching features and improving the recognition accuracy of each material. Based on the material type corresponding to the individual material stacking and the spatial distribution data of the construction site, the on-site material stacking can be managed to achieve reasonable allocation of resources and efficient use of space. By accurately identifying the type of material stacking and reasonably planning the stacking location, the safety risk can be effectively reduced. Reasonable material stacking management makes it more convenient to use materials, and construction personnel can quickly obtain the required materials, reduce waiting time, and improve construction efficiency. At the same time, the optimized material stacking layout avoids construction process interruption and material damage caused by chaotic material stacking, which helps to ensure construction quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the technical solutions in the embodiments of this specification or the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art description. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor. In the drawings:

[0015] Figure 1 A schematic diagram of a flow chart of a multi-modal sensing construction site stockpile management method provided in an embodiment of this specification;

[0016] Figure 2A schematic diagram of the structure of a multi-modal sensing construction site stockpile management device provided in an embodiment of this specification. DETAILED DESCRIPTION

[0017] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this specification.

[0018] The embodiments of this specification provide a multi-modal perception method for managing material piles at a construction site. It should be noted that the execution subject in the embodiments of this specification can be a server or any device with data processing capabilities. Figure 1 A schematic diagram of a multi-modal sensing construction site stockpile management method provided in an embodiment of this specification, such as Figure 1 As shown, it mainly includes the following steps:

[0019] Step S101 , collecting multi-modal material stockpiling data corresponding to on-site material stockpiling at the construction site.

[0020] The multimodal stockpile data includes visible light image sequences, infrared sensor data and stockpile three-dimensional point cloud data;

[0021] In one embodiment of the present specification, multimodal stockpile data corresponding to on-site stockpiles at a construction site is collected, and the multimodal stockpile data includes a visible light image sequence, infrared sensor data, and stockpile three-dimensional point cloud data. The visible light image sequence can be collected by a visible light camera, the infrared sensor data can be collected by an infrared sensor, and the stockpile three-dimensional point cloud data can be collected by a laser radar device.

[0022] Visible light cameras are installed at key locations on the construction site, such as material stacking areas and main passages. Such high-resolution color cameras are waterproof and dustproof, adaptable to harsh outdoor environments, and can be wall-mounted or column-mounted fixtures. At the same time, small unmanned aerial vehicles equipped with high-definition cameras can automatically cruise on preset paths, regularly capture large-area overhead views, and support optical zoom to obtain detailed images at different heights. In this way, visible light cameras can continuously take high-quality color photos at a high frame rate, providing visual information about the color, texture, and shape of the material, which is crucial for distinguishing different types of building materials, such as quickly identifying concrete blocks, bricks, or wood by color differences. In addition, combined with time series analysis, visible light cameras can also capture changes in materials over time, such as the stacking of newly arrived materials and consumption during use.

[0023] Infrared sensors are deployed on the same platform as the visible light camera or in an independent support structure nearby to ensure that the fields of view of the two overlap and facilitate synchronous data collection. Some infrared sensors are also integrated into drone platforms to conduct aerial patrols along with visible light cameras to expand the monitoring range. Infrared sensors generate detailed temperature distribution maps by detecting infrared radiation energy emitted from the surface of objects. For materials that look similar but have different thermal conductivities, such as metals and non-metals, the temperature difference provided by infrared sensors helps to distinguish them more accurately. In addition, infrared sensors can still work effectively at night or in low light conditions, supplementing the functional limitations of visible light cameras.

[0024] LiDAR equipment can be placed on a stable foundation, such as a concrete pier or other solid structure, to ensure that it maintains accuracy during long-term operation. LiDAR determines the distance of an object by emitting laser pulses and measuring the reflection time, thereby creating a detailed three-dimensional point cloud model of the stockpile area. This model not only contains the geometric dimensions of the materials, but also reflects their spatial layout relationship, providing a basis for subsequent volume calculations and quantity statistics. For those materials with irregular shapes or densely stacked materials, such as steel bars and gravel, their contours and internal structures can be accurately depicted, and high resolution and accuracy can be maintained even in densely stacked conditions.

[0025] Step S102, segmenting the on-site stockpile according to the multimodal stockpile data, determining a plurality of stockpile individuals corresponding to the on-site stockpile, and obtaining individual multimodal stockpile data corresponding to each stockpile individual.

[0026] The on-site material pile is segmented according to the multimodal material pile data to determine a plurality of material pile individuals corresponding to the on-site material pile, specifically comprising: segmenting the on-site material pile according to the visible light image and the three-dimensional point cloud data of the material pile in the multimodal material pile data to determine the image segmentation data and the three-dimensional point cloud segmentation data; and using the infrared sensor data, verifying the consistency of the image segmentation data and the three-dimensional point cloud segmentation data to determine a plurality of material pile individuals.

[0027] In one embodiment of the present specification, when segmenting the on-site pile, relying only on visible light images or three-dimensional point cloud data may be inaccurate, such as visible light images may be affected by factors such as lighting conditions and occlusion, and three-dimensional point cloud data may have problems such as uneven point cloud density and noise. The construction site environment is complex, and single sensor data is often difficult to fully and accurately describe the pile situation. Segmentation is performed based on visible light images and three-dimensional point cloud data respectively, and image segmentation data and three-dimensional point cloud segmentation data are determined. The image segmentation data and the three-dimensional point cloud segmentation data are verified for consistency using the infrared sensor data to determine multiple pile individuals. Multimodal data such as visible light images, pile three-dimensional point cloud data, and infrared sensor data are integrated to avoid the limitations of a single data source, and the advantages of different data are fully utilized to achieve more accurate segmentation and identification of pile individuals. Through the comprehensive processing and verification of multiple data, multiple pile individuals can be determined more accurately, providing more accurate basic data for subsequent pile management.

[0028] According to the visible light image and the three-dimensional point cloud data of the multimodal stockpile data, the on-site stockpile is segmented to determine the image segmentation data and the three-dimensional point cloud segmentation data, which specifically includes: performing color space conversion on the visible light image to determine the HSV color space image corresponding to the on-site stockpile, so as to obtain multiple color channel information corresponding to each pixel; performing threshold processing according to the multiple color channel information corresponding to each pixel, segmenting the image in the HSV color space image through a preset threshold range, and determining multiple initial stockpile segmentation areas corresponding to the on-site stockpile; performing edge detection on the HSV color space image by using an edge detection algorithm, extracting the stockpile contour in the HSV color space image based on the detected edges, so as to obtain the geometric features of the stockpile contour; determining the image segmentation data through the geometric features of the stockpile contour and the multiple initial stockpile segmentation areas, wherein the image segmentation data includes the image segmentation results corresponding to multiple stockpile individuals; performing point cloud segmentation on the three-dimensional point cloud data of the stockpile to determine the three-dimensional point cloud segmentation data, wherein the three-dimensional point cloud segmentation data includes the point cloud clustering results corresponding to multiple stockpile individuals.

[0029] In one embodiment of the present specification, the acquired visible light image is converted into a color space from the common RGB (red, green, blue) color space to the HSV (hue, saturation, value) color space. The HSV color space is more in line with the way of perceiving colors, decomposing color information into three channels of hue (Hue), saturation (Saturation) and value (Value), and can more intuitively distinguish different objects by color features. Through this conversion, multiple color channel information corresponding to each pixel in the HSV color space can be obtained, providing a basis for subsequent processing. According to the color channel information of each pixel in the HSV color space, threshold processing is performed. Since different types of piles have different color distribution ranges in the HSV color space, a suitable threshold range is first pre-set to classify the pixels in the HSV color space image. For example, the color of a certain type of pile has a specific value range in hue, saturation and value. When the HSV value of a pixel falls within this preset range, it is classified as an area related to the pile. In this way, the entire image is segmented to determine multiple initial stockpile segmentation areas corresponding to the on-site stockpile. The initial area is preliminarily divided based on color features and includes a set of pixels that partially belong to the same stockpile, but the boundary may not be precise enough.

[0030] It should be noted that when setting the threshold range, different types of piles have unique colors. For example, the color of common sand piles is usually light yellow, cement piles are generally gray, and for building bricks, red is common. The color of the piles will change due to many factors. Under strong direct sunlight, the color of the pile surface will become brighter and the contrast will decrease; while in the shadow area, the color will become darker. For example, when the sun shines from one side in the morning, the light yellow color of the sand pile on the illuminated side will be lighter and brighter, and the backlit side will be darker. Therefore, when setting the threshold, it is necessary to consider the range of these color changes to ensure that the same type of piles under different lighting conditions and material differences can be accurately identified. When setting the threshold range, take sand as an example. For sand, which is relatively single in color and greatly affected by light, the H channel threshold can be set according to its hue (H) range. For example, the hue of sand may be between 30°-60° (the H value range in the HSV model is usually 0°-360°). The saturation (S) and value (V) channels can be used to eliminate the influence of too strong or too dark light. If the saturation S is set between 20%-80% and the lightness V is set between 30%-90%, the sand color can be accurately identified even under different lighting conditions as long as it is within the HSV range. The HSV model represents the hue, saturation, and lightness of the color separately. Light changes mainly affect the lightness channel, while hue and saturation are relatively stable, which is more conducive to setting thresholds based on color characteristics.

[0031] Edge detection is performed on the HSV color space image using an edge detection algorithm (such as the Canny edge detection algorithm). The edge detection algorithm can identify areas in the image where the pixel values ​​change dramatically, that is, the edges of the objects. Based on the detected edges, the pile outline in the HSV color space image is further extracted. In the process of extracting the outline, the geometric features of the pile outline can also be obtained, such as the perimeter, area, shape factor (used to describe the compactness of the shape), the size of the circumscribed rectangle, etc. These geometric features are very important for further accurately describing and distinguishing different pile individuals. The geometric features of the pile outline and the multiple initial pile segmentation regions obtained previously are combined to determine the final image segmentation data. Geometric features can help screen, merge or adjust the initial segmentation regions so that the segmentation results are more consistent with the actual pile individual shape and boundaries. For example, if the outlines of two initial segmentation regions have similar geometric features and are adjacent in space, they may be merged into one pile individual. The final image segmentation data contains the image segmentation results corresponding to multiple pile individuals, which provides information based on two-dimensional images for subsequent fusion with other modal data and determination of pile individuals.

[0032] It should be noted that before segmenting the point cloud data, preprocessing operations are required to remove noise, improve data quality, and ultimately achieve effective segmentation of individual materials in the stockpile area. In order to improve the quality of the point cloud and reduce unnecessary computational burden, the voxel grid filtering algorithm is first applied to divide the entire point cloud into uniform small cubes (voxels), and then the points in each voxel are simplified according to the preset rules (retaining the point closest to the center or the average value in the voxel). Not only can isolated noise points be effectively removed, but the overall structural features can also be kept unchanged, thereby improving the speed and accuracy of subsequent processing. Then, the moving least squares (MLS) method is used to smooth the filtered point cloud. MLS is a local fitting method that approximates the point distribution around a given point by constructing a polynomial surface to achieve a smoothing effect. Specifically, a certain number of neighboring points are selected near each point, and the new fitting position is calculated based on the position and weight of these points. The advantage of this is that the point cloud surface can be made more continuous and flat while retaining important geometric features such as edges and corners.

[0033] In one embodiment of the present specification, point cloud segmentation is performed on the three-dimensional point cloud data of the stockpile, the purpose of which is to divide the point cloud data in the three-dimensional space into different subsets according to the spatial position relationship and geometric features, etc., each subset corresponds to a possible stockpile individual. The method of point cloud segmentation can be based on the method of region growing and the method based on cluster analysis. The method based on region growing is as follows: starting from one or more seed points, according to the set similarity criteria (such as the distance between points, the consistency of normal vectors, etc.), the adjacent points that meet the criteria are gradually merged into the same region, thereby forming different point cloud clusters, each cluster representing a stockpile individual. The method based on cluster analysis, such as DBSCAN (density-based spatial clustering algorithm), divides the density-connected points into the same cluster according to the density distribution of the point cloud, and can find clusters of any shape, which is suitable for processing irregularly shaped stockpile point cloud data. In the above manner, three-dimensional point cloud segmentation data is obtained, and the three-dimensional point cloud segmentation data includes point cloud clustering results corresponding to multiple stockpile individuals.

[0034] The infrared sensing data is used to verify the consistency of the image segmentation data and the three-dimensional point cloud segmentation data to determine multiple stockpiling individuals, specifically including: mapping each image segmentation result in the image segmentation data to the point cloud clustering result in the three-dimensional point cloud segmentation data to determine the spatial correspondence between the image segmentation result and the point cloud clustering result, wherein the spatial correspondence includes a spatially consistent correspondence and a spatially inconsistent correspondence; according to the spatial correspondence, multiple analysis areas corresponding to the infrared sensing data are determined; through the infrared sensing data, a temperature reference index between two adjacent analysis areas is determined to determine at least one temperature boundary, and the image segmentation data and the three-dimensional point cloud segmentation data are verified through the temperature boundary to determine multiple stockpiling individuals.

[0035] In one embodiment of the present specification, each image segmentation result in the image segmentation data obtained based on the visible light image is associated with the point cloud clustering result obtained based on the three-dimensional point cloud data. Since the visible light image provides two-dimensional visual information of the pile, and the three-dimensional point cloud data reflects the three-dimensional spatial position information of the pile, each pile area segmented in the image is found to have a corresponding point cloud cluster in the three-dimensional space based on the coordinate transformation method. For example, the geometric center position of the pile in the image is used, combined with the spatial conversion relationship between the image and the point cloud data, to find the point cloud cluster matching its position in the three-dimensional point cloud. After the mapping is completed, the spatial correspondence between the image segmentation result and the point cloud clustering result is determined. If the point cloud cluster corresponding to a certain pile area segmented in the image in the three-dimensional point cloud has a high consistency in terms of spatial position, shape, etc., that is, it can be well matched, indicating that the image segmentation result and the point cloud clustering result accurately correspond to the same pile individual, which is a spatially consistent correspondence relationship. In general, they are all spatially consistent correspondence relationships. For example, a rectangular stockpile area shown in the image has a corresponding point cloud cluster in the 3D point cloud that also presents a similar rectangular shape and has the same position.

[0036] On the contrary, if there are large differences between the image segmentation result and the point cloud clustering result in terms of spatial position, shape, etc., such as position deviation or shape deviation greater than the preset threshold, and they cannot correspond to the same individual pile, then the corresponding spatially inconsistent correspondence relationship is a spatially inconsistent correspondence relationship. For example, a circular pile area in the image corresponds to a long strip point cloud cluster in the three-dimensional point cloud, and the position is obviously mismatched. If a spatially inconsistent correspondence relationship occurs, it means that there is an acquisition error in this area collected by a certain acquisition source. The integrity of the image data and point cloud data of this area can be verified by the adjacent segmentation results of the image segmentation result with the spatially inconsistent correspondence relationship and the adjacent clustering results of the point cloud clustering result, and it can be determined whether there are factors such as occlusion that affect the comprehensiveness of data collection. For example, the gap area between the image segmentation result with the spatially inconsistent correspondence relationship and the adjacent segmentation result is calculated. If a large gap area appears, that is, the gap area is greater than the preset gap threshold, it may indicate that the image data of the area is missing, for example, there is occlusion during the image acquisition process, resulting in some areas not being captured. Similarly, the gap area between the point cloud segmentation result with inconsistent spatial correspondence and the adjacent segmentation result is calculated. If the gap area is larger than the preset gap threshold, it may indicate that the point cloud data in this area is missing. In this case, the image segmentation result or point cloud clustering result with a gap area not larger than the gap threshold is used as the final segmentation result.

[0037] According to the determined spatial correspondence, multiple analysis areas corresponding to the infrared sensor data are determined, where the analysis areas include image segmentation results and point cloud clustering results under consistent spatial correspondence, and also include final segmentation results determined under inconsistent spatial correspondence. Within the spatial range covered by the infrared sensor data, the corresponding area is defined as the analysis area. Based on the infrared sensor data, a temperature reference index between two adjacent analysis areas is determined to determine at least one temperature boundary.

[0038] The image segmentation data and the 3D point cloud segmentation data are verified by the temperature boundary to determine multiple piles. The matching between the pile boundary and the temperature boundary in the image segmentation result is determined. Ideally, the boundary of different piles in the image should roughly coincide with the temperature boundary. For example, if the boundary between two adjacent piles in the image segmentation data is close to the temperature boundary position determined by the infrared sensor data, this indicates that the image segmentation result in this area is relatively accurate. If it is found that there is a large deviation between the boundary of the image segmentation result and the temperature boundary, such as the temperature boundary shows that this should be the boundary between two different piles, but the image segmentation regards it as a whole, it means that there is a large deviation in the image segmentation result. In the 3D point cloud segmentation data, the correspondence between the boundary of the point cloud clustering result and the temperature boundary in spatial position is determined. Since the 3D point cloud data describes the distribution of the pile in the 3D space, the temperature boundary should be spatially consistent with the boundary between the point cloud clusters. For example, by projecting the temperature boundary into the 3D point cloud space, check whether the boundary of the point cloud cluster is consistent with the projected temperature boundary. According to the deviation between the boundary of the image segmentation result and the temperature boundary, and the deviation between the temperature boundary and the boundary of the point cloud clustering, the individual stockpiles are determined with the segmentation result having the smallest deviation.

[0039] After the individual stockpiles are determined, the individual multimodal stockpiles data corresponding to each individual stockpiles is obtained, where the individual multimodal stockpiles data includes the HSV image data and / or point cloud data corresponding to the individual stockpiles. If the spatial correspondence is spatial consistency, the individual stockpiles include both HSV image data and point cloud data; if the spatial correspondence is spatial inconsistency, it is determined according to the final segmentation result that is screened out, which may be HSV image data or point cloud data.

[0040] Through the above technical scheme, the image segmentation data and the three-dimensional point cloud segmentation data are combined, and the infrared sensor data is used for consistency verification, which can integrate the information obtained by different types of sensors, make up for the shortcomings of a single data source in identifying individual piles, and reduce the recognition errors caused by factors such as illumination and occlusion, so as to more accurately determine the boundaries and shapes of individual piles; there may be temperature differences between different individual piles, and the temperature boundary can more clearly outline the contours of the individual piles. The temperature boundary is determined by infrared sensor data, which is convenient for distinguishing adjacent or similar piles and improving the accuracy of recognition; the spatial correspondence between the image segmentation results and the point cloud clustering results is determined, and the spatial inconsistent correspondence can be found, so as to timely detect possible acquisition errors or data anomalies; multiple analysis areas corresponding to the infrared sensor data are determined according to the spatial correspondence, and temperature analysis can be performed for each analysis area; the image segmentation data and the three-dimensional point cloud segmentation data are verified by the temperature boundary, and a data verification mechanism is established, which can timely discover and correct errors or inconsistencies in the data, avoid the influence of erroneous data on subsequent decisions and operations, and thus improve the reliability and stability of the entire system.

[0041] The infrared sensor data is used to determine a temperature reference index between two adjacent analysis areas to determine at least one temperature boundary, specifically including: dividing each analysis area into sub-areas to determine a plurality of sub-areas corresponding to each analysis area; calculating an average temperature value within each sub-area based on the infrared sensor data to determine a temperature difference between adjacent sub-areas; determining a temperature reference index between adjacent sub-areas by a ratio of the temperature difference between the adjacent sub-areas to the distance between the adjacent sub-areas; and if the temperature reference index is greater than a preset index threshold, determining that the adjacent sub-areas are temperature boundaries.

[0042] In one embodiment of the present specification, in order to analyze the temperature changes in the infrared sensor data in more detail, sub-region division is further performed in each analysis area determined by the spatial correspondence between the image segmentation result and the three-dimensional point cloud segmentation result. Since the temperature difference between different piles of materials may only be obvious at a smaller spatial scale, these subtle temperature changes can be captured by dividing the sub-regions. For example, for a larger analysis area, if the sub-region division is not performed, the temperature difference at different internal locations may be masked due to the calculation of the overall average temperature. The size of the divided sub-region can be determined according to the actual situation and accuracy requirements. For example, it can be a regular square or cube area, and the side length is set according to the accuracy of the infrared sensor and the approximate size of the pile, which may generally be between a few centimeters and tens of centimeters.

[0043] According to the infrared sensor data, the temperature data in each sub-area is statistically analyzed, and the average temperature value in each sub-area is calculated. The average temperature can eliminate the local fluctuation of the temperature in the sub-area and obtain a value that can represent the temperature characteristics of the sub-area. If there are multiple temperature sampling points in a sub-area, the temperature values ​​of these points are added and divided by the number of sampling points to obtain the average temperature of the sub-area. After obtaining the average temperature value of each sub-area, the temperature difference between adjacent sub-areas is calculated. The temperature difference between adjacent sub-areas can reflect the change of temperature in space. By comparing the temperature difference between different adjacent sub-areas, it can be preliminarily determined which areas may have significant temperature changes. The ratio obtained by dividing the temperature difference between adjacent sub-areas by the distance between the sub-areas is the temperature reference index, which comprehensively considers the amplitude of temperature change and the spatial distance, and more accurately reflects the temperature change gradient in space. For example, if the temperature difference between adjacent sub-areas is 5°C and the distance between them is 10 cm, then the temperature reference index is 0.5°C / cm. Compared with simple temperature difference values, temperature reference indicators can more effectively reflect the temperature change characteristics between different piles due to differences in material, thermal properties, etc., because even if the temperature difference value is the same, if the distance is different, the degree of temperature change will be different. The temperature reference indicator quantifies such differences.

[0044] Based on the analysis of a large amount of actual data and the prior knowledge of the temperature difference between different pile materials, a preset indicator threshold is set. When the calculated temperature reference indicator between adjacent sub-areas is greater than this preset indicator threshold, it is determined that there is a temperature boundary between the two adjacent sub-areas. Different pile materials have different thermal properties such as thermal conductivity and specific heat capacity due to different materials. Under the same environmental conditions, there will be differences in the temperature distribution on their surfaces or around them. This difference is manifested as a larger value in the temperature reference indicator. Therefore, by comparing with the preset indicator threshold, the temperature boundary that may correspond to the junction of different pile materials can be identified. For example, after a large number of experiments and data analysis, the preset indicator threshold is determined to be 0.3℃ / cm. When the temperature reference indicator of a pair of adjacent sub-areas is 0.5℃ / cm, it is determined that there is a temperature boundary between the two sub-areas.

[0045] Through the above technical solution, by dividing the analysis area into sub-areas, the originally larger analysis area is refined into multiple smaller sub-areas, so that the analysis of temperature data is more detailed, and more subtle temperature changes in the analysis area can be captured, avoiding ignoring local temperature differences due to overall averaging, thereby more accurately locating the possible location of the temperature boundary; when calculating the temperature reference index, the temperature difference and distance between adjacent sub-areas are combined. Compared with relying solely on temperature difference to judge the temperature boundary, it can more comprehensively reflect the temperature change in space; the temperature difference between different piles is often more obvious at their junctions, and the temperature boundary can be used to identify the temperature difference between different piles. Separating them provides more accurate information for determining the boundaries of individual pile materials. In actual scenarios, there may be some pile materials that look similar but belong to different individuals. It may be difficult to accurately distinguish them only through image segmentation or point cloud segmentation data. By using temperature boundaries as auxiliary information, they can be distinguished based on the temperature differences between individual pile materials, thereby improving the ability to recognize similar individual pile materials. By calculating the average temperature, temperature difference value and temperature reference index, etc., the spatial distribution information of temperature data is fully utilized, making the determination of temperature boundaries more scientific and reliable. The processing method based on the characteristics of the data itself can reduce the influence of external interference factors to a certain extent and improve the stability of data processing results.

[0046] Step S103, identifying individual piles of materials using a pre-built multi-modal pile identification model and individual multi-modal pile data, so as to determine the type of material pile corresponding to each individual pile of materials.

[0047] The individual stockpile is identified by using a pre-constructed multimodal stockpile identification model and the individual multimodal stockpile data to determine the stockpile type corresponding to each of the individual stockpile, specifically including: collecting image data and three-dimensional point cloud data corresponding to multiple stockpile types, randomly changing the image data and the three-dimensional point cloud data to generate multiple extended image data and extended point cloud data, and constructing a material identification data set according to the image data, the extended image data, the three-dimensional point cloud data and the extended point cloud data; training a pre-constructed convolutional neural network model by using the material identification data set to determine the multimodal stockpile identification model; pre-processing the individual multimodal stockpile data to input it into the multimodal stockpile identification model to determine the predicted material classification result and the confidence score corresponding to the predicted material classification result; screening in the predicted material classification result by using the confidence score and a preset confidence threshold to determine the stockpile type corresponding to each of the individual stockpile.

[0048] In one embodiment of the present specification, when selecting a deep learning model, attention should be paid to its generalization ability and good adaptability to large-scale data to ensure that the model can maintain high accuracy and stability in practical applications. A convolutional neural network (CNN) is selected as the basic architecture, and the network structure of ResNet-50 is specifically adopted. ResNet-50 is famous for its introduction of the residual connection mechanism, which effectively solves the gradient vanishing problem in deep networks, allowing the network to mine features at a deeper level while maintaining high computational efficiency. In addition, ResNet-50 has demonstrated its excellent performance in multiple image classification tasks, especially in object recognition in complex backgrounds. By using a pre-trained ResNet-50 model as a starting point, the advantages of transfer learning can be used to quickly adjust the model parameters to adapt to the specific material recognition needs of the construction site, thereby greatly shortening the development cycle and improving model performance.

[0049] During the dataset preparation phase, a large amount of high-quality images and point cloud data are collected, and meticulous data preprocessing is performed to ensure that the model can learn the most effective features from these data. The specific steps are as follows: For each material category (such as steel bars, concrete blocks, bricks, wood, etc.), at least 10,000 high-definition color images and their corresponding three-dimensional point cloud data are collected. Remove invalid or duplicate data to ensure the purity and reliability of the dataset. Check and correct annotation errors to ensure the accuracy of each sample label. Apply a variety of image transformation techniques, such as rotation, scaling, flipping, color jittering, etc. to expand the dataset and increase the robustness of the model under different conditions. The formula is as follows:

[0050] For the original image I, a new image I′ is generated by random transformation T: I′=T(I);

[0051] The complete dataset is divided into training set, validation set and test set in a ratio of 8:1:1. The category distribution in each subset is ensured to be consistent to avoid bias affecting model performance. All images are normalized so that the pixel value range is in [0,1] or [-1,1] to accelerate model convergence.

[0052] The formula is as follows: For the input image X, its standardized form X norm for:

[0053]

[0054] where μ and σ are the mean and standard deviation of the image, respectively.

[0055] During the training process, detailed parameter configurations were set and a series of optimization strategies were used to improve the performance of the model. The following are the specific training settings and the formulas involved:

[0056] The batch size B is set to 32, that is, 32 samples are processed in each iteration. The maximum number of iterations is set to E = 200 epochs, but the actual training will be terminated early when the loss function converges.

[0057] The initial learning rate η0 = 0.001, and the cosine annealing strategy is used to dynamically adjust the learning rate to promote smooth convergence. Learning rate update formula:

[0058] etat=eta0×(1+cos(π×t / T)) / 2;

[0059] Where t is the current iteration number and T is the total iteration number.

[0060] The Adam optimizer is selected, which combines the advantages of momentum method and adaptive learning rate, and can find the optimal solution path in a complex loss landscape. The Adam update rule is as follows: for weight w, gradient g, time step t, first-order moment estimate m, and second-order moment estimate v, we have

[0061]

[0062] Where β1 and β2 are decay rates, which default to 0.9 and 0.999 respectively; ∈ is a very small constant used to prevent the denominator from being zero.

[0063] The cross entropy loss function is used to measure the difference between the predicted result and the true label. For the binary classification problem, the cross entropy loss formula is:

[0064]

[0065] Where y is the true label and y^ is the probability value predicted by the model. For multi-classification problems, it can be expanded to:

[0066]

[0067] Where C is the total number of categories, y and y^ are the true label vector and predicted probability distribution respectively.

[0068] During the training process, the model performance is regularly evaluated on the validation set to monitor changes in indicators such as accuracy, recall, and F1 score. Hyperparameters such as learning rate and regularization strength are dynamically adjusted based on the performance on the validation set to prevent overfitting and optimize the final performance. When the loss on the validation set no longer decreases significantly over multiple consecutive epochs, the early stopping mechanism is triggered to end the training to save computing resources and avoid overfitting.

[0069] In addition, when choosing edge computing devices, you can choose the NVIDIA Jetson AGX Xavier development board as the core component. Jetson AGX Xavier is a powerful computing platform designed for AI applications, equipped with high-performance GPU, CPU and dedicated AI acceleration hardware, which can support complex deep learning models for efficient reasoning tasks. The platform has excellent energy efficiency and can significantly reduce power consumption without sacrificing performance. It is suitable for deployment in energy-sensitive environments such as construction sites. In addition, Jetson AGX Xavier also provides a wealth of interface options, including high-speed network connections, USB ports and GPIO pins, which are convenient for integration with various sensors and external devices, ensuring the flexibility and scalability of the system.

[0070] In one embodiment of the present specification, individual multimodal stockpiling data is preprocessed, and for image data, standardization operations are first performed, such as resizing to a fixed size, normalizing the pixel value range, etc., to ensure that the input format meets the requirements of CNN. For the three-dimensional point cloud data generated by the lidar, a filtering algorithm is implemented to remove noise points, and the point cloud surface is made more continuous and flat through smoothing operations. Input into the multimodal stockpiling recognition model, through multi-level feature extraction and nonlinear transformation of the input data, the predicted material classification results and the confidence scores corresponding to the predicted material classification results are finally output. The above process is completed on the powerful AI acceleration hardware of Jetson AGX Xavier, ensuring extremely high reasoning speed and accuracy. According to the set confidence threshold (0.8), valid predictions above the threshold are screened out, which are considered to be reliable classification results. For each valid prediction, the specific quantity and position coordinates of each material are further determined in combination with the point cloud data obtained in the previous preprocessing stage. This step involves spatial geometric calculations and statistical analysis to ensure that the final output information is both accurate and detailed.

[0071] Through the above technical solution, when materials are mixed and stacked, the features in the image are extremely complex and interfere with each other. After the individual materials are segmented, the complex mixed features can be separated into simple features of a single material, so that the model only needs to focus on identifying the features of a single material, reducing confusion and interference between features, thereby more accurately extracting and matching features and improving the recognition accuracy of each material; in the segmented individual material images, the key features of the material are more prominently displayed, and the model can learn and capture these key features more intensively without being obscured by the features of other irrelevant materials; when directly processing source data containing multiple materials, the model needs to process a large amount of redundant and irrelevant information, and the amount of calculation is huge. After the individual materials are segmented, the amount of data input to the model is greatly reduced, and the model only needs to process the image data of a single material individual, and the amount of calculation is significantly reduced, thereby speeding up the training and reasoning speed of the model and improving the recognition efficiency; the segmented individual material images are more targeted, and can find the feature patterns related to the material more quickly, avoiding blind searches in a large amount of irrelevant data, thereby shortening the feature extraction time and further improving the recognition efficiency. The placement of materials at construction sites is complex and changeable. By segmenting individual materials and then training the model, the model can be exposed to more individual images of single materials at different angles and in different states, thereby learning richer material feature representations. This allows the model to more accurately identify individual materials in various material mixing and stacking scenarios, enhancing the generalization ability of the model and improving the stability and reliability of the model in practical applications.

[0072] Step S104: managing the on-site material piles based on the material type corresponding to each material pile and the pre-acquired spatial distribution data corresponding to the construction site.

[0073] Based on the material type corresponding to each individual pile of materials and the pre-acquired spatial distribution data corresponding to the construction site, the on-site pile of materials is managed, specifically including: obtaining the current stacking parameters corresponding to each individual pile of materials, wherein the current stacking parameters include current stacking position information and current stacking size information; based on the current stacking position information, performing a pile point insertion operation in the spatial distribution data to determine the stacking standard index corresponding to the individual pile of materials according to the material type corresponding to each individual pile of materials and the current stacking size information, wherein the stacking standard index includes a safety distance index, a space occupancy index and an easy access index; and managing the stacking position of the on-site pile of materials through the stacking standard index.

[0074] In one embodiment of the present specification, the current stacking position information (such as the coordinates in the construction site coordinate system) and the current stacking size information (length, width, height or volume, etc.) of each individual stockpile are obtained, and the above parameters reflect the actual stacking state of the individual stockpile at the moment. And the spatial distribution data corresponding to the construction site is obtained. Since the construction process in the construction site is constantly changing, the spatial distribution data here is in a real-time updated state. The precise position coordinates of each individual stockpile currently obtained are combined with the spatial distribution data of the construction site to ensure that the two data are in the same spatial reference system. After the coordinate system is unified, the position information of each individual stockpile is corresponded to the spatial distribution data. If the spatial distribution data is a two-dimensional CAD drawing, the position of the individual stockpile is marked on the drawing according to the coordinates; if it is a three-dimensional model data, the stockpile position is accurately inserted into the corresponding spatial position of the model in the form of a point. Through the current stacking parameters and spatial distribution data, the stacking standard index corresponding to the individual stockpile is determined. Through the stacking standard index, the stacking position of the on-site stockpile is managed.

[0075] According to the material type corresponding to each individual stockpile and the current stacking size information, the stacking standard index corresponding to the individual stockpile is determined, specifically including: based on the material type of each individual stockpile, the standard safety distance, size restriction data and applicable construction process corresponding to the individual stockpile are determined; with the insertion point position of the individual stockpile in the spatial distribution data as the center and the standard safety distance as the radius, a search area is determined; within the search area, dangerous objects are searched through the spatial distribution data to determine the safety distance index of the individual stockpile; through the current stacking size information and the size restriction data of the individual stockpile, the space occupancy of the individual stockpile is evaluated to determine the space occupancy index of the individual stockpile; based on the spatial distribution data, the construction positions corresponding to multiple construction processes are determined, and according to the applicable construction process of the individual stockpile and the construction positions corresponding to the multiple construction processes, the nearest designated construction process is determined to obtain the access distance between the designated construction process and the individual stockpile; according to the access distance and a preset standard access reference threshold, the convenient access index of the individual stockpile is determined.

[0076] In one embodiment of the present specification, different material types have different stacking requirements due to their different physical and chemical properties and the way they are used in construction. Based on the material type of each individual pile, the corresponding standard safety distance (for example, flammable materials must be kept at a certain distance from the fire source), size restriction data (such as restrictions on floor space and height for certain large equipment piles) and applicable construction processes (clarify which specific construction links the material is used for). These data are pre-set based on material characteristics and construction experience, with the insertion point position of the individual pile in the spatial distribution data (i.e., its exact position in the construction site space) as the center and the previously determined standard safety distance as the radius, and a circular or spherical search area (two-dimensional or three-dimensional space) is determined in the spatial distribution data. This area covers the range that may have a safety impact on the individual pile.

[0077] In the designated search area, the search for dangerous objects is carried out with the help of spatial distribution data (such as maps containing location information of buildings, equipment, fire sources, etc.). If dangerous objects (such as fire sources, electrical equipment, etc.) are found in the area, the safety distance index of the individual pile is determined based on the actual distance between the dangerous objects and the individual pile. For example, if the standard safety distance is set to 10 meters, and the fire source is found 8 meters away from the individual pile in the search area, then the safety distance index reflects the difference between this distance and the standard, indicating that there may be safety risks. The current stacking size information of the individual pile is compared and analyzed with the predetermined size limit data. By comparing the space actually occupied by the pile (length, width, height, volume, etc.) with the specified size limit, it is evaluated whether the space occupancy of the individual pile meets the requirements, thereby determining the space occupancy index. If the size limit stipulates that the height of a certain type of pile shall not exceed 3 meters, and the actual height of the individual pile is measured to be 2.5 meters, the comparison can be used to obtain indicators such as the degree of compliance of space occupancy.

[0078] Based on the spatial distribution data, the specific construction locations of multiple construction processes at the construction site are determined. For example, the construction location corresponding to the concrete pouring construction process may be on different floors and areas of the building. According to the applicable construction process of the material stacking individual, among the construction locations corresponding to the multiple construction processes, the designated construction process closest to the material stacking individual is found. The access distance is obtained by calculating the distance between the designated construction process and the material stacking individual. The obtained access distance is compared with the preset standard access reference threshold. If the access distance is less than or equal to the standard access reference threshold, it means that the access is relatively convenient, and the access convenience index is determined accordingly to indicate that the access convenience is high; conversely, if the access distance is greater than the standard access reference threshold, it means that the access is not very convenient, and the access convenience index reflects this situation. For example, the standard access reference threshold is set to 50 meters. If the access distance between a material stacking individual and the nearest construction process is 30 meters, the access convenience index shows that the material stacking is easy to access. Through the safety distance index, space occupancy index and access convenience index determined by the above steps, a comprehensive assessment is made as to whether the current stacking position of each material stacking individual is reasonable. For individual piles that do not meet the index requirements, corresponding adjustment strategies are formulated according to the index deviation, such as adjusting the pile position to meet the safety distance requirements, re-planning the stacking method to optimize space occupancy, or moving to a location closer to the applicable construction process to improve access convenience, etc., so as to achieve scientific and reasonable management of the on-site pile stacking location.

[0079] Through the above technical solution, by determining the standard safety distance based on the material type, and using this to find dangerous objects in the spatial distribution data to determine the safety distance index, the potential safety risks between the pile and the dangerous objects can be effectively identified. For example, for flammable materials, ensure that they maintain a sufficient safety distance from dangerous objects such as fire sources and electrical equipment, thereby greatly reducing the possibility of safety accidents such as fire and explosion, and ensuring the safety of personnel and equipment on the construction site; clear safety distance indicators provide a quantitative basis for the safety management of the construction site, which is helpful for formulating unified and standardized safety standards and operating procedures; determining the size limit data based on the material type of the individual pile, and evaluating the space occupancy index in combination with the current stacking size information, is helpful for the reasonable planning of the stacking space on the construction site, avoiding space waste or excessive occupation due to unreasonable stacking size, and making more efficient use of limited construction sites; optimizing space occupancy can improve the stacking of the construction site without increasing the site area. Material holding capacity, through accurate management of the space occupied by the pile of materials, the site can accommodate more types and quantities of materials to meet the diverse needs of the construction process; according to the applicable construction process of the individual pile of materials, the nearest construction process location is determined, and then the access distance is obtained and compared with the standard access reference threshold to determine the access convenience index, which helps to place the pile of materials closer to the use point, so that construction personnel can quickly obtain the required materials, reduce material handling time and labor costs, and improve construction efficiency. The determination of the access convenience index helps to optimize the overall construction process, so that the material supply and construction operations are closely connected. The smooth supply of materials during the construction process can avoid construction stagnation caused by waiting for materials, and ensure the continuity and efficiency of construction. Detailed and accurate data support is provided for pile management through quantitative indicators (safety distance index, space occupancy index, access convenience index).

[0080] Through the technical solutions of the embodiments of this specification, visible light image sequences, infrared sensor data and three-dimensional point cloud data of pile materials are collected. Multimodal data can describe the pile materials from different dimensions. Compared with a single data source, multimodal data fusion collection can reduce data loss and errors, comprehensively and accurately obtain pile material information, and avoid the problem of incomplete information caused by environmental interference of a certain type of data (such as visible light affected by illumination, point cloud data affected by occlusion); the on-site pile materials are segmented according to the multimodal pile material data, and the advantages of different modal data are comprehensively utilized to improve the accuracy of segmentation. The multimodal collaborative segmentation method can accurately distinguish between pile materials that are stacked on each other and irregular in shape, determine multiple pile material individuals and obtain their corresponding individual multimodal pile material data, so that the analysis of each pile material individual is more detailed and in-depth; the pre-constructed multimodal pile material recognition model and individual multimodal pile material data are used to identify the pile material individual. Compared with the recognition model based only on a single image or point cloud data, the model can be more accurate. It can better cope with the complex and changeable material stacking situation at the construction site and reduce misjudgment caused by similar appearance and similar materials. In addition, when materials are mixed and stacked, the features in the image are extremely complex and interfere with each other. After the material individuals are segmented, the complex mixed features can be separated into simple features of a single material, so that the model only needs to focus on identifying the features of a single material, reducing confusion and interference between features, thereby more accurately extracting and matching features and improving the recognition accuracy of each material. Based on the material type corresponding to the individual stacking material and the spatial distribution data of the construction site, the on-site stacking can be managed to achieve reasonable allocation of resources and efficient use of space. By accurately identifying the type of stacking material and reasonably planning the stacking location, it can effectively reduce safety risks. Reasonable stacking management makes it more convenient to use materials, and construction personnel can quickly obtain the required materials, reduce waiting time, and improve construction efficiency. At the same time, the optimized stacking layout avoids construction process interruption and material damage caused by chaotic material stacking, which helps to ensure construction quality.

[0081] The present specification also provides a multi-modal sensing construction site stockpile management device, such as Figure 2 As shown, the device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above method.

[0082] The embodiments of the present specification also provide a non-volatile computer storage medium storing computer executable instructions, wherein the computer executable instructions are configured to execute the above method.

[0083] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, equipment, and non-volatile computer storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0084] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0085] The devices and media provided in the embodiments of this specification correspond one-to-one to the methods. Therefore, the devices and media also have similar beneficial technical effects as the corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0086] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0087] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0088] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0089] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0090] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0091] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0092] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0093] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0094] The above description is only one or more embodiments of this specification and is not intended to limit this specification. For those skilled in the art, one or more embodiments of this specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of one or more embodiments of this specification shall be included in the scope of the claims of this specification.

Claims

1. A multi-modal sensing construction site stockpile management method, characterized in that: The method comprises: Collecting multimodal stockpile data corresponding to on-site stockpiles at the construction site, wherein the multimodal stockpile data includes visible light image sequences, infrared sensor data, and stockpile three-dimensional point cloud data; According to the multimodal stockpile data, the on-site stockpile is segmented to determine a plurality of stockpile individuals corresponding to the on-site stockpile, so as to obtain individual multimodal stockpile data corresponding to each of the stockpile individuals; Identify the individual piles of materials by using a pre-built multi-modal pile identification model and the individual multi-modal pile data to determine the type of pile corresponding to each individual pile of materials; The on-site material piles are managed based on the material type corresponding to each of the material piles and the pre-acquired spatial distribution data corresponding to the construction site.

2. A multi-modal sensing construction site stockpile management method according to claim 1, characterized in that: Segmenting the on-site stockpile according to the multimodal stockpile data to determine a plurality of stockpile individuals corresponding to the on-site stockpile specifically includes: Segmenting the on-site stockpile according to the visible light image and the three-dimensional point cloud data of the stockpile in the multimodal stockpile data, and determining image segmentation data and three-dimensional point cloud segmentation data; The infrared sensor data is used to verify the consistency of the image segmentation data and the three-dimensional point cloud segmentation data to determine a plurality of individual stockpiles.

3. A multi-modal sensing construction site stockpile management method according to claim 2, characterized in that: The on-site stockpile is segmented according to the visible light image and the three-dimensional point cloud data of the stockpile in the multimodal stockpile data, and the image segmentation data and the three-dimensional point cloud segmentation data are determined, which specifically includes: Performing color space conversion on the visible light image to determine the HSV color space image corresponding to the on-site stockpile, so as to obtain multiple color channel information corresponding to each pixel point; Performing threshold processing according to multiple color channel information corresponding to each pixel point, segmenting the image in the HSV color space image through a preset threshold range, and determining multiple initial stockpile segmentation areas corresponding to the on-site stockpile; Performing edge detection on the HSV color space image using an edge detection algorithm, and extracting a stockpile outline in the HSV color space image based on the detected edges to obtain geometric features of the stockpile outline; Determining image segmentation data based on the geometric features of the stockpile outline and the multiple initial stockpile segmentation regions, wherein the image segmentation data includes image segmentation results corresponding to multiple stockpile individuals; Point cloud segmentation is performed on the three-dimensional point cloud data of the material pile to determine the three-dimensional point cloud segmentation data, wherein the three-dimensional point cloud segmentation data includes point cloud clustering results corresponding to multiple material pile individuals.

4. A multi-modal sensing construction site stockpile management method according to claim 3, characterized in that: Using the infrared sensor data, the image segmentation data and the three-dimensional point cloud segmentation data are verified for consistency to determine a plurality of individual stockpiles, specifically including: Mapping each of the image segmentation results in the image segmentation data to the point cloud clustering results in the three-dimensional point cloud segmentation data to determine a spatial correspondence between the image segmentation results and the point cloud clustering results, wherein the spatial correspondence includes a spatially consistent correspondence and a spatially inconsistent correspondence; Determining a plurality of analysis areas corresponding to the infrared sensing data according to the spatial correspondence; The temperature reference index between two adjacent analysis areas is determined through the infrared sensor data to determine at least one temperature boundary, and the image segmentation data and the three-dimensional point cloud segmentation data are verified through the temperature boundary to determine a plurality of stockpiling individuals.

5. A multi-modal sensing construction site stockpile management method according to claim 4, characterized in that: Determining a temperature reference index between two adjacent analysis areas through the infrared sensing data to determine at least one temperature boundary specifically includes: Divide each analysis area into sub-areas to determine a plurality of sub-areas corresponding to each analysis area; Calculating the average temperature value in each of the sub-areas according to the infrared sensor data to determine the temperature difference between adjacent sub-areas; Determining a temperature reference index between the adjacent sub-areas by a ratio of a temperature difference between the adjacent sub-areas and a distance between the adjacent sub-areas; If the temperature reference index is greater than a preset index threshold, it is determined that the adjacent sub-areas are temperature boundaries.

6. The multi-modal sensing construction site stockpile management method according to claim 1, characterized in that: The individual piles are identified by using a pre-built multi-modal pile identification model and the individual multi-modal pile data to determine the pile type corresponding to each individual pile, specifically including: Collecting image data and three-dimensional point cloud data corresponding to a plurality of stockpile types, randomly changing the image data and the three-dimensional point cloud data to generate a plurality of extended image data and extended point cloud data, and constructing a material identification data set according to the image data, the extended image data, the three-dimensional point cloud data and the extended point cloud data; The pre-built convolutional neural network model is trained using the material identification data set to determine the multi-modal stockpile identification model; Preprocessing the individual multimodal stockpile data to input into the multimodal stockpile identification model, and determining a predicted material classification result and a confidence score corresponding to the predicted material classification result; The predicted material classification results are screened by using the confidence score and a preset confidence threshold to determine the stockpile type corresponding to each of the stockpile individuals.

7. The multi-modal sensing construction site stockpile management method according to claim 1, characterized in that: Based on the material type corresponding to each of the individual stockpiles and the pre-acquired spatial distribution data corresponding to the construction site, the on-site stockpiles are managed, specifically including: Acquire current stacking parameters corresponding to each of the individual stacking materials, wherein the current stacking parameters include current stacking position information and current stacking size information; Based on the current stacking position information, a stacking point insertion operation is performed in the spatial distribution data to determine the stacking standard index corresponding to each individual stacking material according to the material type corresponding to each individual stacking material and the current stacking size information, wherein the stacking standard index includes a safety distance index, a space occupancy index and a convenient access index; The stacking position of the on-site stockpile is managed through the stacking standard index.

8. The multi-modal sensing construction site stockpile management method according to claim 7, characterized in that: According to the material type corresponding to each of the individual piles and the current stacking size information, the stacking standard index corresponding to the individual piles is determined, specifically including: Based on the material type of each individual stockpile, determine the standard safety distance, size limit data and applicable construction process corresponding to the individual stockpile; Determine a search area with the insertion point of the individual stockpile in the spatial distribution data as the center and the standard safety distance as the radius; In the search area, searching for dangerous objects by using the spatial distribution data, and determining the safety distance index of the individual stockpiles; Evaluate the space occupation of the individual pile of materials by using the current stacking size information and the size limit data of the individual pile of materials, and determine the space occupation index of the individual pile of materials; Based on the spatial distribution data, the construction positions corresponding to the multiple construction processes are determined, and the nearest designated construction process is determined according to the applicable construction process of the material pile and the construction positions corresponding to the multiple construction processes, so as to obtain the access distance between the designated construction process and the material pile; The access convenience index of the individual stockpiles is determined according to the access distance and a preset standard access reference threshold.

9. A multi-modal sensing construction site stockpile management device, characterized in that: The device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the method according to any one of claims 1 to 8.

10. A non-volatile computer storage medium storing computer executable instructions, characterized in that: The computer executable instructions are configured to execute the method according to any one of claims 1 to 8.

Images