Image processing-based stockyard aggregate inspection analysis method and system

Abstract

The present application belongs to the technical field of image processing, and specifically relates to a stockyard aggregate inspection analysis method and system based on image processing. The method comprises the following steps: target detection is performed on the aggregate accumulation area in the stockyard, the PTZ angle offset and target zoom factor of each target area are calculated, a cruise trajectory instruction set is generated and sorted and optimized, the PTZ is driven to the target position by an absolute PTZ movement instruction, the position change amount polling and continuous stability determination mechanism are adopted to ensure the stability of the PTZ; the obtained pictures are uploaded to the object storage and sequentially called for local and global index aggregation; the global aggregated data are collected, the snapshot URL is matched with the grouping index, and the four-layer index storage model is written into the database; and the task state flow and real-time progress pushing are realized through the front-end polling and Web Socket double channels. The panoramic trajectory automatic identification and dynamic planning, automatic cruise stable snapshot, multi-dimensional quality analysis and real-time feedback of detection results are realized, and a stockyard aggregate intelligent inspection closed-loop system covering the whole process is constructed.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and in particular relates to a method and system for inspecting and analyzing aggregates in silos based on image processing. Background Technology

[0002] With the continuous expansion of infrastructure construction, sand and gravel aggregates, as the core raw materials for concrete preparation, directly affect the safety and durability of engineering structures due to their quality stability. In the aggregate production and distribution process, silos, as key facilities for storage and transfer, require strict monitoring and control of quality indicators such as aggregate gradation, particle size distribution, and the content of needle-like and flaky particles, in accordance with rigorous engineering standards. Traditional silo aggregate quality testing mainly relies on a combination of manual sampling and laboratory analysis. Technicians periodically enter the silo area to collect samples, which are then sent to the laboratory for physical property testing through sieving, weighing, and other methods. The final determination of aggregate quality is then based on relevant specifications. However, this testing model has gradually revealed several structural defects in practical applications.

[0003] In terms of timeliness, manual inspections are limited by factors such as staff scheduling, working environment, and testing procedures, and can usually only achieve periodic sampling inspections, with intervals of several hours or even days between two inspections. During this period, the quality of aggregates in the silo may fluctuate due to factors such as continuous unloading, accumulation and settling, or mixing of different batches of materials. Technicians find it difficult to grasp the actual state of the aggregates in the silo in a timely manner. Once quality problems occur and flow into subsequent production stages, they will have an irreversible impact on the project quality.

[0004] From a coverage perspective, manual sampling inspection can only obtain samples from a limited number of local points. Due to the randomness of sampling, the results of a single inspection are difficult to accurately reflect the overall distribution of aggregates inside the silo. In particular, the quality of aggregates in areas such as the silo walls and bottom corners is often overlooked, leading to the risk of overgeneralization in the test results. In addition, the determination of aggregate morphology, especially the assessment of the content of needle-like and flaky particles, is highly dependent on the personal experience of technical personnel. Different personnel have subjective differences in their application of standards, and the lack of unified and objective quantitative evaluation standards brings uncertainty to quality control.

[0005] Furthermore, large silos typically involve complex working conditions such as working at heights, dust pollution, and the operation of mechanical equipment. Workers need to carry out inspection work in these harsh environments for extended periods of time, resulting in high labor intensity and significant safety risks, which in turn increases labor and safety management costs.

[0006] To compensate for the shortcomings of manual inspections, video surveillance systems have been widely used in silo management. Existing silo video surveillance systems primarily utilize fixed cameras or rotatable pan-tilt cameras deployed within the silo to enable remote, real-time monitoring of the silo's internal conditions. Technicians in the control room can visually understand the silo's macroscopic status, such as material level and unloading flow rate, through video feeds. However, these systems essentially only function as "eyes," merely acquiring and transmitting raw video signals, lacking the ability to intelligently understand and analyze image content. In other words, the system can "see" the aggregate accumulation within the silo, but it cannot "read" the quality status of the aggregate, automatically identify needle-like or flaky particles, assess particle size distribution, or determine if there are any quality issues.

[0007] Regarding image acquisition strategies, traditional solutions typically employ a preset position capture mode. This involves pre-setting several fixed shooting points in the system, with the pan-tilt unit rotating sequentially to each point to capture images according to preset instructions. While this approach can achieve multi-area coverage to some extent, its core limitation lies in the fact that the preset position settings rely on manual experience and lack intelligent sensing capabilities for the actual operating conditions of the silo.

[0008] When the aggregate accumulation pattern within the silo changes, such as the addition of new accumulation areas or the collapse or slippage of existing areas, the pre-set capture points may not accurately cover the new critical areas, leading to the omission of important quality information. Simultaneously, the number of preset positions is limited by system capacity and rotation efficiency, making it difficult to achieve dense sampling of the entire silo. The number of effective images acquired in a single inspection is limited, resulting in insufficient detection coverage and representativeness. Furthermore, due to the lack of an image quality stability assessment mechanism, capturing images before the mechanical structure is fully stable after the gimbal has rotated to its position may result in motion blur, focus shift, and other issues, affecting the accuracy of subsequent image analysis.

[0009] Therefore, existing technical solutions face multiple challenges in the field of aggregate quality inspection in silos, including poor inspection timeliness, insufficient coverage, strong subjectivity, and high labor intensity. Existing video monitoring systems can only achieve basic remote viewing functions, lacking automated quality analysis capabilities, and preset capture strategies cannot adapt to the dynamic changes in aggregate distribution within the silo. This invention aims to construct a closed-loop intelligent inspection system for aggregates in silos covering the entire process, achieving panoramic trajectory recognition and dynamic planning, automatic cruise and stable capture, multi-dimensional quality analysis, and real-time feedback of inspection results. Summary of the Invention

[0010] The purpose of this invention is to provide a method and system for inspecting and analyzing aggregates in silos based on image processing, so as to solve the technical problems existing in the prior art.

[0011] To solve the above-mentioned technical problems, the present invention adopts the following technical solution: Firstly, an image processing-based method for analyzing aggregate inspection in silos is provided, including the following steps: S1, panoramic image acquisition and trajectory recognition, the specific process includes: The system calls the PTZ control interface to send a command to move the camera to the initial preset position. ,in For horizontal angles, It is a vertical angle. This refers to the zoom level; after the gimbal stabilizes, a panoramic capture is performed to obtain the original image. ; Panoramic image submission and trajectory recognition, converting the original image and current PTZ parameters Submit to the trajectory recognition service via HTTPS interface; The trajectory recognition service uses a target detection model to infer the aggregate accumulation area within the silo and outputs N target detection bounding boxes; for each bounding box, the center pixel coordinates are calculated: ; in For the first The center pixel coordinates of the target detection bounding box The coordinates of the top-left corner of the bounding box are in pixels. The coordinates of the bottom right corner of the bounding box. The x and y coordinates of the center point in the image coordinate system; Based on the obtained bounding box center pixel coordinates, combined with camera intrinsic parameters, it is mapped to the gimbal angle offset: ; , ; in, These are the horizontal and vertical angular offsets from the center of the panoramic image to the center of the target bounding box, respectively. The center pixel coordinates of the panoramic image. The physical size of a single pixel. For camera focal length, This refers to the zoom level when shooting in a panoramic view. Calculate the target zoom level: ; in, The fill factor; Based on the above calculated data, a cruise trajectory command set is generated: ; in , ; Trajectory sorting optimization: Based on the PTZ coordinates of N points, calculate the gimbal movement cost between adjacent points, and sort them using the nearest neighbor algorithm or the TSP approximation algorithm to minimize the total movement path.

[0012] S2: Gimbal stability polling step, which includes: Send movement command: Retrieve the current point command from the trajectory command set T. Send an absolute PTZ movement command to the PTZ unit, with the target location being... ; Polling stability determination: Initialize the stability counter Maximum timeout The polling interval is 30 seconds. It is 500 milliseconds; the first During the next poll, query the current PTZ position of the PTZ: ; Calculate the change in position: ; The stability determination condition is for true If and only if and and ; otherwise It is false; in The horizontal stability threshold, The vertical stability threshold, The zoom direction stabilization threshold.

[0013] The continuous stability criterion is: ; in The threshold for the number of consecutive stable occurrences; like The gimbal was determined to be stable. If the loop exceeds It is forcibly classified as stable; Interruption and rollback: If a task cancellation command is received during the determination process, polling will be terminated immediately, and the PTZ will be rolled back to the initial preset position. .

[0014] S3: Point-by-point cruise capture and sequence analysis, specifically including: Point-by-point navigation: Execute each point sequentially according to the order of the trajectory instruction set T or the optimized order. For each location, the mechanism in step S2 is invoked to wait for the gimbal to stabilize. Once stabilized, high-definition snapshots are performed to acquire images. ; Image upload: Capture the image Upload the image to the object storage service and retrieve the returned image URL, denoted as . ; Perform sequential call analysis: Submit image URLs to the image analysis service in sequence; No. i When calling an image, parameters should be included: ; in It is true if and only if ,otherwise for false The image analysis service outputs single-point indicators, specifically the content of needle-like and flaky aggregates in the image of the i-th point: ; in Let represent the number of needle-like and flaky aggregates identified in the image at the i-th location. Let be the total number of aggregates identified in the image at the i-th location; Global aggregation trigger: When the analysis of the Nth image is completed and When true, the image analysis service triggers a global metric aggregation calculation, aggregating the grouped metrics of all N points to generate a global quality report.

[0015] S4, result aggregation and unified database entry, specifically includes: Collect aggregated data: When the analysis results for the last image are returned, extract global aggregated data: global metrics. Grouping Indicator List Particle size distribution Stone Details List ; Image and indicator matching: Match the URLs of the captured images of each location with the group indicators one by one according to the index, add the bounding box coordinates to the group indicators, and establish spatial association; Unified persistence: A database storage object is constructed and written to the database, while simultaneously associating it with the task ID; the data to be persisted includes: the panorama URL and PTZ parameters. List of URLs for analysis diagrams at each location Coordinates of the bounding box at each point Global content of flaky and needle-shaped aggregates Particle size distribution statistics: ; in For the first j A particle size range; A four-layer index storage model is adopted. ; in Simultaneously, the persistence exceeding the standard judgment result. .

[0016] S5: Task tracking and real-time notifications, specifically including: Task creation and state machine initialization: Generate a unique task ID, create a task state machine object in memory, and initialize it to PENDING; record task metadata: device ID, creation time, trajectory instruction set T, and total number of points N; State transition: Task state The state transition function is When a task starts, its status changes to RUNNING. The progress is updated after each point is completed. Once all points are completed, the status changes to COMPLETED. If an error occurs, the status changes to FAILED. If the task is canceled, the status changes to CANCELLED. Front-end polling: The front-end obtains real-time progress through the / progress / {task Id} interface. The interface returns the cumulative number of needle-like stones. Stone details can be queried as needed via the independent interface / progress / {task Id} / rocks / {tjId}; During the real-time push process via WebSocket, when the task status changes or a key node is completed, a message is broadcast to the clients subscribed to the task via WebSocket. After the analysis is completed, the final detection result is pushed.

[0017] S6, device-level mutual exclusion and exception rollback, specifically includes: Device mutual exclusion check: Before the task starts, call isDeviceRunning(device Id) to check if there is already an active task on the device; query the task status corresponding to device Id in the in-memory ConcurrentHashMap. If there is a task with a status of PENDING or RUNNING, refuse to create a new task and return an error code. Protection during task execution: The device is continuously occupied in the memory map during task execution, and is removed from the map to release the device after the task is completed; Abnormal rollback mechanism: If an abnormality occurs or a cancellation command is received in any step from S1 to S5, the abnormal object is captured, the error log is recorded, the task status is updated to FAILED or CANCELLED, and a command is sent to move the PTZ to the initial preset position. After the rollback is completed, the device mutex is released.

[0018] Furthermore, before the gimbal returns to its original position, a self-check procedure is first executed to detect the operating status of the horizontal, vertical, and zoom motors, ensuring that each motor is within its normal operating range before executing the return command. The target detection model adopts a deep convolutional neural network-based target detection architecture. During the training phase, this model uses an annotated dataset of aggregate images from a silo. The annotation information includes the coordinates of the aggregate bounding boxes and category labels. The model output includes the bounding box coordinates, confidence score, and category classification result for each detected aggregate region. The camera's intrinsic parameter matrix is ​​obtained through a pre-executed camera calibration process. The calibration method adopted is the Zhang Zhengyou calibration method, using a black and white checkerboard calibration board to capture calibration images at multiple different angles. Accurate camera intrinsic parameter data is obtained by calculating the focal length parameter, principal point coordinates, and distortion coefficients in the camera intrinsic parameter matrix.

[0019] Fill factor The value ranges from 0.3 to 0.8. The value of this coefficient is adjusted according to the actual volume of the silo and the density of the aggregate accumulation. A smaller fill factor is used in areas with denser accumulation to ensure that the complete aggregate area is captured, while a larger fill factor is used in areas with sparser accumulation to obtain a higher resolution aggregate image.

[0020] When trajectory sorting optimization uses the nearest neighbor algorithm, the specific implementation process is as follows: First, the initial preset position is... Starting from the current point, the system sequentially selects the nearest unvisited point as the next point to visit, repeating this process until all N points have been visited, ultimately yielding the optimized cruise route sequence.

[0021] Furthermore, the horizontal stability threshold The value is no greater than 5 degrees, and the vertical stability threshold is... The value is no greater than 3 degrees, and the zoom direction stability threshold is... The value of this threshold is no greater than 2. This threshold setting comprehensively considers the mechanical precision of the pan-tilt unit, image clarity requirements, and system response efficiency, aiming to maximize inspection efficiency while ensuring image quality. (Threshold for consecutive stable counts) The value is 3 times, meaning that the stability judgment condition must be met for 3 consecutive polls before the gimbal is considered to have reached a stable state. This design can effectively filter out the short-term jitter caused by the mechanical inertia of the gimbal and avoid performing the capture operation before the gimbal is truly stable.

[0022] The analysis model for image analysis services adopts a multi-task neural network architecture, which includes an aggregate detection branch, a shape classification branch, and a particle size regression branch. The aggregate detection branch outputs the bounding box coordinates and detection confidence of each aggregate region, the shape classification branch classifies each aggregate region into needle-like, sheet-like, cubic, or other shapes, and the particle size regression branch outputs the equivalent particle size value of each aggregate region.

[0023] The global index aggregation calculation process first summarizes the detection results of all N points, and then calculates the global index according to the preset weighting strategy. The weighting strategy takes into account the distribution density of each point in the silo space and the actual accumulation of aggregate in the area, ensuring that the global index can truly reflect the overall quality status of aggregate in the silo.

[0024] Furthermore, the global indicators in the four-layer indicator storage model Includes the following specific fields: Global content of flaky and needle-like aggregates Total aggregate quantity Total number of needle-shaped and flaky aggregates Task completion timestamp and task status identifier.

[0025] Grouping Indicator List Each element in It includes the following specific fields: point index Snapshot Bounding box coordinates Localized needle-like and flaky aggregate content Total amount of local aggregate Quantity of localized needle-shaped and flaky aggregates And the particle size distribution data at that location.

[0026] The particle size distribution statistics adopt an equal-interval division strategy, dividing the aggregate particle size into several equally spaced particle size intervals. Calculate the proportion of aggregate quantity within each particle size range to the total aggregate quantity, and generate particle size distribution histogram data. .

[0027] Furthermore, the implementation logic of the state transition function is as follows: when a task is created but has not yet started execution, the state is PENDING; when the task starts performing gimbal movement operations, the state changes from PENDING to RUNNING; when image acquisition and analysis of all patrol points are completed, the state changes from RUNNING to COMPLETED; when abnormal situations such as network timeout, image analysis service returning an error response, or gimbal unresponsiveness occur during task execution, the state changes from the current state to FAILED; when the user sends a cancellation command and the task is in the PENDING or RUNNING state, the state changes to CANCELLED.

[0028] The front-end polling interval is set to 2 seconds. This interval ensures that users can promptly perceive the task progress while avoiding excessively frequent polling requests that could put unnecessary load on the server. WebSocket push messages include the following types: task status change messages, location progress update messages, exception alarm messages, and task completion messages. Each message carries a task ID, timestamp, and corresponding load data. The client executes the appropriate UI update operation based on the message type.

[0029] Secondly, it provides an image processing-based aggregate inspection and analysis system for silos, including an inspection orchestrator, task manager, image analysis service, trajectory recognition service, index storage service, and Web Socket session service. The index storage service is implemented through a four-layer index storage model, including: global index, grouped index, stone details, and particle size distribution.

[0030] The beneficial effects of this invention include: This invention achieves dynamic cruise path planning based on the actual aggregate distribution in the silo through panoramic image acquisition and trajectory recognition, solving the problem of insufficient coverage in the traditional preset position capture mode. The trajectory recognition can automatically discover newly formed aggregate accumulation areas in the silo and include the corresponding points in the cruise trajectory, ensuring that the quality information of key areas is not missed.

[0031] This invention achieves precise quantitative determination of mechanical motion state through a gimbal stabilization polling mechanism, avoiding image blurring caused by capturing images before the gimbal is fully stabilized. At the same time, by setting reasonable stabilization thresholds and the number of consecutive judgments, the inspection efficiency is maximized while ensuring image quality.

[0032] This invention achieves an end-to-end automated processing flow from image acquisition to automatic analysis through point-by-point cruise capture and sequential analysis. The combined use of single-point analysis results and global aggregation results not only meets real-time requirements but also provides the ability to evaluate from a global quality perspective.

[0033] This invention achieves structured storage of multi-dimensional quality data through result aggregation and unified warehousing. The design of the four-layer indicator storage model can support independent management and related queries of global indicators, group indicators, stone details and particle size distribution, laying a data foundation for subsequent quality traceability and data analysis.

[0034] This invention achieves end-to-end status observability from task creation to completion through task tracking and real-time notification. The dual-channel design of front-end polling and Web Socket push not only meets the needs of regular progress query but also provides the ability to push key events in real time.

[0035] This invention achieves secure management of device resources in multi-task concurrent scenarios and automatic gimbal return in fault scenarios through device-level mutual exclusion and abnormal rollback, avoiding inspection conflicts caused by device contention and uncontrollable gimbal position after abnormal exit. Attached Figure Description

[0036] Figure 1 This is a schematic diagram of the architecture of the image processing-based silo aggregate inspection and analysis system and the four-layer index storage model of the present invention. Figure 2 This is a flowchart illustrating the image processing-based aggregate inspection and analysis method for silos according to the present invention. Figure 3 This is a schematic diagram of the panoramic image trajectory recognition and cruising path of the present invention; Figure 4 This is a timing diagram for the real-time notification of the present invention; Figure 5 This is a schematic diagram of the device mutual exclusion and abnormal rollback mechanism of the present invention. Detailed Implementation

[0037] The following is in conjunction with the appendix Figures 1-5 The present invention will be further described in detail below: The technical solution of this invention mainly targets the automated inspection scenario of aggregate silos. By integrating a PTZ camera, a target detection model, image analysis services, and data storage services, it achieves intelligent inspection of aggregate quality within the silo. The entire system consists of four parts: a hardware layer, an algorithm layer, a service layer, and an application layer. The hardware layer includes a PTZ camera, a camera, supplementary lighting, and a material level sensor; the algorithm layer includes a target detection model and an image analysis model; the service layer includes trajectory recognition services, image analysis services, object storage services, Web Socket services, and a database; and the application layer includes a client application and a data visualization module.

[0038] See Figure 2 In step S1, the system performs panoramic image acquisition and trajectory recognition. This process first moves the camera to the initial preset position by sending a command through the pan-tilt control interface. P o The preset position is determined by the horizontal angle. p o Vertical angle t o and zoom magnification z o Three parameters are uniquely determined. After the gimbal stabilizes, panoramic capture is performed to obtain the original image containing panoramic information of the silo. I panorama .

[0039] Before the gimbal returns to its normal position, a self-test procedure is executed to check the operating status of the horizontal, vertical, and zoom motors. The self-test procedure reads the operating parameters of each motor, including motor current, speed, and position feedback values, to determine if each motor is within its normal operating range. After the self-test is complete, the system records the status indicators of each motor. Only when all motor status indicators are normal can the return-to-position command be executed. The execution time of the self-test procedure is typically controlled within 5 seconds to ensure that it does not affect the overall inspection efficiency.

[0040] The system will display the original image. I panorama and current PTZ parameters P o Submissions are made to the trajectory recognition service via an HTTPS interface. The trajectory recognition service integrates a deep convolutional neural network-based object detection model. This model uses the YOLOv8 network architecture as its backbone, which is pre-trained on the COCO dataset and fine-tuned on a silo aggregate image dataset using transfer learning. The training dataset contains manually annotated silo aggregate images, with annotations including the coordinates of the aggregate bounding boxes and category labels. The model output includes the bounding box coordinates, confidence score, and category classification result for each detected aggregate region. The inference time of the object detection model is approximately 150 milliseconds on a server equipped with an NVIDIA T4 graphics card, which meets the requirements for real-time processing.

[0041] For each detected bounding box in the panoramic image, the system calculates its center pixel coordinates C. i The specific calculation method is as follows: , The coordinates of the center pixel are based on the top left corner of the image as the origin, with the horizontal axis being the u-axis and the vertical axis being the v-axis.

[0042] Based on the obtained bounding box center pixel coordinates, the system performs spatial mapping in conjunction with camera intrinsic parameters to calculate the gimbal angle offset. The camera intrinsic parameter matrix is ​​obtained through a pre-executed camera calibration process using the Zhang Zhengyou calibration method. Calibration images are captured at multiple different angles using a black and white checkerboard calibration board. The calibration board uses a 7×9 black and white checkerboard, with each square measuring 30 mm. During the calibration process, calibration images are captured from at least 20 different angles. Accurate camera intrinsic parameter data is obtained by solving the focal length parameter f, principal point coordinates (u0, v0), and distortion coefficients in the camera intrinsic parameter matrix. The focal length parameter f is typically the physical focal length of the camera lens, such as 12 mm or 25 mm, and the principal point coordinates (u0, v0) are typically close to half the image resolution.

[0043] The formula for calculating the horizontal angular offset is: The formula for calculating the vertical angle offset is: .in,s x and s y These are the physical dimensions of a single pixel in the horizontal and vertical directions, respectively. Their values ​​are determined by the pixel size of the image sensor, with typical values ​​of 3.45 micrometers or 2.4 micrometers. z o This is the zoom level used for panoramic shooting, typically set to the minimum value to obtain the widest field of view. The calculated PTZ coordinates of the target point are: , .

[0044] The formula for calculating the target zoom level is: ; Where W and H are the total width and total height of the image, respectively, and η is the fill factor. The fill factor ranges from 0.3 to 0.8, and its value is adjusted according to the actual volume of the silo and the density of the aggregate packing. A smaller fill factor is used in denser areas to ensure complete capture of the aggregate area, while a larger fill factor is used in sparser areas to obtain a higher resolution aggregate image. In a specific embodiment, for a circular silo with high packing density, the fill factor η is set to 0.4; for a square silo with low packing density, the fill factor η is set to 0.7.

[0045] Based on the above calculated data, the system generates a cruise trajectory instruction set T={(P i B i Let )|i=1,2,…,N}, where each element contains the target PTZ coordinates P. i and the corresponding bounding box coordinates B i The value of N in the cruise trajectory instruction set T depends on the number of valid detection boxes output by the target detection model. In typical application scenarios, N ranges from 8 to 30 points.

[0046] The system calculates the gimbal movement cost between adjacent points based on the PTZ coordinates of N points and uses a nearest neighbor algorithm for sorting to minimize the total movement path. The specific implementation of the nearest neighbor algorithm is as follows: first, an initial preset point P0 is used as the starting point; then, the unvisited point closest to the current point is selected as the next visited point, where the distance is defined as the Euclidean distance between two adjacent PTZ coordinates. This process is repeated until all N points have been visited, ultimately yielding the optimized cruise path sequence. In another embodiment, a TSP approximation algorithm such as the 2-opt algorithm can also be used for trajectory sorting optimization. This algorithm can further reduce the total movement path length, but the computation time is slightly increased.

[0047] See Figure 3It includes two parts: (a) a panoramic view of the silo and the AI ​​recognition area, showing the panoramic view taken by the gimbal at the initial preset position. The image coordinate system (u, v), the center pixel coordinates (u0, v0), and the bounding boxes B1, B2, and B3 of the aggregate accumulation area identified by the AI ​​object detection model are marked in the image; (b) the gimbal cruising path - showing the gimbal cruising path P1→P2→P3 optimized based on the nearest neighbor or TSP approximation algorithm. The absolute PTZ coordinates of each cruising point are marked (u, v). p i , t i , z i The image also shows the gimbal mounting location. It visually presents the pixel coordinates from the panoramic view (…). u i , v i (absolute PTZ coordinates of the gimbal) p i , t i , z i The mapping relationship reflects the zoom cruise strategy of "panoramic view → close-up".

[0048] In step S2, the system performs a gimbal stabilization polling process. The purpose of the stabilization polling mechanism is to ensure that the gimbal has reached a stable state after moving to the target position and completing zoom adjustment, thus avoiding image blurring caused by mechanical inertia.

[0049] The system retrieves the current point instruction T from the trajectory instruction set T. i =(P i B i Send an absolute PTZ movement command to the pan-tilt unit, with the target position being P. i =(p i ,t i ,z i The PTZ control interface uses the ONVIF protocol, and the interface parameters include the target PTZ coordinates, movement speed, and preset position number. The movement speed parameter is usually set to the medium speed setting to ensure a balance between the smoothness of the PTZ movement and the response speed.

[0050] In step S2.2, the system performs a polling stability determination process. The stability counter c is initialized to 0, the maximum timeout T_max is set to 30 seconds, and the polling interval Δt is set to 500 milliseconds. During the k-th poll, the system queries the current PTZ position of the pan-tilt unit. Calculate the change in position: ; The stability criteria are: ; Horizontal stability threshold θ p The value should not exceed 5 degrees, and in practice it can be set to 3 degrees; vertical stability threshold. θ t The value should not exceed 3 degrees, but can be set to 2 degrees in practice; zoom direction stabilization threshold. θ z The value of is no greater than 2, and in specific implementation, it is set to 1.5. The setting of this set of thresholds takes into account the mechanical precision of the pan-tilt unit, the requirements for image clarity, and the system response efficiency, so as to improve the inspection efficiency as much as possible while ensuring image quality.

[0051] The continuous stability criterion is: IsStable = (Σ k=m m+cstable- ¹ Stable (k) ) ≥ ; in C stable The threshold is the number of consecutive stable occurrences. C stable The value is taken three times, meaning that the stability judgment condition must be met for three consecutive polls before the gimbal is considered to have reached a stable state. This design can effectively filter out brief jitters caused by the mechanical inertia of the gimbal, preventing the capture operation from being performed before the gimbal is truly stable. If c≥ C stable If the cycle exceeds [a certain threshold], the gimbal is determined to be stable and the polling is stopped; if the cycle exceeds [a certain threshold], the polling is stopped. T max The system is forced to determine if the system is stable and exit polling to avoid indefinite waiting due to gimbal malfunction.

[0052] In another embodiment, the system can also incorporate a machine learning model to predict the motion state of the gimbal. This prediction model is trained based on historical polling data, and the model input is the current polling position change Δ. k Based on the previous polling data, the model outputs the predicted stabilization time. The introduction of the predictive model can reduce unnecessary polling waiting time while ensuring the accuracy of stabilization determination. When the predictive model determines that the gimbal is about to reach a stable state, it can perform the capture operation in advance, which can improve the overall inspection efficiency by about 15% to 20%.

[0053] If a task cancellation command is received during the decision-making process, the system immediately terminates polling and triggers the PTZ to revert to the initial preset position P0. The movement speed during the reversion process is usually set to a low speed to ensure smooth reversion.

[0054] In step S3, the system performs a point-by-point cruise capture and sequential analysis process.

[0055] The system executes each point T sequentially according to the order of the trajectory instruction set T or the optimized order. i For each location, the stabilization waiting mechanism in step S2 is invoked. After the gimbal stabilizes, high-definition snapshot is performed to acquire image I. i High-definition captured images are typically set to a resolution of 1920×1080 pixels or higher to ensure sufficient image quality for image analysis.

[0056] The system will capture image I i Upload the image to an object storage service and retrieve the returned image URL, which we will denote as the URL. i Image uploads use a RESTful API with HTTPS protocol. The upload request's Content-Type is set to multipart / form-data, and the returned response is a JSON object containing the image access URL. Object storage services can be S3-compatible, such as Alibaba Cloud OSS or Huawei Cloud OBS. The bandwidth requirements for image uploads are determined based on image resolution and inspection timeliness requirements, typically above 10Mbps.

[0057] The system submits image URLs to the image analysis service sequentially. The image analysis service employs a multi-task neural network architecture with three parallel branches: aggregate detection, shape classification, and particle size regression. The aggregate detection branch uses a CenterNet-based detection head, outputting the bounding box coordinates and detection confidence score for each aggregate region. The detection threshold is set to 0.5 to balance the detection rate and false positive rate. The shape classification branch classifies each aggregate region into needle-like, sheet-like, cubic, or other shapes. The classification model uses ResNet-50 as the feature extractor, outputting a four-class probability distribution. The particle size regression branch outputs the equivalent particle size value for each aggregate region based on the regression head, with the particle size value in millimeters. The mean absolute error of the regression model is controlled within 2 millimeters.

[0058] No. i When calling an image, include the parameter (imageUrl=URL). i ,task Id=unique task ID,finish i ), where finish i It is true if and only if i = N, otherwise finish i for false The image analysis service outputs single-location metrics, the first... i The content of flaky and needle-like aggregates in the images of each location ,in Let represent the number of needle-like and flaky aggregates identified in the image at the i-th location. For the firsti The total number of aggregates identified in each image point. The processing time for single-point analysis is approximately 200ms on a server equipped with an NVIDIA T4 graphics card.

[0059] In another embodiment, the system can also implement parallel image analysis. The system maintains an analysis request queue; once an image is uploaded, its URL is added to the queue. The background image analysis service retrieves tasks from the queue and performs parallel processing. The degree of parallelism can be dynamically adjusted based on the number of GPUs on the server, typically set to 4 to 8. This design significantly improves the overall analysis efficiency of multi-point inspections, reducing overall analysis time by approximately 40% to 60%.

[0060] When the Nth image is analyzed and finish i When true, the image analysis service triggers a global metric aggregation calculation. The aggregation calculation process first summarizes the detection results of all N points, and then calculates the global metric according to a preset weighting strategy. The weighting strategy considers the distribution density of each point in the silo space and the actual amount of aggregate accumulated in that area. The specific weight calculation formula is as follows: ; in Density i This is the estimated aggregate density at the i-th point. Volume i This represents the volume of the silo space corresponding to this location. This ensures that the overall indicators accurately reflect the overall quality status of the aggregate within the silo.

[0061] In step S4, the system performs the process of aggregating and uniformly storing the results.

[0062] When the image analysis service returns the analysis results for the last image, the system extracts global aggregated data: global metrics. M global Grouping Indicator List M group =[M1,M2,…,M N Particle size distribution D size Stone Details List Rdetail Global metrics M global Includes the following specific fields: Global content of flaky and needle-like aggregates Total aggregate quantity Total number of needle-shaped and flaky aggregates Task completion timestamp and task status identifier.

[0063] The system maps the captured image URLs of each location to grouped indicators by index, adds the bounding box coordinates to the grouped indicators, and establishes spatial relationships. (Grouped Indicator List) M group Each element in M i It includes the following specific fields: location index i, and captured image URL. i Boundary box coordinates B i Localized needle-like and flaky aggregate content Total amount of local aggregate Quantity of localized needle-shaped and flaky aggregates And the particle size distribution data at that location.

[0064] The system constructs and writes database storage objects to the database, while also associating them with task IDs. The data to be persisted includes: panoramic image URLs and PTZ parameters P0, a list of analysis image URLs for each point [URL1, URL2, ..., URL_N], bounding box coordinates for each point [B1, B2, ..., B_N], and global flaky and elongated aggregate content. Particle size distribution statistics ,in d j For the first j A range of particle sizes.

[0065] The particle size distribution statistics employ an equally spaced division strategy, dividing the aggregate particle size into several equally spaced intervals. In specific implementation, these intervals are divided into 10 categories: 0-5 mm, 5-10 mm, 10-20 mm, 20-30 mm, 30-40 mm, 40-50 mm, 50-60 mm, 60-80 mm, 80-100 mm, and over 100 mm. The system calculates the proportion of aggregate quantity within each particle size interval to the total aggregate quantity, generating a particle size distribution histogram. D size .

[0066] The system adopts a four-layer index storage model: Storage={ M global ,M group ,R detail ,D size},in M group ={(URL i B i M iThe database uses PostgreSQL (version 13 or later) and employs a master-slave replication architecture to ensure high availability. The database tables include the `task`, `point_data`, `global_metrics`, `particle_detail`, and `size_distribution` tables, which are linked together using the `task_id` and `point_index` fields.

[0067] Simultaneously, the result of persistent exceeding the standard was determined. V pass The rule for determining if the content of needle-shaped and flaky aggregate exceeds the standard is: when the total content of needle-shaped and flaky aggregate exceeds the standard... When the alarm threshold is exceeded (which can be set to 15%), V pass Set as false Otherwise set to true .

[0068] In another embodiment, the system can also implement a tiered storage strategy for data, categorizing it into hot and cold data. Recent task data (created within the last 90 days) is stored in a high-performance database to support fast queries, while historical task data (created more than 90 days ago) is automatically archived to an object storage service to reduce storage costs. The data migration process is triggered by a background scheduled task, and the migration strategy is dynamically adjusted based on the data's creation time and access frequency. The migration task executes once per hour, migrating 100 records each time. During the migration, the integrity markers of the original database records are maintained, and the original database records are deleted after the migration is complete.

[0069] In step S5, the system performs a task tracking and real-time notification process.

[0070] The system generates a unique task ID, which uses UUID format to ensure global uniqueness. The system creates a task state machine object in memory, initially in the PENDING state. Task metadata includes device ID, creation time, trajectory instruction set T, and total number of points N. The task ID generation algorithm uses UUID v4, achieving a generation rate of up to 10,000 task IDs per second.

[0071] Task status: State(task Id)∈{PENDING,RUNNING,COMPLETED,FAILED,CANCELLED}; The state transition function is: ; The state transition function is implemented as follows: When a task is created but has not yet started execution, the state is PENDING; when the task starts performing gimbal movement operations, the state changes from PENDING to RUNNING; when image acquisition and analysis of all patrol points are completed, the state changes from RUNNING to COMPLETED; when abnormal situations such as network timeout, image analysis service returning an error response, or gimbal unresponsiveness occur during task execution, the state changes from the current state to FAILED; when the user sends a cancellation command and the task is in the PENDING or RUNNING state, the state changes to CANCELLED.

[0072] In another embodiment, the system can also implement a task priority scheduling mechanism. The task priority parameter is specified when the task is created, with a priority value ranging from 1 to 10, where a higher value indicates a higher priority. A higher-priority task can interrupt a lower-priority task that is currently executing. The interrupted task records its interruption status and re-enters the waiting queue. This mechanism can meet the timeliness requirements of emergency inspection tasks, controlling the maximum waiting time from task creation to execution start within 30 seconds.

[0073] See Figure 4 The frontend obtains the progress percentage through a periodic polling interface, while the push service proactively broadcasts messages to the frontend at key nodes and when tasks are completed, forming a dual-channel real-time notification mechanism of "polling to ensure a baseline + push to improve timeliness". Specifically, the frontend obtains real-time progress through the ` / progress / {task Id}` interface. The polling interval is set to 2 seconds. This interval ensures that users can promptly perceive the task progress while avoiding unnecessary load on the server due to excessively frequent polling requests. This indicates the percentage of points currently completed. The interface returns the cumulative number of needle-shaped stones. This value is used to display the cumulative quantity of currently discovered needle-like and flaky aggregates on the client side. Stone details can be queried on demand via a separate interface / progress / {task Id} / rocks / {tjId}, which supports paginated queries, returning 20 stone detail records per page.

[0074] When a task's status changes or a key milestone is completed, the system broadcasts a message to clients subscribed to that task via WebSocket. The WebSocket service is implemented using the Spring WebSocket framework. When a connection is established, the client submits a subscription request for the task ID, and the server binds the connection object to the task ID. The types of messages pushed via WebSocket include: task status change messages, location progress update messages, exception alarm messages, and task completion messages. Each message carries the task ID, timestamp, and corresponding payload data. The client performs the appropriate UI update operation based on the message type. The message payload is encoded in JSON format, and the maximum message body length is limited to 4KB.

[0075] See Figure 5 In step S6, the system executes a device-level mutual exclusion and exception rollback process.

[0076] Before a task starts, the system calls `isDeviceRunning(device Id)` to check if the device already has an active task. The device mutual exclusion check is implemented using a `Concurrent Hash Map` data structure from the Java concurrency package, with `deviceId` as the key and the task status object as the value. The check process uses the `putIfAbsent` method to perform the mutual exclusion check. If the return value is null, it means the device is idle and can accept new tasks, and the system stores the task status object in the Map; if the return value is not null, it means the device is already occupied, the system rejects the creation of a new task and returns the error code `ERR_DEVICE_BUSY`.

[0077] During task execution, the device is continuously occupied in the memory map to ensure that the same device does not execute multiple inspection tasks simultaneously. After the task is completed, the system removes the corresponding entry from the map to release the device. The release operation is performed when the task status changes to COMPLETED, FAILED, or CANCELLED.

[0078] If an exception occurs or a cancellation command is received during any of steps S1 to S5, the system captures the exception object and records an error log. The error log records the exception type, exception message, stack trace, task ID, and timestamp of occurrence. The system updates the task status to FAILED or CANCELLED and sends a command to move the PTZ to the initial preset position. After the rollback is complete, the device mutex is released.

[0079] The rollback mechanism requires the following condition for rollback: ShouldRollback = (State=FAILED) ∨ (State=CANCELLED) ∨ (Exception ≠ 0) ); Among them, Should Roll back is a boolean value indicating whether to trigger gimbal rollback; State is the current task status: FAILED indicates task failure, CANCELLED indicates task cancellation, and Exception indicates the captured exception object (non-empty indicates an exception has occurred).

[0080] When this condition is met, the system sends a PTZ rollback command to the initial preset position P0, and performs a device mutex lock release operation after the rollback is completed to ensure that the device resources are properly reclaimed.

[0081] The image processing-based aggregate inspection and analysis system for silos includes an inspection orchestrator, task manager, image analysis service, trajectory recognition service, index storage service, and Web Socket session service. The index storage service is implemented through a four-layer index storage model, including: global index, grouped index, stone details, and particle size distribution.

Claims

1. A method for inspection analysis of a stockpile aggregate based on image processing, characterized in that, Includes the following steps: S1: Acquire panoramic images of the inside of the silo, perform inference on the aggregate accumulation area inside the silo based on the target detection model, output N target detection bounding boxes, calculate the absolute PTZ coordinates of each target area in the gimbal coordinate system, generate cruise trajectory instruction set T, and perform trajectory optimization. S2: take out the current point instruction from the track instruction set T S2: take out the current point instruction from the track instruction set S2: take out the current point instruction from the track instruction set S3: Execute each point sequentially according to the order of the cruise trajectory instruction set T. After the gimbal stabilizes, acquire the image, upload the captured image to the object storage service to obtain the image URL, and submit the image URLs to the image analysis service in sequence to perform aggregate needle-like and flaky recognition and particle size analysis, and perform global index aggregation calculation. S4: Extract global aggregated data, match the capture image URLs of each location with the group indicators according to the index, build a database storage object and write it into the database and associate it with the task ID; S5: Generate a unique task ID, create a task state machine object in memory to manage the state transitions in the task lifecycle, and implement real-time progress notifications through the front-end polling interface and Web Socket push. S6: Before starting a task, check if the device already has an active task. During task execution, the device is continuously occupied. If there is an error or the task is canceled, the PTZ will be triggered to return to the initial preset position and the device mutex lock will be released.

2. The image processing-based bin aggregate inspection and analysis method according to claim 1, characterized in that, The specific process of step S1 is as follows: S11: The system calls the PTZ control interface and sends a command to move the camera to the initial preset position. After the gimbal stabilizes, perform panoramic capture to obtain the original image. ; S12: Based on the original image Trajectory recognition is performed based on the current PTZ parameters. Inference is then made about the aggregate accumulation area within the silo based on the target detection model, and the output is... For each target detection bounding box, the center pixel coordinates are calculated. Based on the obtained center pixel coordinates of the bounding boxes, combined with the camera intrinsic parameters, they are mapped to the gimbal angle offset, and the target zoom factor is calculated to generate the cruise trajectory instruction set T. S13: Based on the PTZ coordinates of N points, calculate the gimbal movement cost between adjacent points, sort them using the nearest neighbor or TSP approximation algorithm, and minimize the total movement path.

3. The image processing-based bin aggregate inspection and analysis method according to claim 2, characterized in that, The specific process of step S2 is as follows: S21: Retrieve the current point instruction from the trajectory instruction set T. Send an absolute PTZ movement command to the PTZ unit, with the target location being... ; S22: Perform gimbal polling and stability assessment: Initialize the stable counter Maximum timeout Polling interval ; No. In the next poll, query the current PTZ position of the PTZ and calculate the position change; Stability is determined based on preset stability criteria; S23: If a task cancellation command is received during the determination process, immediately terminate the polling and trigger the gimbal to return to the initial preset position. .

4. The image processing-based bin aggregate inspection and analysis method according to claim 1, characterized in that, The specific process of step S3 is as follows: S31: According to the trajectory instruction set The order or optimized order, for each point in turn. The S2 mechanism is invoked to wait for the gimbal to stabilize. Once stabilized, high-definition image capture is performed. ; S32: Capture image Upload the image to the object storage service and retrieve the returned image URL, denoted as . ; S33: Perform image analysis on the image URLs in sequence and output the single-point index and the content of needle-shaped and flaky aggregates; S34: When the After the image analysis is completed, a global metric aggregation calculation is triggered, aggregating all... Grouping indicators for each location generates a global quality report.

5. The image processing-based bin aggregate inspection and analysis method according to claim 4, characterized in that, The specific process of step S4 is as follows: S41: When returning the analysis results for the last image, extract the global aggregated data, including global metrics. Grouping Indicator List Particle size distribution and a detailed list of stones ; S42: Match the URLs of the captured images at each location with the group indicators one by one according to the index, add the bounding box coordinates to the group indicators, and establish spatial association; S43: Construct a database storage object, write it to the database, and associate it with the task ID. The stored data includes: Panoramic image URL and PTZ parameters List of URLs for analysis diagrams at each location, coordinates of the bounding box for each location, global content of needle-shaped and flaky aggregates, and statistics on particle size distribution.

6. The image processing-based bin aggregate inspection and analysis method according to claim 1, characterized in that, The specific process of step S5 is as follows: S51: Generate a unique task ID, create a task state machine object in memory, with the initial state being PENDING; record task metadata: device ID, creation time, trajectory instruction set T, and total number of points N; S52: Perform state transition. When the task starts, the state changes to RUNNING. The progress is updated after each point is completed. After all points are completed, the state changes to COMPLETED. If an error occurs, the state changes to FAILED. If the task is canceled, the state changes to CANCELLED. S53: Front-end polling process: The front-end obtains the real-time progress through the / progress / {task Id} interface, and the interface returns the cumulative number of needle-shaped stones; S54: During the real-time push process via Web Socket, when the task status changes or a key node is completed, a message is broadcast to the clients subscribed to the task via Web Socket. After the analysis is completed, the final detection result is pushed.

7. The image processing-based bin aggregate inspection and analysis method according to claim 1, characterized in that, The specific process of step S6 is as follows: S61: Before starting a task, call the device ID to check if the device already has an active task; Query the task status corresponding to device ID in the in-memory Concurrent Hash Map. If a task with a status of PENDING or RUNNING exists, refuse to create a new task and return an error code. S62: The device is continuously occupied in the memory map during task execution, and is removed from the map to release the device after the task is completed; S63: If an exception occurs or a cancellation command is received in any step from S1 to S5, capture the exception object, record the error log, update the task status to FAILED or CANCELLED, and send a command to move the PTZ to the initial preset position. After the rollback is completed, release the device mutex lock.

8. A bin aggregate inspection and analysis system based on image processing, used to implement the bin aggregate inspection and analysis method based on image processing as described in any one of claims 1-7, characterized in that, This includes an inspection orchestrator, task manager, image analysis service, trajectory recognition service, indicator storage service, and Web Socket session service; The indicator storage service is implemented through a four-layer indicator storage model, including global indicators, grouped indicators, stone details, and particle size distribution.

Images