An AR technology-based remote collaborative diagnosis system for grain storage equipment

Abstract

The application is suitable for the technical field of remote cooperative diagnosis, and discloses a grain storage equipment remote cooperative diagnosis system based on AR technology, which comprises: a collection module, which collects operation data and field image data of the grain storage equipment, and obtains state vectors and spatial coordinate data at each sampling time based on the collected data; an analysis module, which continuously obtains the state vectors and constructs an analysis period, obtains recent state vectors and previous state vectors in the analysis period, and compares the state vectors to obtain a fault degree; and a demarcation module, which filters fault time according to the fault degree, and demarcates a fault area range based on the spatial coordinate data corresponding to the fault time. The application makes the finally obtained fault degree have differentiated response sensitivity to different types of faults, avoids misjudgment of a single threshold in a complex fault scene, and improves the accuracy and adaptability of fault diagnosis.

Description

Technical Field

[0001] This invention relates to the field of remote collaborative diagnostic technology, specifically to a remote collaborative diagnostic system for grain storage equipment based on AR technology. Background Technology

[0002] Grain storage is a crucial link in ensuring national food security, and the healthy operation of grain storage equipment directly affects the efficiency and safety of grain storage and procurement. Currently, with the accelerated integration of new-generation information technologies such as the Internet of Things, big data, and artificial intelligence with the traditional grain storage industry, some large grain depots have begun deploying multi-source sensing devices such as temperature and humidity sensors and vibration sensors to attempt digital monitoring of the operational status of grain storage equipment. However, the fault diagnosis of existing grain storage equipment still mainly relies on the experience-based judgment of on-site personnel and on-site diagnosis by experts. Although some grain depots have deployed sensors to collect equipment status information, this data is often fragmented and isolated, failing to form a complete monitoring and analysis system. The equipment operation and maintenance of most grassroots grain depots still relies on the traditional post-incident repair model. When equipment malfunctions, on-site operators, lacking professional diagnostic capabilities, find it difficult to determine the cause of the fault in a timely and accurate manner, and must wait for manufacturer technicians or domain experts to arrive on-site to conduct diagnosis and repair. This reliance on on-site personnel results in excessively long fault response times, especially in remote areas with inconvenient transportation. Experts often need several hours or even days to arrive on-site after receiving notification, during which time the equipment is forced to shut down, severely impacting the normal operation of grain storage and procurement.

[0003] In existing technologies, fault diagnosis of grain storage equipment mainly relies on the experience of on-site personnel and on-site diagnosis by experts, lacking AR-based remote collaborative diagnosis methods, resulting in untimely fault response and long maintenance cycles. Summary of the Invention

[0004] The purpose of this invention is to provide a remote collaborative diagnostic system for grain storage equipment based on AR technology, so as to solve the problems of untimely fault diagnosis and response and long maintenance cycle of grain storage equipment in the prior art.

[0005] The objective of this invention can be achieved through the following technical solution: a remote collaborative diagnostic system for grain storage equipment based on AR technology, comprising:

[0006] The data acquisition module collects operational data and on-site image data of the grain storage equipment, and obtains state vectors and spatial coordinate data at each sampling time based on the collected data;

[0007] The analysis module continuously acquires state vectors and constructs analysis periods. It then compares recent and previous state vectors within the analysis period to determine the degree of fault.

[0008] The delineation module filters the fault time based on the fault severity and delineates the fault area range based on the spatial coordinate data corresponding to the fault time.

[0009] The maintenance module acquires on-site image data at the time of the fault based on the fault area and the fault time, determines the marked area, binds the marked area with the fault degree, sends it to the AR glasses for display and performs maintenance, and updates the status vector after maintenance is completed.

[0010] Preferably, the operating data of the grain storage equipment includes temperature value, vibration amplitude, pressure value and current value, and the state vector refers to the arrangement of the temperature value, vibration amplitude, pressure value and current value acquired at each moment in a fixed order.

[0011] Preferably, the specific steps for obtaining spatial coordinate data are as follows:

[0012] The state vector, image frame, and spatial attitude data at each sampling moment are transmitted to the remote server. The spatial attitude data specifically includes the pitch angle, yaw angle, and roll angle of the AR glasses.

[0013] The server extracts feature points from image frames, performs feature point matching on image frames at adjacent time points, and obtains matching feature point pairs.

[0014] The attitude change and translation vector of adjacent image frames are obtained from the spatial attitude data. A rotation matrix is ​​constructed based on the attitude change. The spatial coordinate data of the equipment surface spatial points corresponding to the matching feature point pairs in the server coordinate system are solved by combining the rotation matrix and translation vector through spatial geometric constraints.

[0015] Establish a mapping function between pixel coordinates in an image frame and spatial point coordinates on the equipment surface.

[0016] Preferably, the specific steps for obtaining the degree of fault are as follows:

[0017] The analysis period is formed by the state vectors of B consecutive sampling times, where B is a positive integer. The D1 state vectors at the end of the analysis period are taken as the recent state vectors, and the D2 state vectors at the beginning of the analysis period are taken as the previous state vectors. D1 and D2 are both positive integers less than B, and the sum of D1 and D2 is equal to B.

[0018] Calculate the difference factor and offset factor between the recent state vector and the previous state vector for each running data;

[0019] The degree of abnormality for each type of operational data is determined based on the combination relationship between the difference factor and the offset factor of each operational data.

[0020] The anomaly level of each operational data point is normalized and then weighted and summed using the corresponding offset factor to obtain the fault level at the current moment.

[0021] Preferably, the specific steps for calculating the difference factor between the recent state vector and the previous state vector for each running data are as follows:

[0022] Extract time-series data of each operating parameter from the analysis period to obtain recent and earlier subsequences;

[0023] Calculate the average rate of change and average acceleration of each operating parameter in the recent subsequence, and calculate the reference baseline value, reference fluctuation value and reference average rate of change of each operating parameter in the previous subsequence;

[0024] The deviation value of the rate of change is obtained by subtracting the corresponding reference average rate of change from the recent average rate of change of each operating parameter. The absolute value of the recent average acceleration of the corresponding operating parameter is multiplied by the preset penalty coefficient to obtain the acceleration penalty value. The absolute value of the deviation value of the rate of change of the corresponding operating parameter is added to the acceleration penalty value and then divided by the product of the reference fluctuation value and the preset safety factor to obtain the difference factor of the corresponding operating parameter.

[0025] Preferably, the specific steps for calculating the offset multiple of each running data between the recent state vector and the previous state vector are as follows:

[0026] Obtain the current value of each running data from the state vector at the current sampling time;

[0027] The current offset of the running parameter is obtained by subtracting the reference baseline value of the corresponding previous subsequence from the current value of each running data.

[0028] Divide the current offset by the reference fluctuation value and take the absolute value to obtain the offset multiple of the operating parameter.

[0029] Preferably, the specific steps for determining the degree of abnormality of each type of operational data based on the combination relationship between the difference factor and the offset factor of each operational data are as follows:

[0030] A first difference threshold, a second difference threshold, a first offset threshold, a second offset threshold, and a third offset threshold are preset for each operating parameter, wherein the first difference threshold is greater than the second difference threshold, and the first offset threshold, the second offset threshold, and the third offset threshold decrease sequentially.

[0031] If the difference factor of each operating parameter is greater than the corresponding first difference threshold and the offset multiple of each operating parameter is greater than the corresponding first offset threshold, then the abnormality level is assigned the first level value.

[0032] If the difference factor of each operating parameter is between the corresponding first difference threshold and the second difference threshold, and the offset multiple of each operating parameter is greater than the corresponding second offset threshold, then the abnormality level is assigned the second level value.

[0033] If the difference factor of each operating parameter is less than the corresponding second difference threshold and the offset multiple of each operating parameter is greater than the corresponding third offset threshold, then the abnormality level is assigned the third level value.

[0034] Otherwise, assign the value to the fourth level, with all levels arranged from highest to lowest.

[0035] Preferably, the specific steps for normalizing the anomaly levels of each set of operational data and then weighting and summing them using the corresponding offset factors to obtain the fault level at the current moment are as follows:

[0036] Obtain the sum of the abnormality levels of each running data, and divide the abnormality level of each running data by the sum to obtain the weighting factor of each running data.

[0037] Multiply the weight factor of each running data by the corresponding offset factor to obtain the weighted offset of each running data.

[0038] The fault level at the current moment is obtained by summing the weighted offsets of all running data.

[0039] Preferably, the specific steps for delineating the fault area range based on the spatial coordinate data corresponding to the fault time are as follows:

[0040] For an image frame marked as a fault moment, obtain the spatial point set corresponding to that image frame;

[0041] Divide the spatial point set into several depth layers at equal intervals along the depth direction, calculate the center point of each layer, and detect abrupt transition layers where the distance between the center points of adjacent layers is abnormal.

[0042] Based on the location of the jump layer, a set of candidate fault layers is selected. From the candidate fault layers, a high-density layer with a feature point density higher than the average density is selected. All feature points in the high-density layer constitute a high-density feature point set.

[0043] Outlier feature points are removed from the high-density feature point set to obtain the final fault feature point set;

[0044] The weighted boundary center and boundary radius are calculated based on the spatial coordinates of each feature point in the final fault feature point set, and the fault area boundary box is determined based on the weighted boundary center and boundary radius.

[0045] Preferably, the specific steps for calculating the weighted boundary center and boundary radius based on the spatial coordinates of each feature point in the final fault feature point set, and determining the fault region boundary box based on the weighted boundary center and boundary radius, are as follows:

[0046] The feature point density value of the depth layer where each feature point is located in the final fault feature point set is used as the weighting coefficient of the feature point;

[0047] Calculate the total weighted coordinates of all feature points in the horizontal, vertical and depth directions respectively, and then divide them by the sum of the weighting coefficients of all feature points to obtain the coordinates of the center of the weighted boundary in the three directions.

[0048] Calculate the Euclidean distance from each feature point to the center of the weighted boundary, and take the maximum value as the boundary radius;

[0049] A spherical boundary is determined by using the weighted boundary center as the center and the boundary radius as the radius. Then, a three-dimensional cube region is constructed with the weighted boundary center as the center and the boundary radius as half the side length. This yields the six boundary coordinate values ​​of the fault region boundary box. The six boundary coordinate values ​​refer to the minimum and maximum coordinate values ​​in the horizontal direction, the minimum and maximum coordinate values ​​in the vertical direction, and the minimum and maximum coordinate values ​​in the depth direction.

[0050] The beneficial effects of this invention are:

[0051] 1. This invention constructs a dynamic analysis period consisting of continuously sampled moments. It compares the recent state vector at the end of the period with the earlier state vector at the beginning of the period to obtain the difference factor and offset factor for each operating parameter. The calculation of the difference factor for each operating parameter incorporates the deviation of the recent rate of change from the previous period and a recent acceleration penalty term, effectively identifying abnormal trends such as accelerated rate of change and increased acceleration. The offset factor reflects the deviation of the current instantaneous value from the previous baseline. Based on the combination of the difference factor and the offset factor, the abnormality level of each operating parameter is divided into multiple levels, and the fault degree is obtained by normalizing and weighting the abnormality levels. This invention enables the finally obtained fault degree to have differentiated response sensitivity to different types of faults, avoiding misjudgment by a single threshold in complex fault scenarios, and improving the accuracy and adaptability of fault diagnosis.

[0052] 2. In the spatial localization of fault areas, this invention, for image frames marked as the time of the fault, utilizes the corresponding three-dimensional spatial point set to create layers along the depth direction. By detecting the distance between the center points of adjacent layers, abrupt change layers are identified to pinpoint locations of abrupt structural changes on the equipment surface. Candidate fault layers are then filtered based on the location of these abrupt change layers. Low-density layers are eliminated by comparing feature point density with the average density, and outlier feature points are removed using the statistical distribution of proximity distances. Finally, a set of fault feature points for the fault area is obtained. This invention separates the inherent structural features of the equipment surface from local abnormal areas through abrupt change layer detection. Furthermore, density filtering and outlier removal ensure that the remaining feature point group has spatial consistency, solving the problem of difficulty in locating fault areas due to their mixing with background structures.

[0053] 3. This invention provides a complete remote collaborative diagnostic link, from multi-source operational data acquisition and fault severity quantification analysis to fault area spatial localization and augmented reality spatial annotation. By aligning state vectors with image frames in the time dimension, abnormal fluctuations in operational data can drive the automatic annotation of corresponding spatial locations in image frames. Then, using epipolar geometric constraints, the three-dimensional spatial coordinates of the equipment surface are recovered from the image frame sequence. This allows fault area delineation to be independent of pre-established equipment 3D models and provides adaptive capability for different types of grain storage equipment. After fault elimination, a secondary confirmation is performed by clearing cached data and reconstructing the analysis period. Only when no fault is detected at multiple consecutive sampling times is the fault determined to be eliminated, preventing misjudgment of fault status due to occasional data fluctuations. This forms a complete closed loop from anomaly detection and spatial localization to repair confirmation, enabling on-site personnel to locate fault locations and carry out repairs based on augmented reality annotations without waiting for experts to arrive, thus shortening fault response and repair time. Attached Figure Description

[0054] The present invention will now be further described with reference to the accompanying drawings.

[0055] Figure 1 This is a framework diagram of the system provided in the embodiments of the present invention. Detailed Implementation

[0056] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0057] Please see Figure 1 As shown, the present invention is a remote collaborative diagnostic system for grain storage equipment based on AR technology, comprising:

[0058] Acquisition Module 100: Acquires operational data and on-site image data of grain storage equipment, and obtains state vectors and spatial coordinate data at each sampling time based on the acquired data.

[0059] In this embodiment of the invention, the operating data includes temperature value, vibration amplitude, pressure value, and current value;

[0060] The operational data is obtained by installing temperature sensors, vibration sensors, pressure sensors, and current sensors at predetermined measuring points on the grain storage equipment. Specifically, a temperature sensor is installed on the outer wall of the bearing housing of the grain storage equipment to collect the bearing temperature value during the operation of the grain storage equipment; a vibration sensor is installed on the shell surface of the grain storage equipment to collect the shell vibration amplitude during the operation of the grain storage equipment; a pressure sensor is installed at the pressure measuring port of the hydraulic system pipeline of the grain storage equipment to collect the hydraulic system pressure value during the operation of the grain storage equipment; and a current sensor is installed on the power supply line of the drive motor of the grain storage equipment to collect the drive motor current value during the operation of the grain storage equipment.

[0061] It should be noted that all sensors are sampled synchronously according to a unified sampling clock and sampling period;

[0062] The state vector at each sampling moment refers to the state vector formed by arranging the temperature, vibration amplitude, pressure, and current values ​​acquired at each moment in a fixed order. ,in, Indicates the temperature value. Indicates vibration amplitude, Indicates pressure value and This represents the current value, and n is the index of the sampling time.

[0063] The on-site video data is collected by personnel wearing AR glasses. The camera on the AR glasses collects on-site video data of the grain storage equipment in each sampling period, and obtains the image frame at the nth sampling time. The inertial measurement unit built into the AR glasses simultaneously collects the spatial attitude data of the AR glasses when collecting the image frame. The spatial attitude data specifically includes the pitch angle, yaw angle and roll angle of the AR glasses.

[0064] The steps for acquiring spatial coordinate data are as follows: transmit the state vector, image frame, and spatial attitude data at the nth sampling time to the remote collaborative diagnostic server via 5G communication.

[0065] After receiving all the data, the remote collaborative diagnostic server extracts scale-invariant feature points from each image frame. Each feature point has horizontal and vertical pixel coordinates in the image frame, and the pixel coordinates of all feature points constitute a feature point set.

[0066] The remote collaborative diagnostic server performs feature point matching on image frames at adjacent time points. The matching method is to calculate the Euclidean distance between the pixel coordinates of each feature point in two adjacent image frames, and take a pair of feature points whose Euclidean distance is less than a preset matching threshold as the matching feature points. All matching feature points constitute a set of matching feature point pairs.

[0067] The remote collaborative diagnostic server obtains the attitude values ​​of the AR glasses in adjacent image frames from the spatial attitude data, calculates the attitude change between adjacent image frames, including pitch angle change, yaw angle change and roll angle change, and calculates the translation vector of the AR glasses in adjacent image frames by fusing inertial measurement unit data and image frame feature point matching results through visual inertial odometry calculation method.

[0068] A rotation matrix is ​​constructed based on the attitude change. The epipolar geometric constraint relationship is applied to the normalized coordinates of each pair of matching feature points. The rotation and translation vectors are combined to solve the problem simultaneously, and the spatial coordinate data of the grain storage equipment surface spatial points corresponding to each pair of matching feature points in the AR glasses coordinate system is obtained. The spatial coordinate data includes horizontal coordinate values, vertical coordinate values ​​and depth coordinate values.

[0069] And thus, a mapping function is established between the coordinates of each pixel in the image frame and the coordinates of spatial points on the surface of the grain storage equipment, which is: ,in, For mapping functions, It is a pixel coordinate vector. This is the corresponding spatial coordinate vector.

[0070] It should be noted that the epipolar geometric constraint relationship is a well-known technique in the field, and its specific derivation process will not be elaborated here. Those skilled in the art can use the epipolar geometric constraint relationship in combination with rotation and translation to solve for the spatial coordinate data based on the above description.

[0071] Furthermore, the AR-based remote collaborative diagnostic system for grain storage equipment also includes:

[0072] Analysis module 200: Continuously acquires state vectors and constructs analysis periods, acquires recent state vectors and previous state vectors within the analysis period and compares them to obtain the degree of fault.

[0073] In this embodiment of the invention, the analysis period refers to the time range covered by B consecutive sampling times used to analyze the operating status of grain storage equipment, where B is a positive integer. The analysis period is formed by the remote collaborative diagnostic server arranging the continuously received state vectors in the order of sampling times. Specifically, each time the remote collaborative diagnostic server receives a new state vector, it adds it to the end of the analysis period and discards the state vector at the beginning of the analysis period, that is, the state vector at the earliest sampling time, so that the latest B state vectors are always retained in the analysis period. The B state vectors are arranged in the order of sampling times to form the analysis period at the current moment.

[0074] The recent state vectors refer to the D1 state vectors at the end of the analysis period, and the earlier state vectors refer to the D2 state vectors at the beginning of the analysis period. Here, D1 and D2 are both positive integers less than B, and the sum of D1 and D2 is equal to B.

[0075] It should be noted that the recent state vector and the previous state vector are connected sequentially on the time axis and do not overlap with each other. Together, they constitute all the state vectors in the complete analysis period.

[0076] The fault severity indicates the degree of abnormality in the current operating state of the grain storage equipment; the higher the value, the more severe the fault. The steps to obtain the fault severity are as follows:

[0077] The temperature difference factor, vibration difference factor, pressure difference factor, and current difference factor are obtained separately. The methods for obtaining the temperature difference factor, vibration difference factor, pressure difference factor, and current difference factor are similar. This embodiment uses the acquisition of the temperature difference factor as an example for illustration.

[0078] The steps for obtaining the temperature difference factor are as follows:

[0079] The temperature values ​​at sampling time B are extracted from the analysis period to form a temperature sequence;

[0080] Extract D1 temperature values ​​from the recent state vector in the temperature sequence, arrange them in the order of sampling time to obtain the recent temperature subsequence, calculate the difference between the temperature values ​​at each two adjacent sampling times in the recent temperature subsequence to obtain D1-1 first-order temperature difference values, and obtain the arithmetic mean of D1-1 first-order temperature difference values ​​to obtain the recent average rate of temperature change.

[0081] For each pair of adjacent first-order difference values ​​in the D1-1 temperature first-order difference values, calculate the difference again to obtain the D1-2 second-order difference values, and obtain the arithmetic mean of the D1-2 second-order difference values, which is used as the recent average temperature acceleration.

[0082] Extract D2 temperature values ​​from the previous state vector in the temperature sequence, arrange them in the order of sampling time to obtain the previous temperature subsequence, obtain the arithmetic mean of D2 temperature values ​​in the previous subsequence, record it as the temperature reference baseline value, and calculate the standard deviation of D2 temperature values ​​in the previous subsequence, record it as the temperature reference fluctuation value.

[0083] Calculate the difference between the temperature values ​​at each two adjacent sampling times in the previous subsequence, obtain D2-1 reference first-order difference values, and obtain the arithmetic mean of D2-1 reference first-order difference values, which is used as the reference average rate of change.

[0084] Next, subtract the reference average rate of change from the recent average rate of temperature change to obtain the deviation value of the rate of temperature change. Then, multiply the absolute value of the recent average acceleration of temperature by the preset temperature acceleration penalty coefficient to obtain the temperature acceleration penalty value. Add the absolute value of the deviation value of the rate of temperature change to the temperature acceleration penalty value and then divide it by a product. This product is the result of multiplying the temperature reference fluctuation value by the preset temperature fluctuation safety factor. Finally, the temperature difference factor is obtained.

[0085] It should be noted that the temperature acceleration penalty coefficient and the temperature fluctuation safety factor are determined by statistical analysis of measured data of grain storage equipment under normal operating conditions and simulated fault conditions. The temperature acceleration penalty coefficient is used to amplify the temperature difference factor value when the temperature acceleration deteriorates, so as to identify faults earlier.

[0086] Next, the temperature offset multiple, vibration offset multiple, pressure offset multiple, and current offset multiple are obtained separately. The methods for obtaining the temperature offset multiple, vibration offset multiple, pressure offset multiple, and current offset multiple are similar. This embodiment uses the acquisition of the temperature offset multiple as an example for illustration.

[0087] The temperature offset factor is obtained by obtaining the current temperature value from the state vector at the current sampling time, subtracting the temperature reference value from the current temperature value to obtain the current temperature offset, dividing the current temperature offset by the temperature reference fluctuation value, and taking the absolute value to finally obtain the temperature offset factor.

[0088] Then, the degree of abnormality of each type of multi-source operating data is determined based on the combination relationship between each difference factor and the corresponding offset multiple. In this embodiment, the degree of temperature abnormality is obtained by taking the combination relationship between temperature difference factor and temperature offset multiple as an example. The methods for obtaining the degree of vibration abnormality, pressure abnormality and current abnormality are the same as those for obtaining the degree of temperature abnormality.

[0089] The steps for obtaining the degree of temperature anomaly are as follows:

[0090] A first temperature difference threshold, a second temperature difference threshold, a first temperature offset threshold, a second temperature offset threshold, and a third temperature offset threshold are preset and compared with a temperature difference factor and a temperature offset multiple. The first temperature difference threshold is greater than the second temperature difference threshold, and the first temperature offset threshold, the second temperature offset threshold, and the third temperature offset threshold decrease in sequence.

[0091] It should be noted that the various temperature thresholds are set according to the statistical distribution range of temperature difference factors and temperature deviation multiples of grain storage equipment under normal operating conditions. They are used to quantify the degree of temperature anomaly into different level values, so that different combinations of temperature difference factors and temperature deviation multiples can correspond to the corresponding level values.

[0092] If the temperature difference factor is greater than the first temperature difference threshold and the temperature offset multiple is greater than the first temperature offset threshold, the temperature anomaly level is assigned a first-level value. If the temperature difference factor is less than the first temperature difference threshold but greater than the second temperature difference threshold and the temperature offset multiple is greater than the second temperature offset threshold, the temperature anomaly level is assigned a second-level value. If the temperature difference factor is less than the second temperature difference threshold and the temperature offset multiple is greater than the third temperature offset threshold, the temperature anomaly level is assigned a third-level value. In other cases, the temperature anomaly level is assigned a fourth-level value.

[0093] It should be noted that the first, second, third, and fourth level values ​​are arranged from high to low. The first level value indicates a severe anomaly when both the temperature difference factor and the temperature deviation multiple are in the highest range. The second level value indicates a persistent deviation when both the temperature difference factor and the temperature deviation multiple are in the moderate range. The third level value indicates a data limit violation when the temperature deviation multiple alone exceeds the minimum threshold. The fourth level value indicates that both the temperature difference factor and the temperature deviation multiple are within the normal range. The four levels are arranged from high to low to reasonably distinguish the different fault modes in terms of the degree of anomaly.

[0094] The abnormality levels of temperature, vibration, pressure, and current are normalized. Specifically, the sum of the abnormality levels of temperature, vibration, pressure, and current is obtained, and then the abnormality levels of temperature, vibration, pressure, and current are divided by the sum of the abnormality levels to obtain the corresponding temperature weighting factor, vibration weighting factor, pressure weighting factor, and current weighting factor.

[0095] Multiply the temperature weighting factor by the temperature offset factor to obtain the temperature-weighted offset; multiply the vibration weighting factor by the vibration offset factor to obtain the vibration-weighted offset; multiply the pressure weighting factor by the pressure offset factor to obtain the pressure-weighted offset; multiply the current weighting factor by the current offset factor to obtain the current-weighted offset; and then add the temperature-weighted offset, vibration-weighted offset, pressure-weighted offset, and current-weighted offset to obtain the fault level at the current moment.

[0096] Specifically, by comparing and analyzing recent state vectors with previous state vectors and using corresponding level value judgment rules, different abnormal modes are reasonably classified. Then, based on the degree of abnormality of each operating data, the corresponding weight factors are adaptively adjusted and allocated to generate corresponding weight factors, so that the final fault degree is more in line with the actual fault severity of grain storage equipment.

[0097] Furthermore, the AR-based remote collaborative diagnostic system for grain storage equipment also includes:

[0098] Delineation Module 300: Filters fault times based on fault severity and delineates the fault area range based on the spatial coordinate data corresponding to the fault times.

[0099] In this embodiment of the invention, the specific steps for filtering the fault time based on the fault severity are as follows:

[0100] The fault severity at the current sampling time is compared with a preset fault threshold. If the fault severity at the current sampling time is greater than the preset fault threshold, it is determined that the grain storage equipment has a fault at the current sampling time, and the current sampling time is marked as the fault time. If the fault severity at the current sampling time is not greater than the preset fault threshold, it is determined that the grain storage equipment is in normal operation at the current sampling time.

[0101] The specific steps for defining the fault area are as follows:

[0102] For each fault moment, the image frame of that fault moment and its corresponding spatial point set are acquired. The spatial point set is composed of the spatial coordinate data of the surface space of the grain storage equipment corresponding to all feature points in the image frame of that fault moment in the AR glasses coordinate system.

[0103] Sort all feature points in the spatial point set in ascending order of depth direction coordinate value, and mark the feature point with the smallest depth direction coordinate value as the near seed point and the feature point with the largest depth direction coordinate value as the far seed point.

[0104] Using the near seed point and the far seed point as the two endpoints of the depth direction, the entire spatial point set is divided into G depth layers at equal intervals along the depth direction, where G is a positive integer greater than 3, and each depth layer contains all feature points falling within the depth interval of that layer.

[0105] Calculate the average horizontal and vertical coordinates of all feature points in each depth layer, then take the median coordinate of the depth direction of that depth layer. These three values ​​together constitute the center point of that depth layer. Calculate the Euclidean distance between the center points of two adjacent depth layers. A total of G-1 interlayer distance values ​​are obtained. Calculate the arithmetic mean of all interlayer distance values ​​and record it as the average distance. At the same time, calculate the standard deviation of all interlayer distance values ​​and record it as the standard deviation of the distance.

[0106] For each inter-layer distance value, when the inter-layer distance value is greater than the sum of the products of the average inter-layer distance and the preset inter-layer distance deviation coefficient multiplied by the standard deviation of the distance, the position corresponding to the inter-layer distance value is marked as a jump layer, and the depth layer number in which it is located is recorded.

[0107] It should be noted that the preset interlayer distance deviation coefficient is an empirical value, and its specific value can be adjusted according to the type of grain storage equipment.

[0108] If the number of jump layers is zero, all G depth layers are included in the candidate fault layer set. If the number of jump layers is one or more, all jump layers are used as split points to divide the depth direction into multiple continuous segments. Then, the continuous segment with the highest average feature point density is selected as the candidate fault layer set.

[0109] It should be noted that the average feature point density is the arithmetic mean of the feature point densities of each depth layer within the continuous segment. If the average feature point density of multiple continuous segments is the same, the continuous segment with the most feature points is selected as the candidate fault layer set.

[0110] For each depth layer in the candidate fault layer set, the feature point density of the depth layer is obtained by dividing the number of feature points in the depth layer by the space volume occupied by the depth layer. Then, the arithmetic mean of the feature point densities of all candidate depth layers is calculated and recorded as the average density value. Depth layers with feature point densities greater than the average density value are marked as high-density layers. All feature points in all high-density layers constitute a high-density feature point set.

[0111] Calculate the Euclidean distance between each feature point in the high-density feature point set and all other feature points in the high-density feature point set. Take the minimum Euclidean distance as the nearest neighbor value of the feature point. Calculate the arithmetic mean of the nearest neighbor values ​​of all feature points in the high-density feature point set and record it as the nearest neighbor mean. At the same time, calculate the standard deviation of the nearest neighbor values ​​of all feature points in the high-density feature point set and record it as the nearest neighbor standard deviation. For each feature point in the high-density feature point set, if its nearest neighbor value is greater than the sum of the product of the nearest neighbor mean and the preset first coefficient multiplied by the nearest neighbor standard deviation, the feature point is marked as an outlier feature point and removed from the high-density feature point set. The remaining feature points constitute the final fault feature point set.

[0112] It should be noted that the first coefficient is used to control the outlier removal intensity. The larger the value of the first coefficient, the more feature points are retained, and the smaller the value, the fewer feature points are retained.

[0113] For each feature point in the final set of fault feature points, the feature point density value of the depth layer where the feature point is located is used as the weighting coefficient of the feature point. The total weighted coordinate values ​​of all feature points in the horizontal, vertical and depth directions are calculated respectively. The total weighted coordinate value is the sum of the products of the coordinate values ​​in each direction and the corresponding weighting coefficients. Then, it is divided by the sum of the weighting coefficients of all feature points to obtain the coordinate values ​​of the three directions of the weighted boundary center.

[0114] Calculate the Euclidean distance from each feature point in the final fault feature point set to the center of the weighted boundary, and take the maximum Euclidean distance from all feature points to the center of the weighted boundary as the boundary radius;

[0115] A spherical boundary is determined by taking the weighted boundary center as the sphere center and the boundary radius as the radius. Then, a three-dimensional cube region is constructed with the sphere center as the center and the boundary radius as half the side length. The coordinate range of this cube region in the horizontal, vertical and depth directions is determined by adding and subtracting the boundary radius from the sphere center coordinate value. The boundary value of the above coordinate range is used to determine the fault feature region boundary box in the AR glasses coordinate system at the time of the fault. Finally, the six boundary coordinate values ​​of the fault feature region boundary box at the time of the fault are obtained.

[0116] It should be noted that the six boundary coordinate values ​​refer to the minimum and maximum coordinate values ​​in the horizontal direction, the minimum and maximum coordinate values ​​in the vertical direction, and the minimum and maximum coordinate values ​​in the depth direction.

[0117] Specifically, by layering along the depth direction and detecting abrupt transition layers to screen candidate fault layers, interference caused by the background structure on the surface of grain storage equipment can be eliminated. Then, outliers are removed by filtering by feature point density and statistical analysis of local proximity, so that the fault feature points that are finally retained have consistency. Finally, the boundary center is calculated by weighting layer density and fault degree, and the conversion between spherical boundary and cube boundary box is combined to make the boundary of the defined fault area conform to the spatial distribution of the actual fault location.

[0118] Furthermore, the AR-based remote collaborative diagnostic system for grain storage equipment also includes:

[0119] Inspection module 400: Based on the fault area range and fault time, it acquires on-site image data at the time of the fault, determines the marked area, binds the marked area with the fault degree, sends it to the AR glasses for display and performs repair, and updates the status vector after the repair is completed.

[0120] In this embodiment of the invention, the three-dimensional fault space region in the AR glasses coordinate system at the time of the fault is determined according to the boundary values ​​of the six coordinate ranges. Eight spatial corner points of the three-dimensional fault space region are taken, and the inverse function of the mapping function established by the acquisition module 100 is used to map the three-dimensional spatial coordinate values ​​of the eight spatial corner points back to the pixel coordinate system of the image frame corresponding to the time of the fault, so as to obtain eight corresponding pixel coordinate points.

[0121] From the eight obtained pixel coordinates, find four pixel boundary values, namely the minimum and maximum pixel coordinates in the horizontal direction and the minimum and maximum pixel coordinates in the vertical direction. Use these four pixel boundary values ​​to determine the rectangular annotation area on the image frame.

[0122] The remote collaborative diagnostic server binds the severity of the fault at the moment of the fault to the rectangular marked area, generates AR spatial marking information, and sends it to the AR glasses worn by the personnel on site via the 5G communication module. After receiving the AR spatial marking information, the AR glasses overlay the rectangular marked area on the display screen and display the fault severity value in the vicinity of the rectangular marked area. The personnel on site can know the specific spatial location of the fault on the grain storage equipment through the rectangular marked area overlaid on the AR glasses screen, and know the severity of the fault through the displayed fault severity value.

[0123] Remote experts can identify the fault location by viewing the image frames and rectangular marked areas stored in the remote collaborative diagnostic server and send the repair guidance plan to the AR glasses for on-site personnel to refer to and follow. After the on-site personnel complete the troubleshooting according to the repair guidance plan, they can input the repair completion command through the interactive interface of the AR glasses.

[0124] After receiving the instruction, the remote collaborative diagnostic server marks the current time as the maintenance completion time and starts receiving the state vector of the new sampling time from that time. It clears all cached data accumulated in the previous analysis period, takes the first state vector received after the maintenance completion time as the starting vector of the new analysis period, reconstructs the analysis period, and executes the analysis module 200 and the delineation module 300 in sequence. If the fault degree of B consecutive sampling times is not greater than the preset fault threshold, it is determined that the fault has been eliminated and the equipment status is marked as normal operation.

[0125] It should be noted that when the number of sampling times is less than B, all currently collected sampling times are used as the analysis period. The absolute offsets of the temperature value, vibration amplitude, pressure value, and current value in the state vector of the current sampling time from the corresponding preset normal operation benchmark values ​​are directly calculated. Each absolute offset is divided by the upper limit of the preset safe fluctuation range of the corresponding physical quantity to obtain the simplified offset multiple of each physical quantity. Then, the simplified offset multiples are weighted and summed to obtain the simplified fault degree. If the simplified fault degree is greater than the preset independent simplified fault threshold, it is determined that there is a fault at the current sampling time. The simplified fault threshold is determined based on the simplified fault degree distribution statistics of the historical normal data of grain storage equipment.

[0126] It should be noted that the preset normal operation benchmark value is the long-term statistical average value of each physical quantity of the grain storage equipment under historical normal operation conditions. This value is calculated and stored in advance by the remote collaborative diagnostic server through continuous collection of normal operating condition data at the beginning of the operation of the grain storage equipment.

[0127] If a fault severity greater than the preset fault threshold is detected again during the reconstructed analysis period, it is determined that the fault has not been completely eliminated or a new fault has occurred. This step is then repeated, and new AR annotation information is generated and sent to the AR glasses for display.

[0128] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0129] The preferred embodiments of the invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention.

Claims

1. A remote collaborative diagnostic system for grain storage equipment based on AR technology, characterized in that, include: The data acquisition module collects operational data and on-site image data of the grain storage equipment, and obtains state vectors and spatial coordinate data at each sampling time based on the collected data; The analysis module continuously acquires state vectors and constructs analysis periods. It then compares recent and previous state vectors within the analysis period to determine the degree of fault. The delineation module filters the fault time based on the fault severity and delineates the fault area range based on the spatial coordinate data corresponding to the fault time. The maintenance module acquires on-site image data at the time of the fault based on the fault area and the fault time, determines the marked area, binds the marked area with the fault degree, sends it to the AR glasses for display and performs maintenance, and updates the status vector after maintenance is completed.

2. The remote collaborative diagnostic system for grain storage equipment based on AR technology according to claim 1, characterized in that, The operating data of the grain storage equipment includes temperature value, vibration amplitude, pressure value and current value. The state vector is composed of the temperature value, vibration amplitude, pressure value and current value obtained at each moment, arranged in a fixed order.

3. The remote collaborative diagnostic system for grain storage equipment based on AR technology according to claim 2, characterized in that, The specific steps for obtaining spatial coordinate data are as follows: The state vector, image frame, and spatial attitude data at each sampling moment are transmitted to the remote server. The spatial attitude data specifically includes the pitch angle, yaw angle, and roll angle of the AR glasses. The server extracts feature points from image frames, performs feature point matching on image frames at adjacent time points, and obtains matching feature point pairs. The attitude change and translation vector of adjacent image frames are obtained from the spatial attitude data. A rotation matrix is ​​constructed based on the attitude change. The spatial coordinate data of the equipment surface spatial points corresponding to the matching feature point pairs in the server coordinate system are solved by combining the rotation matrix and translation vector through spatial geometric constraints. Establish a mapping function between pixel coordinates in an image frame and spatial point coordinates on the equipment surface.

4. The remote collaborative diagnostic system for grain storage equipment based on AR technology according to claim 3, characterized in that, The specific steps for obtaining the degree of fault are as follows: The analysis period is formed by the state vectors of B consecutive sampling times. The D1 state vectors at the end of the analysis period are taken as the recent state vectors, and the D2 state vectors at the beginning of the analysis period are taken as the previous state vectors. Here, B, D1, and D2 are all positive integers, and D1+D2=B. Calculate the difference factor and offset factor between the recent state vector and the previous state vector for each running data; The degree of abnormality for each type of operational data is determined based on the combination relationship between the difference factor and the offset factor of each operational data. The anomaly level of each operational data point is normalized and then weighted and summed using the corresponding offset factor to obtain the fault level at the current moment.

5. The remote collaborative diagnostic system for grain storage equipment based on AR technology according to claim 4, characterized in that, The specific steps for calculating the difference factor between the recent state vector and the previous state vector for each running data are as follows: Extract time-series data of each operating parameter from the analysis period to obtain recent and earlier subsequences; Calculate the average rate of change and average acceleration of each operating parameter in the recent subsequence, and calculate the reference baseline value, reference fluctuation value and reference average rate of change of each operating parameter in the previous subsequence; The deviation value of the rate of change is obtained by subtracting the corresponding reference average rate of change from the recent average rate of change of each operating parameter. The absolute value of the recent average acceleration of the corresponding operating parameter is multiplied by the preset penalty coefficient to obtain the acceleration penalty value. The absolute value of the deviation value of the rate of change of the corresponding operating parameter is added to the acceleration penalty value and then divided by the product of the reference fluctuation value and the preset safety factor to obtain the difference factor of the corresponding operating parameter.

6. The remote collaborative diagnostic system for grain storage equipment based on AR technology according to claim 5, characterized in that, The specific steps for calculating the offset factor between the recent state vector and the previous state vector for each running data are as follows: Obtain the current value of each running data from the state vector at the current sampling time; The current offset of the running parameter is obtained by subtracting the reference baseline value of the corresponding previous subsequence from the current value of each running data. Divide the current offset by the reference fluctuation value and take the absolute value to obtain the offset multiple of the operating parameter.

7. A remote collaborative diagnostic system for grain storage equipment based on AR technology according to claim 6, characterized in that, The specific steps for determining the degree of anomaly of each type of operational data based on the combination relationship between the difference factor and the offset factor of each operational data are as follows: A first difference threshold, a second difference threshold, a first offset threshold, a second offset threshold, and a third offset threshold are preset for each operating parameter, wherein the first difference threshold is greater than the second difference threshold, and the first offset threshold, the second offset threshold, and the third offset threshold decrease sequentially; If the difference factor of each operating parameter is greater than the corresponding first difference threshold and the offset multiple of each operating parameter is greater than the corresponding first offset threshold, then the abnormality level is assigned the first level value. If the difference factor of each operating parameter is between the corresponding first difference threshold and the second difference threshold, and the offset multiple of each operating parameter is greater than the corresponding second offset threshold, then the abnormality level is assigned the second level value. If the difference factor of each operating parameter is less than the corresponding second difference threshold and the offset multiple of each operating parameter is greater than the corresponding third offset threshold, then the abnormality level is assigned the third level value. Otherwise, assign the value to the fourth level, with all levels arranged from highest to lowest.

8. The remote collaborative diagnostic system for grain storage equipment based on AR technology according to claim 7, characterized in that, The specific steps for normalizing the anomaly levels of each set of operational data and then weighting and summing them using the corresponding offset factors to obtain the fault level at the current moment are as follows: Obtain the sum of the abnormality levels of each running data, and divide the abnormality level of each running data by the sum to obtain the weighting factor of each running data. Multiply the weight factor of each running data by the corresponding offset factor to obtain the weighted offset of each running data. The fault level at the current moment is obtained by summing the weighted offsets of all running data.

9. A remote collaborative diagnostic system for grain storage equipment based on AR technology according to claim 8, characterized in that, The specific steps for delineating the fault area based on the spatial coordinate data corresponding to the fault time are as follows: For an image frame marked as a fault moment, obtain the spatial point set corresponding to that image frame; Divide the spatial point set into several depth layers at equal intervals along the depth direction, calculate the center point of each layer, and detect abrupt transition layers where the distance between the center points of adjacent layers is abnormal. Based on the location of the jump layer, a set of candidate fault layers is selected. From the candidate fault layers, a high-density layer with a feature point density higher than the average density is selected. All feature points in the high-density layer constitute a high-density feature point set. Outlier feature points are removed from the high-density feature point set to obtain the final fault feature point set; The weighted boundary center and boundary radius are calculated based on the spatial coordinates of each feature point in the final fault feature point set, and the fault area boundary box is determined based on the weighted boundary center and boundary radius.

10. A remote collaborative diagnostic system for grain storage equipment based on AR technology according to claim 9, characterized in that, The specific steps for calculating the weighted boundary center and boundary radius based on the spatial coordinates of each feature point in the final fault feature point set, and determining the fault region boundary box based on the weighted boundary center and boundary radius are as follows: The feature point density value of the depth layer where each feature point is located in the final fault feature point set is used as the weighting coefficient of the feature point; Calculate the total weighted coordinates of all feature points in the horizontal, vertical and depth directions respectively, and then divide them by the sum of the weighting coefficients of all feature points to obtain the coordinates of the center of the weighted boundary in the three directions. Calculate the Euclidean distance from each feature point to the center of the weighted boundary, and take the maximum value as the boundary radius; A spherical boundary is determined by using the weighted boundary center as the center and the boundary radius as the radius. Then, a three-dimensional cube region is constructed with the weighted boundary center as the center and the boundary radius as half the side length. This yields the six boundary coordinate values ​​of the fault region boundary box. The six boundary coordinate values ​​refer to the minimum and maximum coordinate values ​​in the horizontal direction, the minimum and maximum coordinate values ​​in the vertical direction, and the minimum and maximum coordinate values ​​in the depth direction.

Images