A data cleaning method based on physical asset identification

Abstract

The application relates to the field of data processing and discloses a data cleaning method based on physical asset identification, which comprises the following steps: acquiring a physical asset identification data stream; mapping the physical asset identification data stream into a sparse matrix composed of multiple nodes; acquiring the sparse density of the sparse matrix and generating scanning parameters based on the sparse density; performing traversal scanning on the sparse matrix based on the scanning parameters to calculate the difference between adjacent nodes; when the difference is greater than a preset numerical threshold, determining the node with a later timestamp in the adjacent nodes generating the difference as a specific node and acquiring a feature sub-block within a first range of the specific node; and performing local matrix operation on the feature sub-block to calculate cleaning data. Through the technical scheme, invalid calculation resource consumption of a large number of unchanged nodes in global traversal calculation is avoided.

Description

Technical Field

[0001] This application relates to the field of data processing, and in particular to a data cleaning method based on physical asset identification. Background Technology

[0002] In the data processing scenarios of asset management systems, it is usually necessary to clean asset data from different sources to ensure the accuracy of subsequent asset value calculation and monitoring status display as much as possible.

[0003] When processing asset location data reported by terminal devices and status data recorded by business terminals, existing technologies often adopt the approach of receiving all data streams within a fixed time period, importing them into a global data cleaning pool, relying on a unified algorithm to perform global traversal calculations and format cleaning processing on the entire dataset, and then storing it in the database.

[0004] The inventors discovered that the above-mentioned global traversal calculation method consumes a large amount of computing resources on nodes where the data characteristics have not changed significantly when faced with a concurrent inflow of massive heterogeneous asset data, resulting in frequent memory calls and blocking of the calculation thread; the default filling process for missing data does not take into account the business state meaning corresponding to the missing data, which can easily lead to the cleaned data deviating from the actual business logic in value assessment.

[0005] Solving this technical problem is a technical challenge that needs to be overcome by those skilled in the art. Summary of the Invention

[0006] This application provides a data cleaning method based on physical asset identification to at least partially solve the above-mentioned technical problems.

[0007] To achieve the above objectives, this application provides a data cleaning method based on physical asset identification, including: Obtain the physical asset identification data stream; The physical asset identification data stream is mapped as a sparse matrix consisting of multiple nodes; Obtain the sparse density of the sparse matrix and generate scanning parameters based on the sparse density; Based on the scanning parameters, a traversal scan is performed on the sparse matrix to calculate the difference between adjacent nodes; When the difference is greater than a preset numerical threshold, the node with the later timestamp among the adjacent nodes that generated the difference is identified as a specific node and the feature sub-blocks within the first range of the specific node are obtained. Local matrix operations are performed on the feature sub-blocks to calculate the cleaning data; The cleaned data is input into the calculation engine to perform asset value calculation.

[0008] In this embodiment of the application, the above technical solution triggers feature extraction and local matrix operation in the local area only when the difference between adjacent nodes exceeds the threshold. This concentrates the cleaning calculation on the spatiotemporal nodes where the asset status has changed significantly, avoiding the ineffective consumption of computing resources on a large number of unchanged nodes by global traversal calculation. Through the elimination of zero-value nodes and compensation matrix operation, the cleaned data retains the business status information corresponding to the missing data, avoiding the measurement deviation caused by the default filling of missing data.

[0009] Other features and advantages of this application will be described in detail in the following detailed description section. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a flowchart illustrating the steps of a data cleaning method based on physical asset identification provided in an exemplary embodiment of this application. Detailed Implementation

[0012] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the protection scope of this application.

[0013] This application provides a data cleaning method based on physical asset identification. Please refer to [link / reference]. Figure 1 The data cleaning method based on physical asset identification provided in this application includes the following steps: S101. Obtain the physical asset identifier data stream. Specifically, continuously receive real-time data packets carrying physical asset identifiers from terminal devices, business management systems, or third-party data interfaces bound to physical assets, and arrange these data packets into a continuous data stream according to the receiving time sequence; the physical asset identifier here refers to the encoded string used to distinguish and identify a specific physical asset; the physical asset identifier data stream refers to the data queue formed by arranging the above data packets in time sequence.

[0014] S102. Map the physical asset identification data stream into a sparse matrix composed of multiple nodes. Specifically, in industrial sites, data reception is discontinuous due to equipment hibernation or communication packet loss. The physical asset identification data stream is parsed to obtain the spatial coordinate field and timestamp field in each data packet. Based on the preset spatial partitioning interval and time sampling interval, a three-dimensional coordinate system composed of a spatial horizontal axis, a spatial vertical axis, and a time axis is constructed. The business value fields in the data packets are filled into the corresponding position space in the three-dimensional coordinate system according to their associated spatial coordinates and timestamps to generate a three-dimensional data matrix. Zero-value identifiers are filled into the blank positions in the three-dimensional data matrix where no data packets have been received as placeholder nodes, resulting in a sparse matrix containing zero-value nodes and non-zero-value nodes. The sparse matrix refers to a numerical array with a three-dimensional structure of a spatial horizontal axis, a spatial vertical axis, and a time axis, where most nodes are zero-value identifiers and only a few nodes carry actual business values.

[0015] S103. Obtain the sparse density of the sparse matrix and generate scanning parameters based on the sparse density. Specifically, the ratio of the number of non-zero nodes in the sparse matrix to the total capacity of the matrix nodes is calculated to obtain the global sparse density. This global sparse density is substituted into a preset mapping function, and the corresponding scan step size and window size are calculated based on the density value. The scan step size and window size are combined into scanning parameters. When the global sparse density is low, the distribution of non-zero nodes is sparse, so the scan step size is increased accordingly to quickly cross blank areas, and the window size is reduced accordingly to avoid introducing too many zero-value nodes. When the global sparse density is high, the data distribution is dense, so the scan step size is reduced accordingly to perform fine scanning, and the window size is increased accordingly to cover a wider neighborhood range.

[0016] S104. Perform a traversal scan of the sparse matrix based on the scan parameters to calculate the difference between adjacent nodes. Specifically, the scan is performed in a jump-alternating manner along the time axis and the horizontal and vertical axes of space according to the scan step size, and the values ​​in the sparse matrix are read node by node; for every two adjacent nodes on the scan path, the absolute value of the difference between the value of the next node and the value of the previous node is calculated as the difference between the two adjacent nodes; the magnitude of the difference corresponds to the change in the asset status value between adjacent sampling positions or times.

[0017] S105. When the difference is greater than a preset numerical threshold, the node with the later timestamp among the adjacent nodes that generated the difference is identified as a specific node, and feature sub-blocks within the first range of the specific node are obtained. Specifically, the calculated difference is compared with the preset numerical threshold one by one; when the difference between a pair of adjacent nodes exceeds the numerical threshold, it indicates that an abnormal change in the asset status has occurred at the corresponding time point or spatial location. At this time, the node with the later timestamp among the pair of adjacent nodes is recorded as a specific node; with the specific node as the reference center, an enclosing area is defined on the time axis and spatial axis of the sparse matrix according to the window size as the first range, and all node values ​​within the enclosing area are extracted to form feature sub-blocks; the specific node mentioned here refers to the key node position that is determined to be an abnormal change in the asset status during the traversal scan.

[0018] S106. Perform local matrix operations on the feature sub-blocks to calculate the cleaned data. Specifically, instead of performing overall smoothing on the global matrix, directional matrix operations are only performed on the local feature sub-blocks containing abrupt changes. Rows and columns containing zero-value markers in the feature sub-blocks are removed, and non-zero nodes are retained to form non-zero feature sub-blocks. Dimension matching and numerical compensation operations are performed on the non-zero feature sub-blocks using a preset compensation matrix, and the cleaned data is calculated and output using matrix inner product. The cleaned data removes invalid zero-value information from the original data stream and retains the business numerical features that trigger abnormal jumps through local compensation, corresponding to the asset status within the spatiotemporal range.

[0019] The above technical solution maps the physical asset identification data stream into a three-dimensional sparse matrix. Based on the sparse density, scanning parameters are generated to traverse and scan the matrix. Feature extraction and local matrix operations are triggered only when the difference between adjacent nodes exceeds a threshold. The cleaning calculation is concentrated on the spatiotemporal nodes where the asset status has changed significantly, avoiding the ineffective consumption of computing resources for a large number of unchanged nodes by global traversal calculation. Through the removal of zero-value nodes and compensation matrix operations, the cleaned data retains the business status information corresponding to the missing data, avoiding the measurement deviation caused by the default filling of missing data.

[0020] In some embodiments, mapping the physical asset identification data stream as a sparse matrix consisting of multiple nodes includes: Step S201: Parse the data packet format in the physical asset identification data stream to obtain the corresponding spatial coordinate field and timestamp field. Specifically, according to the configured communication protocol, unpack each frame of data packet in the physical asset identification data stream to separate the packet header and packet payload; traverse the field identifiers in the packet payload to identify the location identifier and timing identifier; read the payload data matching the location identifier as the spatial coordinate field, and read the payload data matching the timing identifier as the timestamp field; the aforementioned spatial coordinate field refers to the horizontal and vertical coordinate values ​​describing the physical location of the asset; the timestamp field refers to the time stamp value recording the time when the data packet was generated.

[0021] Step S202: Obtain the configured spatial partitioning interval and temporal sampling interval. Specifically, read the spatial partitioning interval and temporal sampling interval from the preset parameter storage area; the spatial partitioning interval defines the physical distance between adjacent mesh nodes in the three-dimensional coordinate system on the spatial horizontal axis and spatial vertical axis; the temporal sampling interval defines the time span between adjacent time layers.

[0022] Step S203: Based on the spatial partitioning interval and the time sampling interval, construct a three-dimensional coordinate system consisting of a spatial horizontal axis, a spatial vertical axis, and a time axis. Specifically, grid lines are divided along the spatial horizontal and vertical axes using the spatial partitioning interval as the grid unit, and time layers are divided along the time axis using the time sampling interval as the layer unit. The three together constitute a three-dimensional orthogonal coordinate system; each intersection point of this three-dimensional coordinate system is a node position, corresponding to a specific spatial coordinate and time.

[0023] Step S204: Obtain the business value field from the physical asset identification data stream and fill it into the corresponding position space in the three-dimensional coordinate system according to the spatial coordinate field and timestamp field associated with the business value field, generating a three-dimensional data matrix. Specifically, read the business value field from each frame of the physical asset identification data stream, locate the target node in the three-dimensional coordinate system according to the spatial coordinate field and timestamp field of the message, and write the value of the business value field into the matrix cell corresponding to the target node; after traversing and processing all data messages, all matrix cells with written values ​​together constitute the three-dimensional data matrix.

[0024] Step S205: Fill blank nodes in the three-dimensional data matrix corresponding to missing data with zero-value markers to generate a sparse matrix. Specifically, scan all node positions in the three-dimensional data matrix. For blank nodes that did not receive data packets due to device sleep or communication packet loss, uniformly fill in zero-value markers as placeholders. After filling, the three-dimensional data matrix is ​​transformed into a sparse matrix. The zero-value markers are identified and skipped as invalid data in subsequent scanning and calculation, so that computing resources are concentrated on the non-zero node areas that carry actual business values.

[0025] In some embodiments, obtaining the sparse density of the sparse matrix and generating scan parameters based on the sparse density includes: Step S301: Calculate the global sparse density by calculating the ratio of the total number of non-zero nodes to the total capacity of the matrix nodes. Specifically, traverse all nodes of the sparse matrix, count the cumulative number of non-zero nodes, and divide the total number of non-zero nodes by the total capacity of the matrix nodes to obtain the global sparse density; the total capacity of the matrix nodes mentioned above is the product of the number of nodes on the horizontal axis, the number of nodes on the vertical axis, and the number of time axis layers.

[0026] Step S302: Substitute the global sparse density into the mapping function to calculate the scan step size and window size along the time and spatial axes. Specifically, the mapping function includes a step size mapping subfunction and a window mapping subfunction; the step size mapping subfunction outputs a decreasing scan step size value as the global sparse density increases; the aforementioned window mapping subfunction outputs an increasing window size value as the global sparse density increases; the two subfunctions respond to density changes respectively, jointly determining the coarseness or fineness of the scan under the current sparse conditions.

[0027] The step size mapping sub-function can adopt a nonlinear inverse proportional model, for example: The window mapping subfunction uses an exponential adaptive model, for example: ;in, The global sparse density is... To calculate the scan step size, For window size, As the reference step size constant, As a sparsity sensitivity modulator, As the reference window constant, This is the preset upper limit of the window based on the maximum memory throughput limit of the terminal processing device.

[0028] Step S303: Combine the scan step size and window size to generate scan parameters. Specifically, the scan step size value and window size value, expressed in units of the number of nodes, are encapsulated into a set of parameter structures for subsequent traversal scan steps to call.

[0029] The above technical solution calculates the corresponding scanning step size and window size based on the global sparse density of the sparse matrix, so that the scanning parameters match the data distribution characteristics. It can perform fine scanning in dense data areas and quickly skip in sparse data areas, thus balancing scanning efficiency and anomaly detection accuracy.

[0030] In some embodiments, identifying the node with the later timestamp among the neighboring nodes that generate the difference as a specific node includes: Step S401: Based on the scan step size, alternately traverse and scan the node values ​​of the sparse matrix along the time axis and the horizontal and vertical axes of space. Specifically, using the scan step size as the unit of advancement, first scan a row of nodes sequentially in the horizontal direction of space, then scan a new row in the vertical direction of space. After completing the scanning of all nodes in one spatial layer, proceed along the time axis to the next time layer, and so on, alternating to cover all nodes of the sparse matrix.

[0031] Step S402: Calculate the numerical change between two adjacent nodes as the difference. Specifically, on the scan path, subtract the values ​​of two adjacent nodes and take the absolute value to obtain the numerical change between the pair of nodes, which is recorded as the difference. Adjacent nodes refer to two nodes that are visited consecutively on the scan path, and they are adjacent in space or time.

[0032] Step S403: When the difference is greater than a preset numerical threshold, the node with the later timestamp that generated the difference is marked as a specific node. Specifically, the obtained difference is compared with the preset numerical threshold. If the difference exceeds the numerical threshold, it is determined that there is an abnormal change in the asset status at the spatiotemporal location corresponding to the adjacent node. At this time, the node with the later timestamp among the adjacent nodes is selected as a specific node and marked, and its node coordinates are stored in the specific node list.

[0033] Using the above technical solution, the scanning step size is used as the advancing unit to traverse the three-dimensional sparse matrix alternately. The specific node is located by the value change of the adjacent node exceeding the threshold, thus locating the position of abnormal asset state jump.

[0034] In some embodiments, the process of performing a traversal scan of the sparse matrix based on scan parameters to calculate the difference between adjacent nodes further includes: Step S501: After completing each scan step, calculate the ratio of the number of non-zero nodes in the currently scanned region to the total number of scanned nodes, and use this ratio as the local sparsity density. Specifically, during the traversal scan, after each scan step covering a node interval, calculate the ratio of the number of non-zero nodes in that interval to the total number of scanned nodes to obtain a local sparsity density value; this local sparsity density corresponds to the density of the actual data in the currently scanned local region.

[0035] Step S502: Calculate the density deviation between the local sparse density and the global sparse density. Specifically, subtract the global sparse density from the obtained local sparse density and take the absolute value to obtain the density deviation; the magnitude of the density deviation corresponds to the deviation between the sparsity of the current scanned region and the average sparsity of the entire matrix.

[0036] Step S503: When the absolute value of the density deviation is greater than the preset deviation threshold, if the local sparse density is lower than the global sparse density, the scanning step size is reduced and the window size is increased; if the local sparse density is higher than the global sparse density, the scanning step size is increased and the window size is reduced. Specifically, the absolute value of the density deviation is compared with the preset deviation threshold; if it exceeds the deviation threshold and the local sparse density is lower than the global sparse density, it means that the data in the current region is sparser than the overall data. If the original scanning step size is maintained, it may directly cross potential abnormal nodes. At this time, the scanning step size is reduced to make the scan more detailed, and the window size is increased to expand the feature extraction range; if the local sparse density is higher than the global sparse density, it means that the data in the current region is denser than the overall data. At this time, the scanning step size is increased to reduce the scanning time, and the window size is reduced to focus on the core data of the dense region.

[0037] Step S504: Continue to perform subsequent traversals using the adjusted scan step size and window size. Specifically, update the scan parameters with the adjusted scan step size and window size, and perform subsequent traversal scans and feature sub-block extractions using the updated parameters until the density deviation exceeds the threshold again, at which point adjustments are made again.

[0038] The above technical solution adjusts the scanning step size and window size based on the deviation between local and global sparse density during the traversal scanning process. This allows the scanning strategy to be optimized according to the local characteristics of the data distribution, strengthening the scanning in sparse data areas to prevent missed detections, and speeding up the scanning in dense data areas to save computation time.

[0039] In some embodiments, when the difference is greater than a preset numerical threshold, the node whose timestamp generated the difference is later is marked as a specific node, including: Step S601: Obtain the trend of change corresponding to the difference. Specifically, while calculating the difference between adjacent nodes, record the positive or negative sign of the difference between the value of the next node and the value of the previous node; when the value of the next node is greater than the value of the previous node, the trend is increasing, and vice versa.

[0040] Step S602: When the trend of change is increasing, mark the node with the later timestamp as a first-type specific node. Specifically, when the difference is positive, that is, when the value of the later node is higher than the value of the previous node, mark the node with the later timestamp in the pair of nodes as a first-type specific node; the first-type specific node corresponds to the abnormal jump position of the asset status value increase.

[0041] Step S603: When the trend of change is decreasing, mark the node with the later timestamp as a second type of specific node. Specifically, when the difference is negative, that is, when the value of the later node is lower than the value of the previous node, mark the node with the later timestamp in the pair of nodes as a second type of specific node; the second type of specific node corresponds to the abnormal jump position of the decrease in the asset status value.

[0042] Step S604: Combine the first type of specific nodes and the second type of specific nodes into a single specific node list. Specifically, the nodes marked in the two categories are grouped into a single specific node list. In subsequent feature sub-block extraction and local matrix operations, the distribution characteristics of the two types of nodes are processed differently to identify asset status anomalies in different directions.

[0043] The above technical solution categorizes specific nodes into rising and falling categories based on their changing trends, distinguishing between abnormal areas where asset status values ​​increase and abnormal areas where values ​​decrease.

[0044] In some embodiments, obtaining feature sub-blocks within a first range of a specific node includes: Step S701: Delineate a bounding region in the sparse matrix based on the window size, centered on a specific node, as the first range. Specifically, using the three-dimensional coordinates of the identified specific node as the geometric center, expand outwards by a specified number of nodes in the three directions of the horizontal axis, vertical axis, and time axis according to the currently effective window size, thus delineating a three-dimensional bounding region, which is the first range.

[0045] Step S702: Extract all node values ​​within the enclosing region to form feature sub-blocks. Specifically, according to the start and end index range of the enclosing region in the three axes, extract all node values ​​within that range from the sparse matrix and organize them into a local sub-matrix according to the original spatial arrangement, which is the feature sub-block.

[0046] Step S703: Delineate an enclosing region in the sparse matrix based on the window size, centered on a specific node, as the first range. This includes: obtaining the timestamp corresponding to the business state in the physical asset identification data stream and the timestamp associated with the current spatial coordinates, and obtaining the time difference; converting the time difference into a time compensation coefficient based on a time mapping function; extending the window size along the time axis based on the time compensation coefficient; and constructing an asymmetric enclosing region with unequal spans along the time axis and the spatial axis based on the extended window size, as the first range. Specifically, changes in the state of physical assets exhibit a transmission lag effect in the time dimension. Specifically, the asset state change time recorded by the business end often lags behind the physical state jump time collected by the terminal sensors. To address this time difference, the time difference between the timestamp of the business state record and the sampling timestamp associated with the current spatial coordinates is obtained. This time difference is input into a preset time mapping function to calculate a time compensation coefficient. A time compensation coefficient greater than one indicates the additional window extension required along the time axis due to the business state transmission delay. The original window size is multiplied by this time compensation coefficient and then superimposed onto the original time axis length to obtain the extended window size. The asymmetric enclosing region constructed based on the extended window size has a larger span along the time axis than along the spatial axis, thus acquiring associated data distributed across a wider time interval due to the business transmission delay. The aforementioned time mapping function refers to the calculation rule for mapping the time difference to the time compensation coefficient; the larger the time difference, the larger the compensation coefficient, but it tends towards a preset upper limit to prevent excessive window expansion.

[0047] The time mapping function can be approximated using a restricted exponential model, and its mathematical expression can be: ;in, This is the time compensation coefficient. The time difference is... The maximum expansion factor constant, determined by the rated operating cycle of the physical equipment, is set to approximate the upper limit value. The time delay damping attenuation factor; the time delay damping attenuation factor The value depends on the average latency of the network communication environment in which the physical asset is located.

[0048] The above technical solution obtains feature sub-blocks based on the window size with a specific node as the center. When there is a data reporting delay, the window is extended along the time axis by the time compensation coefficient, so that the feature sub-blocks cover a wider range in the time dimension to obtain data that arrives late, thus avoiding the omission of abnormal data due to reporting delay.

[0049] In some embodiments, after constructing an asymmetric enclosing region with unequal spans along the time and space axes, the method further includes: Step S801: Detect whether the asymmetric enclosing region touches the boundary of the sparse matrix; when the boundary of the asymmetric enclosing region on any time axis or spatial axis exceeds the effective range of the sparse matrix, it is determined to be a boundary out-of-bounds. Specifically, check whether the start index and end index of the asymmetric enclosing region in the three directions of the spatial horizontal axis, spatial vertical axis, and time axis exceed the index range defined by the sparse matrix; if the start index in any direction is less than zero or the end index is greater than the maximum index value in that direction, it is determined that a boundary out-of-bounds has occurred in that direction.

[0050] Step S802: For the axis where the boundary exceeds the limit, using the boundary of the sparse matrix as the axis of symmetry, mirror the node values ​​corresponding to the width inside the boundary to the outside of the boundary to generate virtual extended nodes. Specifically, for the axis where the boundary exceeds the limit, using the boundary of the sparse matrix on that axis as the plane of symmetry, mirror the node value sequence adjacent to the inside of the boundary with the same width exceeding the limit to the outside of the boundary to fill the nodes required for the exceeding region; these mirrored copies generate virtual extended nodes, whose values ​​are taken from the real data inside the matrix but whose positions are outside the effective range of the matrix.

[0051] Step S803: Fill the virtual extension nodes into the out-of-bounds portion of the asymmetric enclosing region to maintain the integrity of the feature sub-block's boundary range. Specifically, fill the generated virtual extension nodes into the out-of-bounds portion of the asymmetric enclosing region to ensure that the feature sub-block maintains its complete boundary range without changing the original window size requirements, thus avoiding feature sub-block size reduction or data loss due to boundary overflow.

[0052] Step S804: When performing local matrix operations on feature sub-blocks, configure virtual extended nodes with lower computational weights than actual nodes. Specifically, during subsequent local matrix operations, for virtual extended nodes contained in feature sub-blocks, set their weight coefficients in the matrix operations to be lower than the weight values ​​of the actual nodes inside the matrix, in order to reduce the influence of mirrored data on the computation results.

[0053] The above technical solution generates virtual extended nodes to fill the out-of-bounds area by mirroring the data inside the boundary when the feature sub-block window touches the matrix boundary. This allows the feature sub-block to maintain its full size and participate in the calculation even in boundary scenarios, avoiding incomplete local matrix calculation data caused by boundary truncation.

[0054] In some embodiments, local matrix operations are performed on feature sub-blocks to calculate cleaned data, including: Step S901: Remove rows and columns containing zero-value markers from the feature sub-blocks to obtain non-zero feature sub-blocks and record the corresponding coordinate retention indices. Specifically, traverse the rows and columns of the feature sub-blocks, identify rows and columns containing only zero-value markers, delete these rows and columns from the feature sub-blocks, and retain rows and columns containing at least one non-zero value to form non-zero feature sub-blocks; record the coordinate indices of the retained rows and columns in the original feature sub-blocks as coordinate retention indices.

[0055] Step S902: Generate a compensation matrix; based on the coordinate-preserving index, truncate the dimensions of the compensation matrix to generate a compensation sub-matrix with the same size as the non-zero feature sub-block. Specifically, the compensation matrix is ​​a numerical correction matrix pre-constructed based on the prior distribution of asset data features, and its size covers the maximum possible dimension of the entire feature sub-block; according to the coordinate-preserving index recorded in step S901, the values ​​of the corresponding rows and columns are truncated from the compensation matrix to extract a compensation sub-matrix with the exact same number of rows and columns as the non-zero feature sub-block.

[0056] Specifically, the construction logic of the compensation matrix is ​​as follows: Establish the dimensional dimensions of the compensation matrix. Initialize a three-dimensional base compensation matrix, whose spatial horizontal and vertical dimensions are equal to the preset maximum window size, and whose time axis dimension is equal to the average duration of a single typical abnormal state evolution in the historical dataset. Assign prior values ​​to the elements within the matrix; extract state transition samples of this type of physical asset from the historical database within a set period, and calculate the average gradient of state characteristics relative to a specific central node, i.e., the point of anomaly occurrence, at different spatiotemporal relative distances; use the reciprocal or normalized value of the average gradient as prior weights, and sequentially write them into the corresponding spatiotemporal node positions of the base compensation matrix; for example, the closer the matrix element is to the central node, the higher its compensation weight value; the farther the distance, the lower the weight value decreases according to a normal distribution curve.

[0057] Step S903: Multiply the non-zero feature sub-blocks with the compensation sub-matrix using matrix inner product to calculate the cleaned data. Specifically, perform matrix inner product operation on the non-zero feature sub-blocks and the compensation sub-matrix, that is, multiply each element in the non-zero feature sub-blocks with the corresponding element in the compensation sub-matrix one by one and then sum them to obtain a scalar value as the cleaned data. This cleaned data removes zero-value noise from the original feature sub-blocks and corrects the numerical deviation caused by the sparse distribution of data through the compensation sub-matrix, corresponding to the asset status characteristics in the spatiotemporal region.

[0058] By using the above technical solution, after removing the zero-value rows and columns in the feature sub-blocks and then performing a local inner product operation with the compensation matrix, the interference of invalid zero values ​​on the calculation results is eliminated, and the sparse region is numerically corrected by the compensation matrix. This allows the cleaned data to retain core information while improving numerical accuracy.

[0059] In some embodiments, filling blank nodes in a three-dimensional data matrix corresponding to missing data with zero-value identifiers to generate a sparse matrix includes: Step S1001: Obtain the first coordinate position corresponding to the missing data in the three-dimensional data matrix. Specifically, after generating the three-dimensional data matrix, traverse all node coordinates of the matrix and record the coordinates of nodes that have not received data packets as the first coordinate position, i.e., the node positions where data is missing.

[0060] Step S1002: Obtain the asset status records corresponding to the time period of the first coordinate position in the physical asset identification data stream. Combine the configured asset ledger attributes with the equipment depreciation coefficient and perform a weighted calculation to generate a business bias weight. Specifically, for each first coordinate position, query the asset status records in the physical asset identification data stream within the corresponding time period; read the ledger attributes of the asset, such as asset type, service life, and maintenance records, from the asset ledger database; and perform a weighted calculation on the asset status records based on the preset equipment depreciation coefficient to generate a business bias weight. The business bias weight represents the degree of numerical bias inferred from the asset's business attributes and physical aging patterns at the position corresponding to the missing data, ensuring that the subsequently filled cleaned data is not only mathematically continuous but also conforms to the expected physical wear and tear of the equipment within the corresponding service life.

[0061] Step S1003: Construct a state mapping matrix that is parallel to the three-dimensional data matrix in the spatial and temporal axes. Specifically, create a state mapping matrix with the same dimensions as the three-dimensional data matrix. The number of nodes in the three directions of the spatial horizontal axis, spatial vertical axis, and time axis is the same as that in the three-dimensional data matrix, and it is initialized to all zeros.

[0062] Step S1004: Write the business bias weight into the mapping node corresponding to the first coordinate position in the state mapping matrix. Specifically, fill the coordinate node with the same first coordinate position in the state mapping matrix with the business bias weight calculated in step S1002, so that the node in the state mapping matrix at the data missing position carries the bias information of business semantics.

[0063] Step S1005: Fill the blank nodes in the three-dimensional data matrix with zero-value markers to generate a sparse matrix. Specifically, following the method in step S205, fill the blank nodes in the three-dimensional data matrix with zero-value markers to generate a sparse matrix.

[0064] Step S1006: After calculating the cleaned data through local matrix operations, the business bias weights at the corresponding positions in the state mapping matrix are obtained and weighted together with the cleaned data to update the cleaned data. Specifically, after the cleaned data is output through local matrix operations, the business bias weights at the corresponding positions in the state mapping matrix are read according to the current specific node position. The cleaned data and the business bias weights are then weighted and fused to obtain the updated cleaned data. The cleaned data corrected by the business bias weights is numerically closer to the actual business logic, avoiding deviations caused by purely mathematical calculations.

[0065] Through the above technical solution, an additional state mapping matrix is ​​constructed during the zero-value filling process of the sparse matrix to record the business bias weights of the missing positions. After the cleaned data calculation is completed, the cleaned data is corrected with the business bias weights, so that the cleaned data incorporates business semantic information such as asset ledger attributes and equipment attenuation coefficients when filling data gaps. This solves the problem that the calculation results deviate from the actual business logic caused by simple zero-value filling.

[0066] In some embodiments, generating a compensation matrix includes: Step S1101: Obtain the proportional difference value between the extended length of the asymmetric enclosing region along the time axis and the spatial axis. Specifically, calculate the ratio of the actual length of the asymmetric enclosing region along the time axis to the original window length along the spatial axis to obtain the proportional difference value; this proportional difference value represents the degree of asymmetry between the time dimension and the spatial dimension caused by time compensation.

[0067] Step S1102: Obtain the basic compensation matrix; based on the proportional difference value, use a matrix interpolation algorithm to perform dimension expansion and filling on the basic compensation matrix along the corresponding time axis to generate an aligned compensation matrix, which serves as the compensation matrix. Specifically, the basic compensation matrix is ​​a benchmark compensation matrix pre-constructed based on the typical distribution of asset data features, and its dimension corresponds to the spatial axial dimension of the asymmetric enclosed region; based on the proportional difference value, a matrix interpolation algorithm is used to interpolate and expand the dimension of the basic compensation matrix along the time axis, that is, inserting new compensation layers between the original time layers, so that the time dimension of the compensation matrix is ​​aligned with the time dimension of the feature sub-blocks after time compensation, generating an aligned compensation matrix as the compensation matrix used in practice.

[0068] The above technical solution aligns the basic compensation matrix according to the proportional difference between the feature sub-blocks in the time axis and the spatial axis, so that the time dimension of the compensation matrix matches the actual size of the asymmetric enclosed region, and the compensation operation covers the newly added node region in the time dimension due to the delay compensation.

[0069] In some embodiments, after obtaining the feature sub-blocks, the method further includes: Step S1201: Calculate the distribution density of the first type of specific nodes and the second type of specific nodes within the feature sub-block. Specifically, after obtaining the feature sub-block, calculate the number and distribution position of the first type of specific nodes and the number and distribution position of the second type of specific nodes within the enclosing area corresponding to the feature sub-block, and calculate the distribution density of the two types of nodes within the volume of the enclosing area.

[0070] Step S1202: Calculate the local fluctuation entropy based on the distribution density, and use the local fluctuation entropy to correct the preset numerical threshold to generate an adaptive threshold. Specifically, substitute the distribution densities of the first type of specific nodes and the second type of specific nodes into the information entropy calculation formula to obtain the local fluctuation entropy; the local fluctuation entropy represents the degree of disorder in the distribution of abnormal nodes within the feature sub-block, and the higher the entropy value, the higher the degree of mixing between the two types of nodes and the more complex the abnormal pattern; use the local fluctuation entropy as a correction factor to scale and adjust the preset numerical threshold to generate an adjusted threshold suitable for the current local area.

[0071] Step S1203: In subsequent traversal scanning, an adaptive threshold is used to replace the preset numerical threshold to perform the determination step of specific nodes. Local fluctuation entropy is negatively correlated with the adaptive threshold, and the correction magnitude for the first type of specific nodes is greater than that for the second type of specific nodes. Specifically, when the feature sub-block or its adjacent area is encountered during subsequent traversal scanning, the calculated adjusted threshold is used instead of the preset numerical threshold to determine whether the node is a specific node. When the local fluctuation entropy is high, the abnormal pattern in that area is complex; in this case, the numerical threshold is lowered to improve detection sensitivity and avoid missing subtle anomalies. When the local fluctuation entropy is low, the abnormal pattern is simple and clear; in this case, the numerical threshold is maintained or appropriately increased to reduce misjudgments. Simultaneously, since abnormal increases in numerical values ​​usually receive more attention than abnormal decreases in asset value monitoring, the threshold correction magnitude for the first type of specific nodes is greater than that for the second type of specific nodes, making it more sensitive to abnormal upward trends.

[0072] The above technical solution calculates the local fluctuation entropy by utilizing the distribution density of two specific nodes within the feature sub-block and corrects the numerical threshold, allowing the threshold to be adjusted according to the actual fluctuation characteristics of the abnormal region. This improves detection sensitivity in abnormally complex regions and reduces the false judgment rate in abnormally simple regions. At the same time, by providing a larger correction range for rising anomalies, it enhances the risk perception capability for abnormal growth in asset value.

[0073] In some embodiments, after updating the cleaned data, the method further includes: Step S1301: Extract the ratio of the number of non-zero elements participating in the matrix inner product operation in the compensation submatrix to the total number of elements in the compensation submatrix, as the structural integrity factor. Specifically, when performing the matrix inner product operation between the compensation submatrix and the non-zero feature sub-blocks, count the number of non-zero elements actually used in the compensation submatrix, i.e., the number of elements participating in the multiplication operation, and divide it by the total number of elements in the compensation submatrix to obtain the structural integrity factor. The structural integrity factor represents the degree of structural coverage of the compensation matrix when performing the current operation. The closer the factor is to one, the higher the utilization rate of the compensation matrix and the higher the data quality.

[0074] Step S1302: The structural integrity factor and the business bias weight are fused to generate a confidence coefficient. Specifically, the structural integrity factor and the business bias weight obtained in step S1002 are calculated according to a preset fusion formula to generate a confidence coefficient. The confidence coefficient comprehensively represents the credibility of the cleaned data in both data structure and business semantics dimensions. The higher the coefficient, the more reliable the quality of the cleaned data.

[0075] Step S1303: Encapsulate the updated cleaned data and the calculated confidence coefficients into a weighted cleaned data packet. Specifically, the updated cleaned data and the corresponding calculated confidence coefficients are combined into a single data packet structure and transmitted to the asset value calculation engine. In subsequent value calculations, the calculation engine weights the cleaned data based on the calculated confidence coefficients, assigning greater weight to cleaned data with high confidence and reducing the impact of data with low confidence.

[0076] Through the above technical solution, after the cleaned data is updated, the confidence coefficient is generated based on the structural utilization rate and business bias weight of the compensation matrix and output together with the cleaned data. This allows the subsequent asset value calculation to be weighted according to the data quality, reducing the disturbance of the calculation results when the data quality is low, and improving the reliability of the asset value calculation.

[0077] In some embodiments, it also includes: Step S1401: Calculate the spatial distance between each specific node, and group specific nodes whose spatial distance is less than the configured aggregation radius into the same abnormal node cluster. Specifically, after obtaining all specific nodes, calculate the spatial Euclidean distance between each pair of specific nodes one by one, and group all specific nodes whose spatial distance is less than the preset aggregation radius into the same abnormal node cluster; the abnormal node cluster represents a set of multiple abnormal jump nodes that are spatially adjacent, indicating that there is a concentrated asset status anomaly in this spatial area.

[0078] Step S1402: Calculate the ratio of the number of first-type specific nodes to the number of second-type specific nodes in each abnormal node cluster. Specifically, for each abnormal node cluster, count the number of specific nodes marked as first-type and the number of specific nodes marked as second-type, and calculate the ratio between the two.

[0079] Step S1403: When the quantity ratio indicates that the number of the first type of specific nodes is greater than that of the second type of specific nodes, the corresponding abnormal node cluster is marked as an expanding abnormal region; when the quantity ratio indicates that the number of the second type of specific nodes is greater than that of the first type of specific nodes, the corresponding abnormal node cluster is marked as a contracting abnormal region. Specifically, the number of the two types of nodes in each abnormal node cluster is compared; if the proportion of the first type of nodes exceeds that of the second type of nodes, it indicates that the region is mainly characterized by an increase in asset status values, and it is marked as an expanding abnormal region; if the proportion of the second type of nodes exceeds that of the first type of nodes, it indicates that the region is mainly characterized by a decrease in asset status values, and it is marked as a contracting abnormal region.

[0080] Step S1404: For the inflated anomaly region, feature sub-block extraction is performed using the boundary of the smallest bounding rectangle of the anomaly node cluster as the first range, and an upper bound truncation calculation is performed during local matrix operations. Specifically, for the inflated anomaly region, the first range for feature sub-block extraction is the smallest bounding rectangle of the anomaly node cluster in the spatial dimension, without outward expansion; an upper bound truncation is performed on the output result during local matrix operations, that is, the value of the calculation result is limited to not exceeding a preset upper limit value, to prevent the abnormal inflation of asset value from causing the calculation result to be distorted.

[0081] Step S1405: For shrinking anomaly regions, feature sub-block extraction is performed within the first range after expanding the minimum bounding rectangle of the anomaly node cluster outwards by a configured buffer distance. Lower bound compensation calculation is performed during local matrix operations. Specifically, for shrinking anomaly regions, the first range for feature sub-block extraction is defined as the area around the minimum bounding rectangle of the anomaly node cluster, expanded by a preset buffer distance in all directions to cover potential precursor data around the anomaly region. Lower bound compensation is performed on the output results during local matrix operations, i.e., numerical increases are applied to calculation results below a preset lower limit to prevent excessive downgrading of asset values.

[0082] The above technical solution clusters spatially adjacent specific nodes into anomalous node clusters and distinguishes between expanding and contracting anomalous regions based on the proportion of node types. Differentiated feature sub-block extraction ranges and computational correction strategies are adopted for different types of anomalous regions, enabling the cleaning calculation to be customized for the characteristics of anomalous regions. This not only suppresses the amplification effect of expanding anomalies on the measurement results, but also avoids the excessive underestimation of asset value by contracting anomalies.

[0083] In some embodiments, the cleaned data is input into the calculation engine to perform asset value calculation, including: Step S1501: Parse the cleaned data and extract the corresponding asset operating status values. Specifically, after receiving the cleaned data, the calculation engine obtains the asset operating status values ​​from the cleaned data, including status parameters that represent the actual usage status of the asset, such as the asset's usage frequency, runtime, and load intensity.

[0084] Step S1502: Based on the associated physical asset identifier, query the preset asset ledger database to obtain the corresponding initial asset value and benchmark depreciation model. Specifically, using the currently processed physical asset identifier as the query key, retrieve the asset's registration information from the asset ledger database, read the asset's initial purchase value or replacement value as the initial asset value, and obtain the benchmark depreciation model applicable to this asset type; the benchmark depreciation model is a mathematical function describing the decrease in value of this type of asset with usage time and intensity.

[0085] Step S1503: Substitute the asset operating status values ​​into the benchmark depreciation model to calculate the depreciation coefficient. Specifically, the asset operating status values ​​obtained from the cleaned data are used as input variables and substituted into the benchmark depreciation model. The model calculates the depreciation coefficient for the current assessment period based on the asset's actual usage frequency, load intensity, and operating time. The depreciation coefficient represents the proportion of asset value loss under current usage conditions.

[0086] Step S1504: Perform a value reduction calculation on the initial asset value using the depreciation factor and output the asset value assessment result. Specifically, multiply the depreciation factor by the initial asset value to obtain the asset depreciation amount for the current valuation period, subtract the depreciation amount from the initial asset value to obtain the remaining asset value, and output this as the asset value assessment result.

[0087] The above technical solution inputs the cleaned data, which has undergone partial cleaning and business bias weight correction, into the calculation engine. Based on the benchmark depreciation model that matches the asset type, the depreciation coefficient based on the actual usage status is calculated, thus completing the refined calculation of asset value. Compared with the traditional method based on a fixed depreciation rate, this method is closer to the actual depreciation status of the asset.

[0088] All of the above-mentioned optional technical solutions can be combined in any way to form optional embodiments of this application, and will not be described in detail here.

[0089] The above are merely preferred embodiments of this application and are not intended to limit this application in any way. Any simple modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of this application without departing from the scope of the technical solution of this application shall still fall within the scope of the technical solution of this application.

Claims

1. A data cleaning method based on physical asset identification, characterized in that, include: Obtain the physical asset identification data stream; The physical asset identification data stream is mapped as a sparse matrix consisting of multiple nodes; Obtain the sparse density of the sparse matrix and generate scanning parameters based on the sparse density; Based on the scanning parameters, a traversal scan is performed on the sparse matrix to calculate the difference between adjacent nodes; When the difference is greater than a preset numerical threshold, the node with the later timestamp among the adjacent nodes that generated the difference is identified as a specific node and the feature sub-blocks within the first range of the specific node are obtained. Local matrix operations are performed on the feature sub-blocks to calculate the cleaning data.

2. The method according to claim 1, characterized in that, Mapping the physical asset identification data stream into a sparse matrix consisting of multiple nodes includes: Parse the data packet format in the physical asset identification data stream to obtain the corresponding spatial coordinate field and timestamp field; Obtain the configured spatial partitioning interval and time sampling interval; Based on the spatial division interval and the time sampling interval, a three-dimensional coordinate system is constructed, consisting of a spatial horizontal axis, a spatial vertical axis, and a time axis. Obtain the business value field from the physical asset identification data stream and fill the business value field into the location space corresponding to the three-dimensional coordinate system according to the spatial coordinate field and timestamp field associated with the business value field to generate a three-dimensional data matrix; The sparse matrix is ​​generated by filling blank nodes in the three-dimensional data matrix that correspond to missing data with zero-value identifiers.

3. The method according to claim 2, characterized in that, Obtaining the sparse density of the sparse matrix and generating scanning parameters based on the sparse density includes: The global sparse density is calculated by calculating the ratio of the total number of non-zero nodes in the sparse matrix to the total capacity of the matrix nodes, and this ratio is used as the sparse density. Substitute the global sparse density into the mapping function to calculate the scan step size and window size in the time axis and spatial axis; The scanning parameters are generated by combining the scanning step size with the window size.

4. The method according to claim 3, characterized in that, The node with the later timestamp among the adjacent nodes that generated the difference is identified as a specific node, including: Based on the scan step size, the node values ​​of the sparse matrix are scanned alternately along the time axis and the horizontal and vertical axes of space. Calculate the change in value between two adjacent nodes, and use this as the difference; When the difference is greater than the preset numerical threshold, the node whose timestamp generated the difference is later is marked as the specific node.

5. The method according to claim 4, characterized in that, In the process of performing a traversal scan of the sparse matrix based on the scan parameters to calculate the difference between adjacent nodes, the method further includes: After each scan step is completed, the ratio of the number of non-zero nodes in the currently scanned region to the total number of scanned nodes is calculated as the local sparse density. Calculate the density deviation between the local sparse density and the global sparse density; When the absolute value of the density deviation is greater than a preset deviation threshold, if the local sparse density is lower than the global sparse density, the scan step size is reduced and the window size is increased; if the local sparse density is higher than the global sparse density, the scan step size is increased and the window size is reduced.

6. The method according to claim 5, characterized in that, When the difference is greater than the preset numerical threshold, the node whose timestamp generated the difference is later is marked as the specific node, including: Obtain the trend of change corresponding to the difference; When the trend of change is increasing, the nodes with later timestamps are marked as the first type of specific nodes; When the trend of change is decreasing, the nodes with later timestamps are marked as the second type of specific nodes; The first type of specific nodes and the second type of specific nodes are collectively referred to as the specific nodes.

7. The method according to claim 6, characterized in that, Obtaining feature sub-blocks within a first range of the specific node includes: A bounding region is defined in the sparse matrix with the specific node as the center, based on the window size, as the first range; Extract the values ​​of all nodes within the enclosed area to form the feature sub-block; A bounding region is defined within the sparse matrix, centered on the specific node and based on the window size, as the first range, including: Obtain the timestamp corresponding to the business status in the physical asset identification data stream and associate it with the timestamp of the current spatial coordinates to obtain the time difference; The time difference is converted into a time compensation coefficient based on the time mapping function; The length of the window size is extended along the time axis based on the time compensation coefficient; Based on the extended window size, an asymmetric enclosing region with unequal spans along the time axis and the spatial axis is constructed as the first range.

8. The method according to claim 7, characterized in that, After constructing the asymmetric enclosing region with unequal spans along the time and space axes, the method further includes: Detect whether the asymmetric enclosing region touches the boundary of the sparse matrix; when the boundary of the asymmetric enclosing region exceeds the effective range of the sparse matrix on any time axis or spatial axis, it is determined to be a boundary out-of-bounds condition. For the axis of boundary overflow, with the boundary of the sparse matrix as the axis of symmetry, the node values ​​corresponding to the width inside the boundary are mirrored and copied to the outside of the boundary to generate virtual extended nodes. The virtual extended nodes are filled into the out-of-bounds portion of the asymmetric enclosing region, so that the boundary range of the feature sub-block remains intact; When performing local matrix operations on the feature sub-block, the virtual extended node is configured with a lower computational weight than the actual node.

9. The method according to claim 8, characterized in that, Local matrix operations are performed on the feature sub-blocks to calculate the cleaned data, including: The row and column containing the zero-value identifier in the feature sub-block are removed to obtain non-zero feature sub-blocks and the corresponding coordinate retention index is recorded. Generate a compensation matrix; based on the coordinate preservation index, truncate the dimensions of the compensation matrix to generate a compensation submatrix with the same size as the non-zero feature sub-blocks; The cleaned data is calculated by multiplying the non-zero feature sub-blocks with the compensation sub-matrix using a matrix inner product method.

10. The method according to claim 9, characterized in that, The step of filling blank nodes in the three-dimensional data matrix corresponding to missing data with zero-value markers to generate the sparse matrix includes: Obtain the first coordinate position in the three-dimensional data matrix corresponding to the missing data; The asset status records corresponding to the time period at the first coordinate position in the physical asset identification data stream are obtained and weighted by combining the configured asset ledger attributes and equipment attenuation coefficient to generate business bias weights. Construct a state mapping matrix that is parallel to the three-dimensional data matrix in both the spatial and temporal axes; Write the business bias weight into the mapping node in the state mapping matrix corresponding to the first coordinate position; The blank nodes in the three-dimensional data matrix are filled with zero-value markers to generate the sparse matrix; After the local matrix operation is performed to calculate the cleaned data, the business bias weight at the corresponding position in the state mapping matrix is ​​obtained and weighted together with the cleaned data to complete the update of the cleaned data.

Images