A Method and System for Device Health Status Diagnosis Based on Visual and Digital Twin Mapping

Abstract

This invention relates to the field of equipment health monitoring technology, specifically to a method and system for diagnosing equipment health status based on visual and digital twin mapping. Addressing the problems of inaccurate correspondence between visual data and digital twin models, and the lack of automatic deviation calculation between theoretical and actual states in existing technologies, this invention acquires visual image data, sensor monitoring data, and current operating parameters of the target equipment. After time alignment, coordinate unification, and preprocessing, the visual image is mapped to the digital twin model to form the actual observed state. The theoretical health state is determined by combining the current operating parameters and the digital twin model. Visual and sensor deviations are calculated and jointly analyzed to output the health status diagnosis result. This method is applicable to the diagnosis of industrial equipment operating status and anomaly identification.

Description

Technical Field

[0001] This invention relates to the field of equipment health monitoring technology, specifically to a method and system for diagnosing equipment health status based on visual and digital twin mapping. Background Technology

[0002] With the continuous advancement of Industry 4.0 and intelligent manufacturing, industrial equipment is gradually developing towards high integration, intelligence, and complexity. The operating status of equipment directly affects production continuity, product quality, and enterprise efficiency. Early equipment maintenance mainly relied on reactive repairs and periodic preventative maintenance, which has gradually evolved into condition monitoring and predictive maintenance based on sensor data. In recent years, the continuous integration of IoT, AI, computer vision, and digital twin technologies, utilizing cameras, inspection robots, drones, and other technologies to collect visual information and combine it with digital models to monitor equipment operating status, has become an important development direction for equipment health management.

[0003] However, existing technologies still have shortcomings: there is a lack of high-precision spatiotemporal correspondence between visual data and digital twin models, making it difficult to accurately locate anomalies in images to their corresponding parts in the model; there is a lack of automated deviation calculation mechanisms between theoretical and actual observed states, and equipment anomaly identification still relies heavily on human experience; at the same time, the ability to fuse and analyze multi-source heterogeneous data is insufficient, resulting in limited accuracy and stability of health status assessment. Therefore, there is a need for equipment health status diagnosis methods and systems based on visual and digital twin mapping to solve the above problems. Summary of the Invention

[0004] (a) Technical problem to be solved: In view of the shortcomings of the existing technology, the present invention provides a device health status diagnosis method and system based on visual and digital twin mapping, which solves the above-mentioned problems.

[0005] (II) Technical Solution: To achieve the above objectives, the present invention provides the following technical solution: a device health status diagnosis method based on visual and digital twin mapping, comprising: S1. Acquire visual image data, sensor monitoring data, and current operating parameters of the target device; S2. Perform time alignment, coordinate unification, and preprocessing on the visual image data and the sensor monitoring data; S3. Map the preprocessed visual image data to the digital twin model corresponding to the target device to obtain the actual observation state of the target device, wherein S3 includes: S31. Establish the projection relationship between the pixels of the visual image and the points in the world coordinate system space. S32. Based on the projection relationship and multi-view synchronously captured images, determine the spatial point corresponding to the pixel; S33. Map the spatial points to the corresponding geometric points on the surface of the digital twin model; S34. Write the visual attributes corresponding to the visual image data into the corresponding geometric point, and perform dynamic updates and multi-view fusion to obtain the actual observation state. S4. Based on the current operating parameters and the digital twin model, determine the theoretical health status of the target equipment; S5. Calculate the deviation between the actual observed state and the theoretical health state to obtain the deviation result; S6. Determine the health status diagnosis result of the target device based on the deviation result.

[0006] Furthermore, S2 includes: The visual image data and the sensor monitoring data are timestamped and aligned. Transform data from different sensor coordinate systems to a unified world coordinate system; Image enhancement and key point detection are performed on the visual image data; The visual image data is then subjected to distortion correction, wherein the distortion correction satisfies the following conditions: , , ;in, These are the pixel coordinates before distortion correction. These are the pixel coordinates after distortion correction. Principal point coordinates The radial distortion coefficient is... The tangential distortion coefficient is... These are the physical coordinates of the image.

[0007] Furthermore, S31 includes: Based on the camera intrinsic and extrinsic parameter matrices, a projection relationship is established between the pixel coordinate system, the image physical coordinate system, the camera coordinate system, and the world coordinate system. This projection relationship satisfies the following: , ;in, This refers to the actual size of a single pixel in both directions. For the camera intrinsic parameter matrix, Let be a rotation matrix. It is a translation vector. This is the depth scaling factor. These are spatial coordinates in the world coordinate system.

[0008] Furthermore, S32 includes: Based on epipolar geometric constraints and triangulation of multi-view synchronously captured images, spatial points corresponding to pixels are determined, and the depth information of these spatial points satisfies: , ;in, These are the projection points of the same spatial point from two different viewpoints. Based on the matrix, The depth value of the spatial point. For the camera's focal length, The baseline distance between the optical centers of the two cameras. For parallax, and , These are the horizontal pixel coordinates of the two projection points; and a pixel-by-pixel depth map is generated based on the depth information. .

[0009] Furthermore, S33 includes: The digital twin model is discretized into multiple triangular facets. Each triangular facet From vertex constitute; Solve for spatial points using the following formula. The nearest point to the surface of the digital twin model: ;in, , ; The solution to the nearest point problem gives a spatial point The location of the nearest point on the surface of the digital twin model and the triangular facet to which it belongs; When the spatial point The distance to the surface of the digital twin model is less than the tolerance threshold. When determining the spatial point Mapped to the corresponding geometric points on the surface of the digital twin model; An octree spatial index is used to spatially index the triangular facets.

[0010] Furthermore, S34 includes: The visual attributes corresponding to the visual image data are written into the corresponding geometric points, and dynamic texture updates and multi-view confidence weighted fusion are performed to satisfy: , , ;in, This represents the accumulated texture state at the current moment. The texture image written at the current moment. For update rate, For digital twin model surface points The fusion of visual attribute values, For the first The visual attribute values ​​corresponding to each viewpoint For the first Weight of each perspective For the first The angle between the observation direction from each viewpoint and the surface normal. For the first Observation distance from each perspective For the first Image quality rating from each perspective.

[0011] Furthermore, S4 includes: Based on a pre-established database of operating parameters and theoretical states Determine the theoretical health status of the target equipment, satisfying: , ;in, These are the current operating parameters. For the first Preset operating parameters for the group To and The corresponding theoretical state matrix, For the current operating condition parameters corresponding to the first Interpolation weights of preset operating condition parameters, This is the bandwidth parameter.

[0012] Furthermore, S4 also includes: The theoretical health status is statistically updated based on historical normal operation data to satisfy: , ;in, For the first Historical normal observation vector at time , This represents the total number of historical normal observation vectors. The length of the sliding window. This is the vector of theoretical health state mean values. For the first The updated theoretical health status mean vector.

[0013] Furthermore, S5 includes: Build area The visual feature vectors satisfy: ; Visual bias is determined based on the difference between theoretical visual features and actual visual features, satisfying the following: , ;in, geometric point The actual visual feature vector at that location. geometric point Theoretical visual feature vector at the location, This is the visual feature covariance matrix under normal conditions; The sensor bias is determined based on the difference between theoretical and actual sensor parameters, satisfying the following: , ;in, This is the actual sensor parameter vector. For the theoretical sensor parameter vector, For the number of sensors, For the first The standard deviation of each sensor under normal conditions; S6 includes: performing joint analysis based on the visual bias and the sensor bias to determine the comprehensive anomaly probability, satisfying: ;in, To comprehensively consider the probability of anomalies, Let be the conditional probability of visual deviation under abnormal conditions. This represents the conditional probability of sensor deviation under abnormal conditions. This represents the anomalous prior probability. The marginal probability of visual bias. The edge probability of sensor bias; The system outputs the health status diagnosis results of the target device based on the comprehensive anomaly probability.

[0014] This invention also provides a device health status diagnosis system based on visual and digital twin mapping, comprising: The data acquisition module is used to acquire visual image data, sensor monitoring data, and current operating parameters of the target device; The preprocessing module is used to perform time alignment, coordinate unification, and preprocessing on the visual image data and the sensor monitoring data; A twin mapping module is used to map the preprocessed visual image data to the digital twin model corresponding to the target device, thereby obtaining the actual observation state of the target device; the twin mapping module includes: The projection relationship establishment unit is used to establish the projection relationship between visual image pixels and points in the world coordinate system space; The spatial point determination unit is used to determine the spatial point corresponding to the pixel based on the projection relationship and the multi-view synchronously captured image. A geometric point mapping unit is used to map the spatial points to corresponding geometric points on the surface of the digital twin model; The visual attribute fusion unit is used to write the visual attributes corresponding to the visual image data into the corresponding geometric point, and to perform dynamic updates and multi-view fusion to obtain the actual observation state. The theoretical state determination module is used to determine the theoretical health state of the target equipment based on the current operating parameters and the digital twin model; The deviation calculation module is used to calculate the deviation based on the actual observed state and the theoretical health state, and obtain the deviation result. The diagnostic output module is used to determine the health status diagnostic result of the target device based on the deviation result.

[0015] (III) Beneficial Effects: Compared with the prior art, the present invention provides a method and system for diagnosing device health status based on visual and digital twin mapping, which has the following beneficial effects: 1. This device health status diagnosis method and system based on visual and digital twin mapping establishes the projection relationship between visual image pixels and spatial points in the world coordinate system. Combined with depth recovery from multi-view synchronized images, mapping of spatial points to corresponding geometric points on the surface of the digital twin model, and a visual attribute writing and fusion mechanism, it achieves precise mapping between visual data and the digital twin model. This enables accurate mapping of device appearance anomalies and local state changes to specific parts of the digital twin model, solving the problems of lacking high-precision spatiotemporal correspondence between visual data and the digital twin model, and the difficulty in accurately pinpointing abnormal locations in existing technologies.

[0016] 2. This equipment health status diagnosis method and system based on visual and digital twin mapping determines the theoretical health status of the target equipment based on current operating parameters and a digital twin model. It then calculates the deviation between the actual observed status and the theoretical health status, constructing a quantitative analysis mechanism for visual and sensor deviations, and ultimately outputs the health status diagnosis result. This transforms equipment health status judgment from manual experience comparison to automated calculation and analysis based on models and data, solving the problems of existing technologies lacking an automated deviation calculation mechanism between theoretical and actual observed status and relying heavily on manual experience for anomaly identification.

[0017] 3. This equipment health status diagnosis method and system based on visual and digital twin mapping jointly analyzes visual and sensor biases and outputs the health status diagnosis results of the target equipment based on the comprehensive anomaly probability, enabling visual and sensor information to work synergistically in a unified diagnostic process. This improves the accuracy and stability of equipment health status assessment, reduces the risk of misjudgment from a single data source, and solves the problems of insufficient multi-source heterogeneous data fusion and analysis capabilities and limited accuracy and reliability of health status assessment in existing technologies. Attached Figure Description

[0018] Figure 1 A flowchart illustrating the device health status diagnosis method based on visual and digital twin mapping provided by the present invention; Figure 2 A schematic diagram of the device health status diagnosis system based on visual and digital twin mapping provided by the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] To enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0021] Please see Figure 1 , Figure 1 This is a flowchart illustrating the device health status diagnosis method based on visual and digital twin mapping provided by the present invention; the present invention provides a device health status diagnosis method based on visual and digital twin mapping, including: S1. Acquire visual image data, sensor monitoring data, and current operating parameters of the target device; Specifically, visual acquisition units, sensor acquisition units, and operating parameter reading units are deployed for the target equipment. First, a correspondence is established between the target equipment identifier, the acquisition location identifier, and the data source identifier. Then, each acquisition unit performs data acquisition around the same target equipment. The target equipment can be a fan, pump, compressor, machine tool spindle, conveyor mechanism, or other operating equipment in the production line.

[0022] Visual image data is acquired by a visual acquisition unit. This unit can be one or more of a fixed industrial camera, a pan-tilt camera, a camera mounted on an inspection robot, or a camera mounted on a drone. The acquired content includes images of the target equipment's outer surface, partial images of key components, images of moving parts, and images of the operating area. Visual image data must at least include image frame content, acquisition time, acquisition viewpoint, and corresponding device identifier. The image frame content characterizes the appearance of the target equipment at the acquisition time, the acquisition viewpoint distinguishes the image source from different shooting positions, and the corresponding device identifier identifies the target equipment to which the image data belongs.

[0023] Sensor monitoring data is acquired by a sensor acquisition unit. This unit may include one or more of the following: vibration sensors, temperature sensors, current sensors, displacement sensors, and acoustic emission sensors. Each sensor outputs a monitoring quantity related to the operating status of the target equipment, forming a sensor monitoring data set. The sensor monitoring data includes at least the monitored value, sensor type, installation location, and acquisition time. The installation location characterizes the part of the equipment corresponding to the monitored value, and the sensor type distinguishes the sources of different physical quantities.

[0024] Current operating parameters are acquired by the operating parameter reading unit. This unit can read the target equipment's current operating parameters from the controller, data acquisition card, host computer, or fieldbus interface. Current operating parameters include one or more of the following: speed, load, flow rate, pressure, inlet and outlet temperature, drive frequency, and valve opening degree.

[0025] During the acquisition process, visual image data, sensor monitoring data, and current operating parameters are all written to the same device data cache and aggregated according to the target device identifier. After this processing, a raw observation data set for the same target device can be formed, which includes appearance image information, physical monitoring information, and operating condition information.

[0026] S2. Perform time alignment, coordinate unification, and preprocessing on the visual image data and the sensor monitoring data; Specifically, the visual image data and sensor monitoring data are first uniformly organized to ensure that data from different sources have a consistent temporal and spatial reference. Then, data results that can be directly used for subsequent mapping calculations are output. The data processed in this step includes organized image frame sequences, sensor observation results corresponding to the image frames, and data identification information corresponding to the spatial location of the equipment.

[0027] Time alignment is used to establish the corresponding time relationship between image frames and sensor observations, enabling image information and monitoring information at the same analysis time to jointly characterize the actual operating status of the target device. Coordinate unification is used to eliminate positional deviations caused by inconsistent coordinate definitions among different acquisition units, ensuring that the visual acquisition position, sensor installation position, and target device spatial position correspond to the same spatial reference system. Preprocessing is used to remove non-target influencing factors from the raw acquisition data, forming input data suitable for subsequent digital twin mapping and deviation calculation.

[0028] Furthermore, in one embodiment provided by the present invention, S2 includes: The visual image data and the sensor monitoring data are timestamped and aligned. Transform data from different sensor coordinate systems to a unified world coordinate system; Image enhancement and key point detection are performed on the visual image data; The visual image data is then subjected to distortion correction, wherein the distortion correction satisfies the following conditions: , , ;in, These are the pixel coordinates before distortion correction. These are the pixel coordinates after distortion correction. Principal point coordinates The radial distortion coefficient is... The tangential distortion coefficient is... These are the physical coordinates of the image.

[0029] Specifically, in the timestamp alignment stage, the acquisition time corresponding to each image frame is first read, followed by the sampling time corresponding to each sensor monitoring value, and a time correspondence is established using a unified system clock as a reference. When the image acquisition frequency and the sensor sampling frequency are inconsistent, the image frame time can be used as the alignment reference, and monitoring values ​​at the same or adjacent times can be selected from the sensor data sequence. Alternatively, interpolation calculations can be performed based on the sampling points before and after to obtain the sensor monitoring results corresponding to the image frame.

[0030] In the coordinate transformation stage, the position parameters of each sensor's installation location in the local coordinate system are first determined. Then, based on the equipment installation relationship, bracket installation relationship, or field calibration results, a transformation relationship from the local coordinate system to the world coordinate system is established. For monitoring data such as displacement, vibration, temperature, or acoustic emission, while retaining the monitoring values, their installation location and orientation information are transformed to the world coordinate system to obtain sensor position data in a unified spatial reference. After the transformation, the monitoring locations corresponding to different sensors can be located within the same equipment space model.

[0031] In the image enhancement stage, to address issues such as uneven brightness, localized over-darkness, blurred edges, or noise interference in the original image, grayscale adjustment, contrast stretching, noise reduction filtering, or sharpening are performed to give the device outline, connection boundaries, surface texture, and local anomalies a clearer representation in the image. In the keypoint detection stage, corner points, edge intersections, contour feature points, or stable texture points on the device surface are extracted from the enhanced image, and the detection results are used as reference features for subsequent spatial point calculation and geometric position matching. Keypoints can be extracted using methods based on grayscale changes, gradient changes, or local neighborhood responses, as long as they can stably represent the position of the target device surface structure.

[0032] In the distortion correction stage, lens distortion in the visual image is compensated. For any pixel in the image, its radial distance to the principal point is first calculated based on the image's physical coordinates. Combined with the radial distortion coefficient and tangential distortion coefficient The horizontal and vertical pixel coordinates are corrected separately to obtain the corrected result. Specifically, radial distortion corrects the drum-shaped or pincushion-shaped changes in the image edge region caused by lens imaging, while tangential distortion corrects the image point tilt shift caused by lens mounting deviations. After distortion correction, the positions of equipment edges, key points, and local structures in the image are more consistent with the actual imaging positions.

[0033] S3. Map the preprocessed visual image data to the digital twin model corresponding to the target device to obtain the actual observation state of the target device; Specifically, this step is used to convert the device appearance information in the two-dimensional image into spatial state information of the surface of the digital twin model. In practice, the correspondence between image pixels and device spatial positions is first determined based on the imaging relationship of the visual acquisition system. Then, the spatial point positions of the device surface are recovered by combining multi-view images. After that, the spatial points are located to the specific geometric positions on the surface of the digital twin model. Finally, visual attributes such as color, texture, temperature distribution, or edge changes in the image are written into the corresponding geometric positions to form the actual observation state consistent with the current moment.

[0034] The actual observed state is not a single image result, but rather a set of visual states corresponding to each effective area of ​​the digital twin model surface at the current moment. This set preserves both the spatial distribution of changes in the device's appearance and the correspondence between image information from different acquisition perspectives and the device surface, thus allowing subsequent deviation calculations to be directly based on the surface positions in the digital twin model. After this processing, local color differences, surface wear, edge changes, abnormal thermal distribution, or changes in moving areas in the image can all be mapped to specific locations in the digital twin model, rather than remaining at the level of the original two-dimensional image.

[0035] Wherein, S3 includes: S31. Establish the projection relationship between the pixels of the visual image and the points in the world coordinate system space. Furthermore, in one embodiment provided by the present invention, S31 includes: Based on the camera intrinsic and extrinsic parameter matrices, a projection relationship is established between the pixel coordinate system, the image physical coordinate system, the camera coordinate system, and the world coordinate system. This projection relationship satisfies the following: , ;in, This refers to the actual size of a single pixel in both directions. For the camera intrinsic parameter matrix, Let be a rotation matrix. It is a translation vector. This is the depth scaling factor. These are spatial coordinates in the world coordinate system.

[0036] Specifically, this step first establishes the conversion relationship between imaging coordinates. For any image pixel, the actual size of the single pixel in both directions is first determined. and principal point coordinates This process converts pixel coordinates into physical coordinates, thereby determining the physical location of the image point in the imaging plane. After this conversion, the pixel position in the image no longer simply represents a row and column index, but has a physical meaning corresponding to the imaging system parameters.

[0037] In the transformation between image physical coordinates and camera coordinates / world coordinates, the camera intrinsic parameter matrix is ​​used. Imaging parameters such as focal length and principal point are characterized using a rotation matrix. Translation vector This represents the camera's pose and position relative to the world coordinate system. This establishes a projection link from image pixels to the device's spatial location. For the same pixel, given a depth scaling factor... Then, the position of the corresponding world coordinate system point can be deduced; for any point in space, its projection position in the image can also be obtained based on the projection relationship.

[0038] The purpose of this projection relationship is to connect the visual image with the three-dimensional space in which the device is located, providing a unified mathematical basis for subsequent spatial point reconstruction and surface mapping of the digital twin model. After this step is completed, a definite correspondence rule is formed between image pixels, imaging parameters, and the spatial position of the device, eliminating the need for subsequent manual specification of empirical correspondences between image regions and device parts.

[0039] S32. Based on the projection relationship and multi-view synchronously captured images, determine the spatial point corresponding to the pixel; Furthermore, in one embodiment provided by the present invention, S32 includes: Based on epipolar geometric constraints and triangulation of multi-view synchronously captured images, spatial points corresponding to pixels are determined, and the depth information of these spatial points satisfies: , ;in, These are the projection points of the same spatial point from two different viewpoints. Based on the matrix, The depth value of the spatial point. For the camera's focal length, The baseline distance between the optical centers of the two cameras. For parallax, and , These are the horizontal pixel coordinates of the two projection points; and a pixel-by-pixel depth map is generated based on the depth information. .

[0040] Specifically, this step utilizes images acquired simultaneously by multiple cameras to reconstruct the three-dimensional position of the device surface. For any pixel to be matched in the image, its corresponding projection point is first found in another viewpoint image, and then the fundamental matrix is ​​used... Constrain the epipolar correspondence between two projection points. This constraint limits the search range of corresponding points to the vicinity of the epipolar line, thereby determining the matching position of the same device surface point in images from different viewpoints.

[0041] After obtaining the matching projection points from the two viewpoints, based on the camera focal length Baseline distance between the two cameras And the horizontal pixel difference between the two projection points The depth value of the corresponding spatial point is calculated. The depth value reflects the distance information of the spatial point relative to the camera. Combined with the aforementioned projection relationship, the three-dimensional coordinates of the spatial point in the world coordinate system can be further obtained. After this processing, a single pixel in the two-dimensional image can be converted into a three-dimensional spatial point corresponding to the device surface.

[0042] For continuous image regions, the above calculations can be performed pixel by pixel to generate a depth map. Each pixel in the depth map corresponds to a depth value, which represents the spatial hierarchy of that location from the current viewpoint. For the outer surface area of ​​a device, the depth map can reflect the location distribution of surface contour undulations, local protrusions, depressions, or structural boundaries, providing direct input for subsequently mapping spatial points onto the surface of the digital twin model.

[0043] S33. Map the spatial points to the corresponding geometric points on the surface of the digital twin model; Furthermore, in one embodiment provided by the present invention, S33 includes: The digital twin model is discretized into multiple triangular facets. Each triangular facet From vertex constitute; Solve for spatial points using the following formula. The nearest point to the surface of the digital twin model: ;in, , ; The solution to the nearest point problem gives a spatial point The location of the nearest point on the surface of the digital twin model and the triangular facet to which it belongs; When the spatial point The distance to the surface of the digital twin model is less than the tolerance threshold. When determining the spatial point Mapped to the corresponding geometric points on the surface of the digital twin model; An octree spatial index is used to spatially index the triangular facets.

[0044] Specifically, this step locates the recovered 3D spatial points to their exact positions on the surface of the digital twin model. In practice, the digital twin model is first discretized into multiple triangular facets, each facet having its local surface geometry defined by three vertices. The discretized model facilitates local surface search and geometric localization for any given spatial point.

[0045] For any spatial point The algorithm calculates the distances from the spatial point to the surfaces of each triangular facet, and finds the position with the minimum distance while satisfying the centroid coordinate constraints. The solution provides two things: the position of the nearest point on the surface of the digital twin model and the triangular facet number to which that nearest point belongs. This allows a point in 3D space to be mapped to a specific geometric location on the surface of the digital twin model, rather than just a general area.

[0046] After solving for the nearest point, compare the spatial points. The distance between the nearest point on the model surface. When this distance is less than a tolerance threshold... When the distance is greater than the tolerance threshold, the nearest point is determined as the mapping result; when the distance is greater than the tolerance threshold, the nearest point is determined as the mapping result. If the spatial point is more likely to originate from a background area, an occluded area, or an unrelated area outside the device, it should not be written into the digital twin model. This ensures that all points entering the model surface satisfy the spatial proximity condition, preventing the mismapping of non-device area information onto the target device surface.

[0047] Since digital twin models typically contain a large number of triangular faces, traversing them one by one would incur significant computational costs. Therefore, before searching the model's surface, an octree spatial index can be used to partition and manage all triangular faces. When a spatial point enters the search, its corresponding spatial region is first determined in the octree, and then candidate triangular faces are searched within that region. This narrows the search scope, allowing the surface localization process for each spatial point to be completed within a local area.

[0048] S34. Write the visual attributes corresponding to the visual image data into the corresponding geometric point, and perform dynamic updates and multi-view fusion to obtain the actual observation state. Furthermore, in one embodiment provided by the present invention, S34 includes: The visual attributes corresponding to the visual image data are written into the corresponding geometric points, and dynamic texture updates and multi-view confidence weighted fusion are performed to satisfy: , , ;in, This represents the accumulated texture state at the current moment. The texture image written at the current moment. For update rate, For digital twin model surface points The fusion of visual attribute values, For the first The visual attribute values ​​corresponding to each viewpoint For the first Weight of each perspective For the first The angle between the observation direction from each viewpoint and the surface normal. For the first Observation distance from each perspective For the first Image quality rating from each perspective.

[0049] Specifically, after completing the geometric location mapping, this step writes the visual content from the image onto the surface of the digital twin model. Visual attributes can include color values, grayscale values, texture distribution, edge intensity, thermal imaging temperature values, or other surface characterization information corresponding to image pixels. For each mapped geometric point, the visual attributes at its corresponding pixel location are first read, and then these visual attributes are written to the corresponding location on the digital twin model surface, so that the model surface has a state expression consistent with the actual appearance of the device at the current moment.

[0050] In continuous data acquisition scenarios, the same geometric point may be repeatedly observed at multiple time points. To ensure that the surface state reflects continuous temporal changes, a cumulative texture update method is used to handle the relationship between the current texture and historical textures. The texture image at the current time point is shown below. Cumulative texture state compared to the previous moment By update rate Combine them to obtain the cumulative texture state at the current moment. New observations will be incorporated into the current model surface state, while existing historical data will be retained proportionally, thus ensuring that short-term occlusion, illumination fluctuations, or transient noise will not directly disrupt the continuous representation of the surface state.

[0051] When observing the same geometric point simultaneously from multiple perspectives, the visual attributes corresponding to different perspectives may differ. Therefore, the observation results from multiple perspectives are weighted and fused. For surface points... , No. The weight of each viewpoint is determined by the angle between the observation direction and the surface normal. Observation distance and image quality rating The viewpoint is determined collaboratively. The closer the observation direction is to the surface normal direction, and the closer the distance, the better the image sharpness and exposure; therefore, the greater the weight of that viewpoint. The weights of each viewpoint are then determined accordingly. After performing a weighted summation, the surface points can be obtained. fusion of visual attribute values The resulting actual observation state includes both the image observation results at the current moment and the surface state information resulting from the combined effect of multiple effective viewpoints.

[0052] S4. Based on the current operating parameters and the digital twin model, determine the theoretical health status of the target equipment; Specifically, this step is used to establish a baseline state for comparison of the target device under the current operating conditions. In practice, the operating parameters of the target device at the current moment are first read, and then the digital twin model corresponding to the target device is invoked to calculate the state that the target device should exhibit under normal operating conditions based on the current operating conditions.

[0053] The theoretical health state is not a direct replication of historical observations, but rather a model representation of the target equipment's normal operating state under current conditions. This state can be represented by a state matrix, a set of eigenvectors, or a set of model surface parameters, as long as it corresponds one-to-one with the actual observed state in spatial location or parameter dimensions. This approach allows subsequent steps to compare theoretical and actual results under the same operating conditions, avoiding state confusion caused by direct comparisons between different operating conditions.

[0054] Furthermore, in one embodiment provided by the present invention, S4 includes: Based on a pre-established database of operating parameters and theoretical states Determine the theoretical health status of the target equipment, satisfying: , ;in, These are the current operating parameters. For the first Preset operating parameters for the group To and The corresponding theoretical state matrix, For the current operating condition parameters corresponding to the first Interpolation weights of preset operating condition parameters, This is the bandwidth parameter.

[0055] Specifically, in this embodiment, a database of operating parameters and theoretical states is first established around the target device. Each set of data in the database includes a set of preset operating parameters. and the theoretical state matrix corresponding to this set of operating parameters The preset operating parameters can include one or more of the following: rotational speed, load, flow rate, pressure, and ambient temperature; the theoretical state matrix can correspond to the standard visual features, standard displacement distribution, standard temperature distribution, or standard sensor parameter distribution of the digital twin model surface. Once the database is established, it can cover the baseline state range of the target equipment under different normal operating conditions.

[0056] When the system reads the current operating parameters First calculate With each set of preset operating parameters The distance relationship between them is used to determine the interpolation weights corresponding to each set of preset working condition parameters. The smaller the distance, the closer the preset operating parameters are to the current operating condition, and the greater the contribution of the corresponding theoretical state matrix to the final result; the larger the distance, the smaller the contribution. Then, the theoretical state matrices of each group are weighted and summed according to their corresponding interpolation weights to obtain the theoretical health state under the current operating condition. .

[0057] Bandwidth parameters Used to control the range of influence of operating distance on interpolation weights. When When the value is small, the theoretical health status is mainly determined by a few database results that are closest to the current operating conditions; when... When the value is large, the theoretical health state will incorporate theoretical state matrices from more adjacent operating conditions. In this way, the theoretical health state under continuous operating conditions can be obtained based on a discrete operating condition database, so that the baseline state corresponding to the current operating conditions has the characteristic of continuous change.

[0058] Furthermore, in one embodiment provided by the present invention, S4 further includes: The theoretical health status is statistically updated based on historical normal operation data to satisfy: , ;in, For the first Historical normal observation vector at time , This represents the total number of historical normal observation vectors. The length of the sliding window. This is the vector of theoretical health state mean values. For the first The updated theoretical health status mean vector.

[0059] Specifically, in this embodiment, based on determining the theoretical health state using operating parameters and a theoretical state database, the theoretical health state is further statistically corrected by incorporating normal observation data accumulated during the long-term operation of the target equipment. Historical normal observation vector. It can be composed of visual features, sensor parameters, and operating parameters collected under normal conditions, or it can be composed of the parts directly related to the health status assessment. By averaging the historical normal observation vectors at multiple times, a theoretical health status mean vector reflecting the statistical center position of the target equipment's normal operation can be obtained. .

[0060] During continuous equipment operation, the normal state itself will slowly change due to wear, environmental changes, or long-term service. Therefore, the theoretical health state mean vector is updated using a sliding window method. Whenever a new normal observation vector is obtained... When entering a window, simultaneously remove the earliest observation vector from the window. Then, the current mean vector is corrected according to the update formula to obtain the first... After the next update After this processing, the theoretical health status is no longer fixed at the standard state of the equipment in its initial stage, but can reflect the normal range of changes of the target equipment in its current service stage.

[0061] The statistically updated theoretical health status can be used in conjunction with the theoretical status results obtained through database interpolation. The database interpolation results characterize the theoretical baseline corresponding to the current operating condition, while the statistically updated results characterize the actual normal center formed by the target equipment during long-term normal operation. Combining these two allows subsequent deviation calculations to be carried out jointly around the current operating conditions and the current service stage.

[0062] S5. Calculate the deviation between the actual observed state and the theoretical health state to obtain the deviation result; Specifically, this step converts the current observation results into quantifiable deviation results. In practice, a comparison relationship is first established under a unified location benchmark and unified parameter dimensions. Then, the actual state and theoretical state on the surface of the digital twin model are compared point-by-point, region-by-region, or parameter-by-parameter. The resulting deviation results not only characterize whether the target device is currently deviating from its normal state, but also characterize the location of the deviation, the strength of the deviation, and the data type corresponding to the deviation.

[0063] Deviation results can be output in one or more forms, such as deviation vector, deviation scalar, deviation heatmap, or set of abnormal regions. Among them, deviation results mapped to the surface of the digital twin model are used to reflect the spatial changes of the device's appearance, texture, edges, and local motion state; deviation results corresponding to the sensor positions are used to reflect changes in physical monitoring quantities such as vibration, temperature, current, displacement, or acoustic emission.

[0064] Furthermore, in one embodiment provided by the present invention, S5 includes: Build area The visual feature vectors satisfy: ; Visual bias is determined based on the difference between theoretical visual features and actual visual features, satisfying the following: , ;in, geometric point The actual visual feature vector at that location. geometric point Theoretical visual feature vector at the location, This is the visual feature covariance matrix under normal conditions; The sensor bias is determined based on the difference between theoretical and actual sensor parameters, satisfying the following: , ;in, This is the actual sensor parameter vector. For the theoretical sensor parameter vector, For the number of sensors, For the first The standard deviation of a sensor under normal conditions.

[0065] Specifically, in this embodiment, the area to be analyzed is first located on the surface of the digital twin model. Construct visual feature vectors. Region It can correspond to the neighborhood of a single geometric point, a single surface region, or a local functional area on the device surface. The visual feature vector consists of color features, texture features, edge features, and motion features. Color features characterize the color distribution or thermal imaging value distribution within the region; texture features characterize surface roughness, wear texture, or crack texture; edge features characterize changes in contour boundaries and structural lines; and motion features characterize the local displacement trend or change in motion direction between adjacent moments. By uniformly incorporating multiple visual dimensions into the same feature vector, the appearance of the same region can have a unified form of comparison.

[0066] In the visual deviation calculation stage, for geometric points The visual deviation vector is obtained by performing a difference operation between the actual visual feature vector and the theoretical visual feature vector at a given location. This deviation vector represents the geometric point. The deviation across each visual feature dimension is then analyzed. Next, the visual feature covariance matrix under normal conditions is considered. A weighted distance calculation is performed on the visual bias vector to obtain the visual bias scalar. The covariance matrix is ​​used to characterize the fluctuation range and interrelationships of each visual feature dimension under normal conditions, allowing the bias results to simultaneously consider the scale of change of different feature dimensions. After calculation, the corresponding geometric points can be... Write back to the surface of the digital twin model to generate visual deviation distribution results, which are used to characterize the location and extent of abnormal areas on the device surface.

[0067] In the sensor deviation calculation stage, the actual sensor parameter vector at the current moment and the theoretical sensor parameter vector under the same operating condition are first read, and then the difference between the two is calculated dimension by dimension to obtain the sensor deviation vector. Because different sensors correspond to different physical dimensions and normal fluctuation ranges, the deviation value of each sensor channel is further calculated using the standard deviation under normal conditions. Perform normalization processing, then process all The normalized results of each sensor channel are summarized to obtain the sensor bias scalar. This is how it was obtained. It is used to reflect the overall deviation of the target device across multiple physical monitoring quantities, without masking the deviation changes of other monitoring channels due to the large-dimensional value of a single sensor.

[0068] After the above processing, the deviation results output by S5 include at least visual deviation results and sensor deviation results. The visual deviation results are based on the surface position of the digital twin model, while the sensor deviation results are based on the physical monitoring parameters. Together, they constitute the deviation description of the target device at the current moment.

[0069] S6. Determine the health status diagnosis result of the target device based on the deviation result; Specifically, visual deviation results and sensor deviation results are first read, and then a joint judgment relationship is established around the same target device and the same analysis time to avoid giving diagnostic conclusions based on only a single data source. Visual deviation results are used to reflect the appearance deviation between the surface state of the target device and its theoretical state, while sensor deviation results are used to reflect the physical deviation between the operating parameters of the target device and its theoretical parameters.

[0070] Health status diagnosis results can be output in one or more levels: healthy, mildly abnormal, moderately abnormal, and severely abnormal. Alternatively, they can be output in a combination of abnormality probability values, abnormality level indicators, and abnormality location descriptions. For visual deviation results that have already formed a spatial distribution on the surface of the digital twin model, the location of the abnormal area can also be output as part of the diagnostic result, so that the diagnostic conclusion includes both status category and spatial location information.

[0071] Furthermore, in one embodiment provided by the present invention, S6 includes: performing joint analysis based on the visual bias and the sensor bias to determine the comprehensive anomaly probability, satisfying: ;in, To comprehensively consider the probability of anomalies, Let be the conditional probability of visual deviation under abnormal conditions. This represents the conditional probability of sensor deviation under abnormal conditions. This represents the anomalous prior probability. The marginal probability of visual bias. The edge probability of sensor bias; The system outputs the health status diagnosis results of the target device based on the comprehensive anomaly probability.

[0072] Specifically, first scalarize the visual bias and sensor deviation scalar As input for joint analysis, the overall anomaly probability is calculated based on the statistical distribution relationship between the two types of deviations under abnormal conditions. Anomaly Prior Probability It can be determined based on the target equipment's historical failure frequency, service life, maintenance records, or preset operating rules. Conditional probability of visual bias. The conditional probability of sensor bias is used to characterize the likelihood of the current level of visual bias occurring under abnormal conditions. Marginal probability of visual bias used to characterize the likelihood of the current sensor bias level occurring under abnormal conditions; Marginal probability of sensor deviation This is used to characterize the distribution of the two types of deviations in the overall sample. After calculating using the above probability relationship, the comprehensive anomaly probability corresponding to the current time can be obtained.

[0073] In the output phase, the overall anomaly probability is first mapped to a preset diagnostic interval, and then the current health status diagnosis result of the target device is given. For example, when the overall anomaly probability is lower than the first probability threshold, a healthy status is output; when the overall anomaly probability is between the first and second probability thresholds, a slightly abnormal status is output; when the overall anomaly probability is between the second and third probability thresholds, a moderately abnormal status is output; and when the overall anomaly probability is higher than the third probability threshold, a severely abnormal status is output. In addition to the status level, the model location corresponding to the abnormal area, the main source of deviation, and the current time identifier can also be output simultaneously, forming a complete diagnostic result record.

[0074] Please see Figure 2 , Figure 2 A schematic diagram of the device health status diagnosis system based on visual and digital twin mapping provided by the present invention. The present invention also provides a device health status diagnosis system based on visual and digital twin mapping, comprising: The data acquisition module is used to acquire visual image data, sensor monitoring data, and current operating parameters of the target device; The preprocessing module is used to perform time alignment, coordinate unification, and preprocessing on the visual image data and the sensor monitoring data; A twin mapping module is used to map the preprocessed visual image data to the digital twin model corresponding to the target device, thereby obtaining the actual observation state of the target device; the twin mapping module includes: The projection relationship establishment unit is used to establish the projection relationship between visual image pixels and points in the world coordinate system space; The spatial point determination unit is used to determine the spatial point corresponding to the pixel based on the projection relationship and the multi-view synchronously captured image. A geometric point mapping unit is used to map the spatial points to corresponding geometric points on the surface of the digital twin model; The visual attribute fusion unit is used to write the visual attributes corresponding to the visual image data into the corresponding geometric point, and to perform dynamic updates and multi-view fusion to obtain the actual observation state. The theoretical state determination module is used to determine the theoretical health state of the target equipment based on the current operating parameters and the digital twin model; The deviation calculation module is used to calculate the deviation based on the actual observed state and the theoretical health state, and obtain the deviation result. The diagnostic output module is used to determine the health status diagnostic result of the target device based on the deviation result.

[0075] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0076] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A device health status diagnosis method based on visual and digital twin mapping, characterized in that, include: S1. Acquire visual image data, sensor monitoring data, and current operating parameters of the target device; S2. Perform time alignment, coordinate unification, and preprocessing on the visual image data and the sensor monitoring data; S3. Map the preprocessed visual image data to the digital twin model corresponding to the target device to obtain the actual observation state of the target device, wherein S3 includes: S31. Establish the projection relationship between the pixels of the visual image and the points in the world coordinate system space. S32. Based on the projection relationship and multi-view synchronously captured images, determine the spatial point corresponding to the pixel; S33. Map the spatial points to the corresponding geometric points on the surface of the digital twin model; S34. Write the visual attributes corresponding to the visual image data into the corresponding geometric point, and perform dynamic updates and multi-view fusion to obtain the actual observation state. S4. Based on the current operating parameters and the digital twin model, determine the theoretical health status of the target equipment; S5. Calculate the deviation between the actual observed state and the theoretical health state to obtain the deviation result; S6. Determine the health status diagnosis result of the target device based on the deviation result.

2. The device health status diagnosis method based on visual and digital twin mapping according to claim 1, characterized in that, S2 includes: The visual image data and the sensor monitoring data are timestamped and aligned. Transform data from different sensor coordinate systems to a unified world coordinate system; Image enhancement and key point detection are performed on the visual image data; The visual image data is then subjected to distortion correction, wherein the distortion correction satisfies the following conditions: , , ;in, These are the pixel coordinates before distortion correction. These are the pixel coordinates after distortion correction. Principal point coordinates The radial distortion coefficient is... The tangential distortion coefficient is... These are the physical coordinates of the image.

3. The device health status diagnosis method based on visual and digital twin mapping according to claim 2, characterized in that, S31 includes: Based on the camera intrinsic and extrinsic parameter matrices, a projection relationship is established between the pixel coordinate system, the image physical coordinate system, the camera coordinate system, and the world coordinate system. This projection relationship satisfies the following: , ;in, This refers to the actual size of a single pixel in both directions. For the camera intrinsic parameter matrix, Let be a rotation matrix. It is a translation vector. This is the depth scaling factor. These are spatial coordinates in the world coordinate system.

4. The device health status diagnosis method based on visual and digital twin mapping according to claim 3, characterized in that, S32 includes: Based on epipolar geometric constraints and triangulation of multi-view synchronously captured images, spatial points corresponding to pixels are determined, and the depth information of these spatial points satisfies: , ;in, These are the projection points of the same spatial point from two different viewpoints. Based on the matrix, The depth value of the spatial point. For the camera's focal length, The baseline distance between the optical centers of the two cameras. For parallax, and , These are the horizontal pixel coordinates of the two projection points; and a pixel-by-pixel depth map is generated based on the depth information. .

5. The device health status diagnosis method based on visual and digital twin mapping according to claim 4, characterized in that, S33 includes: The digital twin model is discretized into multiple triangular facets. Each triangular facet From vertex constitute; Solve for spatial points using the following formula. The nearest point to the surface of the digital twin model: ;in, , ; The solution to the nearest point problem gives a spatial point The location of the nearest point on the surface of the digital twin model and the triangular facet to which it belongs; When the spatial point The distance to the surface of the digital twin model is less than the tolerance threshold. At that time, determine the spatial point Mapped to the corresponding geometric points on the surface of the digital twin model; An octree spatial index is used to spatially index the triangular facets.

6. The device health status diagnosis method based on visual and digital twin mapping according to claim 5, characterized in that, S34 includes: The visual attributes corresponding to the visual image data are written into the corresponding geometric points, and dynamic texture updates and multi-view confidence weighted fusion are performed to satisfy: , , ;in, This represents the accumulated texture state at the current moment. The texture image written at the current moment. For update rate, For digital twin model surface points The fusion of visual attribute values, For the first The visual attribute values ​​corresponding to each viewpoint For the first Weight of each perspective For the first The angle between the observation direction from each viewpoint and the surface normal. For the first Observation distance from each perspective For the first Image quality rating from each perspective.

7. The device health status diagnosis method based on visual and digital twin mapping according to claim 1, characterized in that, S4 includes: Based on a pre-established database of operating parameters and theoretical states Determine the theoretical health status of the target equipment, satisfying: , ;in, These are the current operating parameters. For the first Preset operating parameters for the group To and The corresponding theoretical state matrix, For the current operating condition parameters corresponding to the first Interpolation weights of preset operating condition parameters, This is the bandwidth parameter.

8. The device health status diagnosis method based on visual and digital twin mapping according to claim 7, characterized in that, S4 further includes: The theoretical health status is statistically updated based on historical normal operation data to satisfy: , ;in, For the first Historical normal observation vector at time , This represents the total number of historical normal observation vectors. The length of the sliding window. This is the vector of theoretical health state mean values. For the first The updated theoretical health status mean vector.

9. The device health status diagnosis method based on visual and digital twin mapping according to claim 8, characterized in that, S5 includes: Build region The visual feature vectors satisfy: ; Visual bias is determined based on the difference between theoretical visual features and actual visual features, satisfying the following: , ;in, geometric point The actual visual feature vector at that location. geometric point Theoretical visual feature vector at the location, This is the visual feature covariance matrix under normal conditions; The sensor bias is determined based on the difference between theoretical and actual sensor parameters, satisfying the following: , ;in, This is the actual sensor parameter vector. For the theoretical sensor parameter vector, For the number of sensors, For the first The standard deviation of each sensor under normal conditions; S6 includes: performing joint analysis based on the visual bias and the sensor bias to determine the comprehensive anomaly probability, satisfying: ;in, To comprehensively consider the probability of anomalies, Let be the conditional probability of visual deviation under abnormal conditions. This represents the conditional probability of sensor deviation under abnormal conditions. This represents the anomalous prior probability. The marginal probability of visual bias. The edge probability of sensor bias; The system outputs the health status diagnosis results of the target device based on the comprehensive anomaly probability.

10. A device health status diagnosis system based on visual and digital twin mapping, characterized in that, include: The data acquisition module is used to acquire visual image data, sensor monitoring data, and current operating parameters of the target device; The preprocessing module is used to perform time alignment, coordinate unification, and preprocessing on the visual image data and the sensor monitoring data; The twin mapping module is used to map the preprocessed visual image data to the digital twin model corresponding to the target device to obtain the actual observation state of the target device; The twin mapping module includes: The projection relationship establishment unit is used to establish the projection relationship between visual image pixels and points in the world coordinate system space; The spatial point determination unit is used to determine the spatial point corresponding to the pixel based on the projection relationship and the multi-view synchronously captured image. A geometric point mapping unit is used to map the spatial points to corresponding geometric points on the surface of the digital twin model; The visual attribute fusion unit is used to write the visual attributes corresponding to the visual image data into the corresponding geometric point, and to perform dynamic updates and multi-view fusion to obtain the actual observation state. The theoretical state determination module is used to determine the theoretical health state of the target equipment based on the current operating parameters and the digital twin model; The deviation calculation module is used to calculate the deviation based on the actual observed state and the theoretical health state, and obtain the deviation result. The diagnostic output module is used to determine the health status diagnostic result of the target device based on the deviation result.

Images