Object dynamic deformation monitoring method and system
Through the combination of binocular vision camera and inertial measurement unit, the real-time and accuracy of traditional deformation measurement in close range is solved, and high-precision object deformation monitoring is achieved, which is suitable for scenes such as amusement facilities.
Patent Information
- Application Number
- CN202510442743.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-09
- Publication Date
- 2025-07-11
AI Technical Summary
Traditional deformation measurement methods are limited by the measurement range, environmental requirements or high operating complexity during close-range measurement, making it difficult to achieve real-time and accurate monitoring of the equipment.
A binocular vision camera is used to collect multi-frame images of the object to be measured, combined with visual segmentation technology and inertia measurement unit, and precise monitoring of object deformation is achieved through pixel-level displacement correction and real coordinate system conversion.
High-precision and real-time object deformation monitoring is achieved at close range, reducing operational complexity, improving measurement stability and robustness, and suitable for complex environments.
Smart Images
Figure CN120298453A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of deformation monitoring, and particularly to a method and system for dynamically monitoring the deformation of an object. Background Art
[0002] With the development of mechanical equipment, in the actual application process of equipment with dynamic vibration deformation, in order to ensure the safety and stability of equipment operation, it is necessary to monitor the deformation of an object in real time. For example, with the rapid development of the amusement facility industry, the requirements for the safety and stability of amusement facilities are increasing day by day. During the operation of amusement facilities, deformations may occur due to various factors (such as wind force, load changes, etc.). If these deformations exceed the safe range, they will pose a serious threat to the lives and safety of tourists. Therefore, it is particularly important to perform high-precision deformation monitoring on amusement facilities.
[0003] However, traditional deformation measurement methods such as laser ranging and total station are often limited by the measurement range, environmental requirements, or high operation complexity and heavy equipment that is difficult to carry during close-range measurement, making it difficult to achieve real-time and accurate monitoring of the deformation of equipment. Summary of the Invention
[0004] The purpose of this application is to provide a method and system for dynamically monitoring the deformation of an object, which can closely and accurately monitor the deformation of the object to be measured in real time.
[0005] To achieve the above purpose, this application provides the following solutions:
[0006] In the first aspect, this application provides a method for dynamically monitoring the deformation of an object, including:
[0007] Collect multiple frames of images of the object to be measured at different times;
[0008] Perform target segmentation on each frame of image respectively to determine the mask position of the object to be measured in each frame of image;
[0009] Determine the pixel-level displacement of the object to be measured according to the mask position of the object to be measured in each frame of image;
[0010] Convert the pixel-level displacement into a displacement in the real coordinate system to determine the deformation of the object to be measured.
[0011] Optionally, collecting multiple frames of images of the object to be measured at different times specifically includes: using a binocular vision camera to collect multiple frames of images of the object to be measured at different times.
[0012] Optionally, target segmentation is performed on each frame of image respectively to determine the mask position of the object to be measured in each frame of image, specifically including: determining the region of interest in the first frame of image; performing visual segmentation on the object to be measured within the region of interest by using a segmentation diffusion algorithm to determine the visual mask of the object to be measured in the first frame of image; performing target segmentation on each frame of image respectively to determine the position of the visual mask in each frame of image, so as to obtain the mask position of the object to be measured in each frame of image.
[0013] Optionally, according to the mask position of the object to be measured in each frame of image, the pixel-level displacement of the object to be measured is determined, specifically including: determining the pixel-level displacement of the object to be measured according to the change in the mask position of the object to be measured in two adjacent frames of images.
[0014] Optionally, the object dynamic deformation monitoring method further includes: real-time detecting the motion state data of the binocular vision camera; determining the pose information of the binocular vision camera when collecting each frame of image according to the motion state data of the binocular vision camera and multiple frames of images of the object to be measured at different times; correcting the pixel-level displacement according to the pose information of the binocular vision camera when collecting each frame of image.
[0015] Optionally, the pose information includes a rotation matrix and a translation vector; the following formula is used to correct the pixel-level displacement: ΔD true = R·[ΔD - t]; where ΔD is the pixel-level displacement, ΔD true is the corrected pixel-level displacement, R is the rotation matrix, and t is the translation vector.
[0016] Optionally, real-time detecting the motion state data of the binocular vision camera specifically includes: using an inertial measurement unit to real-time detect the motion state data of the binocular vision camera.
[0017] In a second aspect, the present application provides an object dynamic deformation monitoring system, including:
[0018] An image acquisition device, configured to acquire multiple frames of images of the object to be measured at different times;
[0019] An image processing device, configured to perform target segmentation on each frame of image respectively to determine the mask position of the object to be measured in each frame of image; determine the pixel-level displacement of the object to be measured according to the mask position of the object to be measured in each frame of image; convert the pixel-level displacement into a displacement in a real coordinate system to determine the deformation of the object to be measured.
[0020] Optionally, the image acquisition device is a binocular vision camera.
[0021] Optionally, the binocular vision camera is mounted on a drone.
[0022] According to the specific embodiments provided by the present application, the present application has the following technical effects:
[0023] The present application provides a method and system for monitoring the dynamic deformation of an object. By collecting multiple frames of images of the object to be measured at different times, the continuous change process of the object deformation can be captured. The target segmentation technology is used to determine the mask position of the object to be measured in each frame of the image, and the contour of the object to be measured can be accurately extracted, so as to accurately judge its deformation situation. The pixel-level displacement is converted into the displacement in the real coordinate system, realizing the quantitative analysis of dynamic deformation monitoring, making the deformation monitoring results more intuitive and easy to understand, and providing a strong guarantee for the safe operation of the equipment. The present application realizes dynamic deformation monitoring through image processing, and can improve the accuracy and real-time performance of object dynamic deformation monitoring in a short-distance environment. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.
[0025] Figure 1 It is a schematic diagram of the overall process of a method for monitoring the dynamic deformation of an object provided in an embodiment of the present application;
[0026] Figure 2 It is a schematic diagram of the detailed process of a method for monitoring the dynamic deformation of an object provided in an embodiment of the present application;
[0027] Figure 3 It is a schematic diagram of the selection of the region of interest and the visual mask recognition result of the Ferris wheel in an embodiment of the present application;
[0028] Figure 4 It is a schematic diagram of the deformation measurement result of the axial components of the Ferris wheel in an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS
[0029] The following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, rather than all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present application.
[0030] The purpose of the present application is to provide a method that can perform high-precision and continuous non-target optical deformation measurement on the object to be measured at a relatively short distance (within 15 m).
[0031] In recent years, visual imaging and visual measurement technologies have been widely used in various fields, especially showing great potential in the field of optical measurement. As an image processing technology based on the principle of human binocular vision, binocular vision technology can simulate the human binocular vision system, use two cameras to capture images from different perspectives, and obtain the depth information and deformation of an object by calculating the image differences. This technology not only has the advantages of non-contact and strong real-time performance, but also has low cost and portability, and can achieve high-precision deformation measurement in a non-target form at a relatively short distance. This application combines advanced visual segmentation technology, which can further improve the accuracy and stability of measurement. In addition, measurement errors are effectively reduced through pixel-level displacement measurement and camera pose compensation, improving measurement accuracy.
[0032] To make the above objects, features, and advantages of this application more obvious and understandable, the following further details this application in conjunction with the accompanying drawings and specific embodiments.
[0033] In an exemplary embodiment, as Figure 1 and Figure 2 shown, a method for monitoring the dynamic deformation of an object is provided. This method is executed by a computer device, and specifically can be executed alone by a computer device such as a terminal or a server, or jointly executed by a terminal and a server. The method for monitoring the dynamic deformation of an object includes the following steps 101 to 104.
[0034] Step 101, collect multiple frames of images of the object to be measured at different times.
[0035] In a specific application example, a binocular vision camera is used to collect multiple frames of images of the object to be measured at different times. This application only relies on a binocular vision camera to achieve close-range measurement of the vibration deformation of a dynamic object.
[0036] In actual application, first calibrate the binocular vision camera. Through calibration, obtain the internal parameters (including corner points, principal point coordinates, etc.) and distortion parameters of the binocular vision camera. Ensure that the image coordinates can be accurately mapped to the real-world coordinate system in subsequent calculations through the calibration process of the binocular vision camera.
[0037] This application can monitor the deformation of the object to be measured in a short period of time or in a long period of time. Specifically, it can be carried out by a drone carrying a binocular vision camera, or the binocular vision camera can be fixed at a set position.
[0038] Step 102, perform target segmentation on each frame of image respectively to determine the mask position of the object to be measured in each frame of image.
[0039] In a specific application example, the region of interest in the first frame image is determined. The segmentation diffusion algorithm is used to visually segment the object to be measured within the region of interest, and the visual mask of the object to be measured in the first frame image is determined. Target segmentation is performed on each frame of the image respectively, and the position of the visual mask in each frame of the image is determined to obtain the mask position of the object to be measured in each frame of the image.
[0040] In actual application, through the Graphical User Interface (GUI) running on the computer terminal, the user selects a region of interest in the first frame image. In the graphical user interface, the user can use input means such as a mouse or a touch screen to demarcate the region of interest in the first frame image. This region of interest will clarify the spatial range of subsequent analysis and measurement, thus avoiding redundant calculations for the entire scene and saving computing resources for subsequent processing.
[0041] After the determination of the region of interest is completed, the fine-tuned and trained SegmentAnything Model2 is called to visually segment the object to be measured within the region of interest. The segmentation diffusion model is based on deep learning methods and can automatically identify and segment the object to be measured based on a simple enhancement of the first frame image. Through this process, a visual mask corresponding to the object to be measured can be obtained, and this visual mask is presented in the form of a specific region in the image, effectively distinguishing the object to be measured from the background.
[0042] Step 103: Determine the pixel-level displacement of the object to be measured according to the mask position of the object to be measured in each frame of the image.
[0043] In a specific application example, the pixel-level displacement of the object to be measured is determined according to the change in the mask position of the object to be measured in two adjacent frames of the image. Specifically, after obtaining the visual mask, the visual mask is tracked in each frame of the image, that is, whenever a new frame of the image is acquired, the pixel-level displacement value is calculated by comparing the position change of the visual mask in the current frame and the previous frame.
[0044] In actual applications, the binocular vision camera itself is not always in a completely static state. Environmental vibrations, carrier movements, and slight movements of the operator will all cause changes in the camera pose. These unstable factors will introduce pose estimation errors, thus affecting the accuracy and consistency of optical deformation measurement. For this reason, this application introduces an Inertial Measurement Unit (IMU) to monitor the motion status of the binocular vision camera in real time and combines it with the visual optical flow information to compensate for the pose of the binocular vision camera.
[0045] Before performing step 104, the object dynamic deformation monitoring method further includes the following steps 201 to 203.
[0046] Step 201: Real-time detect the motion state data of the binocular vision camera. Specifically, use an inertial measurement unit to real-time detect the motion state data of the binocular vision camera. The IMU can continuously output data such as the acceleration, angular velocity, and azimuth change of the binocular vision camera, providing necessary references for pose compensation.
[0047] Step 202: Determine the pose information of the binocular vision camera when collecting each frame of image according to the motion state data of the binocular vision camera and multiple frames of images of the object to be measured at different times. The pose information includes a rotation matrix and a translation vector.
[0048] The motion state data collected by the IMU provides timely feedback for the short-term pose change of the binocular vision camera, assisting the optical flow operation result for precise pose correction. The mutual complementation of these two types of information can make the pose estimation remain highly stable and accurate when encountering insufficient texture or rapid movement, and can effectively counteract the jitter and drift of the binocular vision camera in a rapidly changing environment.
[0049] Considering the requirements of image resolution and measurement accuracy, this application can use the Simultaneous Localization and Mapping (SLAM) technology to obtain more refined displacement information. According to the motion state data of the binocular vision camera and multiple frames of images of the object to be measured at different times, use the SLAM technology to determine the pose of the binocular vision camera.
[0050] Step 203: Correct the pixel-level displacement according to the pose information of the binocular vision camera when collecting each frame of image. Specifically, use the formula ΔD true = R·[ΔD - t] to correct the pixel-level displacement. Where, ΔD is the pixel-level displacement, including the displacement in the x direction and the displacement in the y direction, and ΔD true is the corrected pixel-level displacement, R is the rotation matrix, and t is the translation vector.
[0051] Through the correction process of the above steps 201 to 203, when applied to the scenario of an unmanned aerial vehicle (UAV) equipped with a binocular vision camera, no additional fixing device is required, and high-precision optical deformation measurement can still be achieved. Whether it is the interference of wind force, the uneven weight of the UAV, or other vibration factors resulting in small offsets, they can all be effectively suppressed by the above correction method, thus providing a solid foundation for subsequent measurement and analysis.
[0052] Step 104: Convert the pixel-level displacement into the displacement in the real coordinate system to determine the deformation of the object to be measured.
[0053] To achieve deformation measurement in the real world sense, this application uses the internal parameters and distortion parameters of a binocular vision camera to map pixel-level displacements to the real world coordinate system. Specifically, the formula disc = disi × l × z / f is used to obtain the displacement of the object to be measured in the real coordinate system. Among them, dis c is the displacement of the object to be measured in the real coordinate system. dis i is the pixel-level displacement, l is the physical pixel size of the binocular vision camera, z is the distance between the optical center of the binocular vision camera and the object to be measured, and f is the focal length of the binocular vision camera. In the above conversion process, according to the camera imaging model, the pixel coordinates can be converted into the actual length scale in meters or millimeters, and the real displacements of the deformation of the object to be measured in the x direction and the y direction can be obtained. Furthermore, by analyzing and calculating consecutive frame images over a period of time, the cumulative offset or deformation of the object to be measured relative to the initial state can be obtained.
[0054] The object dynamic deformation monitoring method provided by this application is applicable to measuring the jitter displacement of components in various scenarios such as amusement facilities, construction machinery, building components, and even scientific research experiments without the assistance of a target. It ensures high measurement accuracy and high deployment flexibility in a relatively short distance environment within 15 meters. In a specific application example, as Figure 3 shown, the blue frame is the region of interest manually selected, and the masked area in the blue frame is the visual mask of the object to be measured. As Figure 4 shown, the axial component of the Ferris wheel has a displacement of 2.5 mm in the first frame image, 20.64 mm in the second frame image, 38.70 mm in the third frame image, and 30.96 mm in the fourth frame image.
[0055] In summary, the beneficial effects of this application are:
[0056] First, this application effectively solves the problem in the prior art that it is difficult to achieve continuous and accurate optical deformation measurement of any target object at a relatively short distance (within 15 meters). The prior art is often limited by the measurement range or insufficient environmental adaptability, and often requires additional conditions such as a target or a fixed measurement distance, and cannot meet the requirements of real-time deformation detection at close range. In this application, through precise calibration of the binocular vision camera, the region of interest is specified using the GUI interface and combined with the fine-tuned training of SAM2 for accurate visual segmentation of the object to be measured, thereby obtaining a high-quality visual mask. On this basis, high-precision displacement analysis is performed on this visual mask in multiple frames of images, and the pixel-level displacement is converted into the displacement in the real-world coordinates using the internal parameters of the binocular vision camera, so as to achieve fine deformation detection of any object at a relatively short distance. Compared with the prior art, this application can operate stably under various complex backgrounds and lighting conditions without the need for additional special treatment of the environment or the target surface, realizing target-free optical measurement, effectively reducing the operation complexity, and improving the practicability.
[0057] Second, this application improves the robustness and stability of the measurement process to the pose changes of the binocular vision camera. Traditional methods are prone to measurement errors or require complex calibration and post-processing when dealing with camera jitter or pose offset, and it is difficult to obtain continuous and accurate deformation data. This application integrates an inertial measurement unit to monitor the pose of the binocular vision camera in real time, and uses optical flow and related pose compensation algorithms for fusion processing. Even in an environment where the binocular vision camera is unstable or slightly vibrating, the high reliability of the measurement results can be maintained. Without the need for additional mechanical stabilization devices, or under the condition that the mechanical structure is not completely stable, it can also effectively cope with problems such as camera movement, carrier shaking, and unstable operation, ensuring that the finally output deformation measurement data is stable, continuous, and highly accurate.
[0058] Based on the same inventive concept, the embodiment of this application also provides an object dynamic deformation monitoring system for implementing the object dynamic deformation monitoring method involved above. The solution provided by this system to solve the problem is similar to the solution described in the above method. Therefore, the specific limitations in one or more embodiments of the object dynamic deformation monitoring system provided below can refer to the limitations on the object dynamic deformation monitoring method in the above text, and will not be repeated here.
[0059] In an exemplary embodiment, an object dynamic deformation monitoring system is provided, including: an image acquisition device and an image processing device.
[0060] The image acquisition device is used to acquire multiple frames of images of the object to be measured at different times. In a specific application example, the image acquisition device is a binocular vision camera. The binocular vision camera is mounted on a drone.
[0061] The image processing device is used to perform target segmentation on each frame of image respectively, and determine the mask position of the object to be measured in each frame of image. According to the mask position of the object to be measured in each frame of image, the pixel-level displacement of the object to be measured is determined. The pixel-level displacement is converted into a displacement in the real coordinate system to determine the deformation of the object to be measured.
[0062] In a specific application example, the image processing device is a main control computing unit (such as an industrial control computer or a high-performance computer).
[0063] In addition, the object dynamic deformation monitoring system further includes an inertial measurement unit, a storage unit, and a display unit. The inertial measurement unit is used to detect the motion state data of the binocular vision camera. The image processing device stores the final deformation measurement result in the storage unit and presents it to the user on the display unit or the GUI interface.
[0064] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data need to comply with relevant regulations.
[0065] In this application, all actions of obtaining signals, information, or data are carried out on the premise of complying with the corresponding data protection regulations and policies of the country where the device is located and obtaining the authorization given by the owner of the corresponding device.
[0066] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, database, or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memories. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.
[0067] The technical features of the above embodiments can be combined arbitrarily. For the sake of concise description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope recorded in this specification.
[0068] Specific examples are used in this article to elaborate on the principles and implementation manners of the present application. The descriptions of the above embodiments are only used to help understand the method and its core idea of the present application; at the same time, for those of ordinary skill in the art, according to the idea of the present application, there will be changes in the specific implementation manners and application scopes. In summary, the content of this specification should not be construed as a limitation to the present application.
Claims
1. A method for monitoring the dynamic deformation of an object, characterized in that, The object dynamic deformation monitoring method includes: Collecting multiple frames of images of the object to be measured at different times; Performing target segmentation on each frame of image respectively to determine the mask position of the object to be measured in each frame of image; Determining the pixel-level displacement of the object to be measured according to the mask position of the object to be measured in each frame of image; Converting the pixel-level displacement into a displacement in the real coordinate system to determine the deformation of the object to be measured.
2. The method for monitoring dynamic deformation of an object according to claim 1, wherein, Collecting multiple frames of images of the object to be measured at different times specifically includes: Using a binocular vision camera to collect multiple frames of images of the object to be measured at different times.
3. The method for monitoring dynamic deformation of an object according to claim 1, wherein Performing target segmentation on each frame of image respectively to determine the mask position of the object to be measured in each frame of image specifically includes: Determining the region of interest in the first frame of image; Performing visual segmentation on the object to be measured in the region of interest using a segmentation diffusion algorithm to determine the visual mask of the object to be measured in the first frame of image; Performing target segmentation on each frame of image respectively to determine the position of the visual mask in each frame of image, so as to obtain the mask position of the object to be measured in each frame of image.
4. The method for monitoring dynamic deformation of an object according to claim 1, characterized in that, Determining the pixel-level displacement of the object to be measured according to the mask position of the object to be measured in each frame of image specifically includes: Determining the pixel-level displacement of the object to be measured according to the change in the mask position of the object to be measured in two adjacent frames of images.
5. The method for monitoring dynamic deformation of an object according to claim 2, wherein The object dynamic deformation monitoring method further includes: Real-time detecting the motion state data of the binocular vision camera; Determining the pose information of the binocular vision camera when collecting each frame of image according to the motion state data of the binocular vision camera and multiple frames of images of the object to be measured at different times; Correcting the pixel-level displacement according to the pose information of the binocular vision camera when collecting each frame of image.
6. The method for monitoring the dynamic deformation of an object according to claim 5, wherein, The pose information includes a rotation matrix and a translation vector; Using the following formula to correct the pixel-level displacement: ΔD true = R·[ΔD - t]; where ΔD is the pixel-level displacement, and ΔD true is the corrected pixel-level displacement, R is the rotation matrix, and t is the translation vector.
7. The method for monitoring the dynamic deformation of an object according to claim 5, characterized in that Real-time detecting the motion state data of the binocular vision camera specifically includes: Using an inertial measurement unit to real-time detect the motion state data of the binocular vision camera.
8. An object dynamic deformation monitoring system, which is applied to the object dynamic deformation monitoring method described in any one of claims 1-7, and is characterized in that, The object dynamic deformation monitoring system includes: An image acquisition device for collecting multiple frames of images of the object to be measured at different times; An image processing device for performing target segmentation on each frame of image respectively to determine the mask position of the object to be measured in each frame of image; determining the pixel-level displacement of the object to be measured according to the mask position of the object to be measured in each frame of image; converting the pixel-level displacement into a displacement in the real coordinate system to determine the deformation of the object to be measured.
9. The object dynamic deformation monitoring system according to claim 8, wherein The image acquisition device is a binocular vision camera.
10. The object dynamic deformation monitoring system according to claim 9, characterized in that, The binocular vision camera is mounted on a drone.