Mechanical deformation augmented reality simulation method and system based on pixel-oriented finite element

Abstract

The invention discloses a mechanical deformation augmented reality simulation method and system based on a pixel-oriented finite element, and the method comprises the steps: obtaining a multi-angle image of a simulation object in an undeformed state, and building a bidirectional geometric mapping relation between a world coordinate system and an image plane; establishing a three-dimensional finite element model of the simulation object; constructing a visibility classification matrix; solving three-dimensional physical displacement of finite element nodes of a simulation object in a loaded state, mapping the three-dimensional physical displacement to a two-dimensional image coordinate system, calculating two-dimensional pixel displacement of each finite element node, and interpolating to generate a global pixel displacement field; mapping original image pixels of the visible part of the simulation object to deformed sub-pixel positions, and distributing pixel brightness of the original image to adjacent pixels; and correcting the deformed image after brightness migration, and performing brightness processing on the background area of the corrected deformed image to obtain a simulation deformed image. According to the invention, high-precision live-action simulation of structural deformation can be realized, and the visual authenticity and engineering applicability of mechanical deformation simulation are remarkably improved.

Description

Technical Field

[0001] The present invention relates to a mechanical deformation augmented reality simulation method and system based on pixel-guided finite element, belonging to the intersection of computer vision and computational mechanics. Background Art

[0002] The finite element method (FEM), a core technology in computational mechanics, is widely used in aerospace, mechanical engineering, civil engineering, and biomedical engineering to analyze and predict the behavior of physical systems. Simultaneously, computer vision (CV), through the implementation of automated image processing algorithms, advanced numerical simulation techniques, and improved human-computer interaction frameworks, has become a key tool for improving the efficiency, accuracy, and usability of FEM analysis. The visual properties of real scenes and objects provide new perspectives and representations for traditional FEM analysis and visualization, thereby expanding its potential applications in scientific research and engineering practice.

[0003] Current research on the integration of finite element and computer vision mainly includes the following four methods:

[0004] (1) Digital Image Correlation (DIC): The digital image correlation method verifies the finite element model by preparing a speckle pattern on the surface of the structure and analyzing the brightness changes of the image before and after loading. For example, by quantifying the displacement distribution of an aluminum alloy specimen under tensile load, the local strain phenomenon related to the Portevin-Le Chatelier band can be characterized; in addition, when using DIC technology to study the fatigue crack propagation behavior of materials, the accuracy of the model can be verified by comparing experimental and finite element results. However, DIC technology relies on artificial speckle to obtain digital image data, and its analysis process has no direct connection with the original surface texture, actual brightness, and background environment of the object. Therefore, there is no direct data flow interaction between DIC and finite element.

[0005] (2) Image-driven geometric modeling technology: Based on image processing technology, researchers can extract geometric and material data from images and quickly construct finite element models of complex three-dimensional structures. For example, by combining X-ray computed tomography (CT) with the finite element method, the compressive strength of concrete specimens can be evaluated, verifying the effectiveness of image processing technology in constructing refined three-dimensional finite element models of heterogeneous materials, thereby significantly improving the consistency between numerical predictions and experimental results. However, the geometric information extracted from the image in this method is only used in the finite element modeling stage and does not participate in the solution of the finite element equations and the post-processing process. This limitation makes the representation of changes in visual properties such as surface texture and brightness of objects insufficient, resulting in a decrease in the accuracy of real scene reconstruction and weakening the visual fidelity of finite element simulation.

[0006] (3) Synthetic image generation technology: Traditional synthetic image generation methods are mainly aimed at virtual reality applications, focusing on rendering visual effects such as lighting and texture. However, they lack realism when simulating dynamic phenomena such as object deformation and damage. The introduction of the finite element method has effectively improved such defects. For example, nonlinear finite element simulation of post-earthquake building damage and generation of damage images that conform to physical laws, or combined with Blender graphics software to achieve load deformation rendering of mechanical structures. However, existing methods still lack the ability to restore the characteristic details of real objects and environmental context information, and their accuracy and detail expression still need to be further improved.

[0007] (4) Augmented Reality Finite Element (AR-FEM): Although traditional finite element methods can provide numerical results of the physical behavior of objects, they do not consider the integration of the surrounding environment. Augmented reality (AR) technology significantly improves visualization effects and supports real-time interaction by superimposing finite element results on the real environment. For example, modal analysis results can be dynamically presented on physical objects, enhancing engineers' understanding of complex mechanical behavior. Although AR-FEM improves the intuitiveness and immersion of finite element result interpretation, it focuses on the integration of the surrounding environment and fails to fully preserve the visual properties of the object surface, such as texture and brightness.

[0008] Therefore, although the digital image correlation method can be associated with the finite element results by analyzing the geometric deformation of artificial labels, it lacks direct data flow interaction; image-driven geometric modeling technology extracts geometric information from the real scene to construct the finite element model, but ignores the background environment; finite element-based synthetic image generation technology can render virtual deformed objects, but loses the visual attributes and background context of the real scene; although augmented reality finite element realizes the superposition of virtual objects and real backgrounds, it fails to retain the geometric details and visual features of real objects. Summary of the Invention

[0009] In view of this, the present invention provides a method, system, computer equipment and storage medium for mechanical deformation augmented reality simulation based on pixel-guided finite elements. It addresses the problems of one-way interaction between traditional finite element methods and computer vision technology, insufficient data fusion, and neglect of visual features and background. It can establish a two-way data channel between finite element calculations and real scenes, achieve high-precision real-scene simulation of structural deformation, and significantly improve the visual authenticity and engineering applicability of mechanical deformation simulation.

[0010] The first object of the present invention is to provide a mechanical deformation augmented reality simulation method based on pixel-guided finite element.

[0011] The second object of the present invention is to provide a mechanical deformation augmented reality simulation system based on pixel-guided finite elements.

[0012] A third object of the present invention is to provide a computer device.

[0013] A fourth object of the present invention is to provide a storage medium.

[0014] The first object of the present invention can be achieved by adopting the following technical solutions:

[0015] A mechanical deformation augmented reality simulation method based on pixel-guided finite element, the method comprising:

[0016] Acquire multi-angle images of the simulated object in its undeformed state, use the Zhang Zhengyou calibration method to identify the camera's intrinsic and extrinsic parameter matrices, and establish a bidirectional geometric mapping relationship between the world coordinate system and the image plane;

[0017] Based on the geometric features and material properties of the simulation object, a three-dimensional finite element model of the simulation object is established, and the three-dimensional coordinates of each finite element node are based on the world coordinate system;

[0018] Construct a visibility classification matrix that matches the image resolution, calculate and sort the depth values ​​of the finite element nodes in the camera coordinate system, and update the visibility classification matrix based on the occlusion of each finite element node to distinguish the visible part of the foreground object from the background area.

[0019] Solve the 3D physical displacement of the finite element nodes of the simulation object under load, map the 3D physical displacement to the 2D image coordinate system, calculate the 2D pixel displacement of each finite element node, and interpolate to generate the global pixel displacement field;

[0020] Map the original image pixels of the visible part of the simulated object to the deformed sub-pixel positions according to the global pixel displacement field, and distribute the pixel brightness of the original image to adjacent pixels according to the area ratio to achieve the migration of the original image brightness;

[0021] The deformed image after brightness migration is corrected, and the background area of ​​the corrected deformed image is brightness processed according to the visibility classification matrix and the pixel brightness of the original image to obtain a simulated deformed image.

[0022] Furthermore, the Zhang Zhengyou calibration method is used to identify the camera's intrinsic parameter matrix and extrinsic parameter matrix, and establish a bidirectional geometric mapping relationship between the world coordinate system and the image plane, specifically including:

[0023] Based on Zhang Zhengyou's calibration method, a calibration plate is used as a reference plane, and the spacing between the checkerboard corners is defined as a known physical size. The calibration plate covers the entire image plane and ensures that the checkerboard plane is parallel to the surface of the simulated object or at a known angle.

[0024] Use the Harris corner detection algorithm to extract the pixel coordinates of the corner points of the calibration plate and record the corresponding coordinates of the pixel coordinates in the world coordinate system;

[0025] The reprojection error is minimized by a nonlinear optimization algorithm, the intrinsic and extrinsic parameter matrices of the camera are iteratively solved, and a bidirectional geometric mapping relationship between the world coordinate system and the image plane is established.

[0026] Furthermore, the depth values ​​of the finite element nodes in the camera coordinate system are calculated and sorted, and the visibility classification matrix is ​​updated according to the occlusion of each finite element node to distinguish the visible part of the foreground object from the background area, specifically including:

[0027] The finite element node coordinates are converted into camera coordinates through matrix transformation, and the depth values ​​of all finite element nodes are calculated, and all finite element nodes are arranged in ascending order according to the depth values;

[0028] Traverse the sorted finite element node queue. If the current finite element node is not blocked, mark the current finite element node as a visible node and add it to the visible node list. Update the visible node classification value to 0. If the current finite element node is blocked, mark it as invisible and continue processing the next node.

[0029] If the current finite element node is visible, and the finite element unit where the current finite element node is located contains at least two visible nodes other than the current finite element node, then a minimum enclosing convex polygon is constructed based on the two-dimensional projection coordinates of the current finite element node and other visible nodes, and the pixel area covered by the convex polygon is marked as visible, and the classification value of the area is updated to 0;

[0030] Through iterative updating, the visibility classification matrix is ​​updated according to the classification value of each finite element node to distinguish the visible part of the foreground object from the background area.

[0031] Furthermore, solving the three-dimensional physical displacement of the finite element nodes of the simulation object under load, mapping the three-dimensional physical displacement to a two-dimensional image coordinate system, calculating the two-dimensional pixel displacement of each finite element node, and generating a global pixel displacement field specifically includes:

[0032] Perform stress analysis on the simulation object and solve the three-dimensional physical displacement of the finite element nodes of the simulation object under load;

[0033] Based on the pinhole imaging model, the three-dimensional physical displacement of the finite element nodes is converted into two-dimensional pixel displacement in the image coordinate system, and the global pixel displacement field is generated through shape function interpolation.

[0034] Furthermore, the method of mapping the original image pixels of the visible part of the simulated object to the deformed sub-pixel positions according to the global pixel displacement field and distributing the pixel brightness of the original image to adjacent pixels according to the area ratio to achieve the migration of the original image brightness specifically includes:

[0035] Based on the global pixel displacement field, each pixel in the original image is mapped to a new position in the deformed image, and new visual content is generated through brightness diffusion and superposition;

[0036] Assign brightness weight factors to adjacent pixels according to their area ratios, distribute the pixel brightness of the original image to the adjacent pixels based on the weight factors, and superimpose the distributed pixel brightness on the brightness field of the deformed image.

[0037] Furthermore, the correction of the deformed image after brightness migration specifically includes:

[0038] For the migrated image, the brightness weight factor of each pixel in the brightness field is normalized to keep the total brightness of the image constant, and a threshold filter is set to remove edge jaggedness.

[0039] Furthermore, the brightness processing of the background area of ​​the corrected deformed image according to the visibility classification matrix and the pixel brightness of the original image specifically includes:

[0040] According to the visibility classification matrix, for the persistent visible background that is not occluded in both the original image and the corrected deformed image, the persistent visible background of the corrected deformed image is made to directly inherit the pixel brightness of the original image. For the background that is occluded in the original image but newly revealed in the corrected deformed image, the pixel brightness of the newly revealed background in the corrected deformed image is obtained by interpolation of the neighboring area excluding the visible part of the object.

[0041] The third object of the present invention can be achieved by adopting the following technical solutions:

[0042] A mechanical deformation augmented reality simulation system based on pixel-guided finite element, the system comprising:

[0043] The first establishment module is used to obtain multi-angle images of the simulated object in the undeformed state, use the Zhang Zhengyou calibration method to identify the camera's intrinsic parameter matrix and extrinsic parameter matrix, and establish a bidirectional geometric mapping relationship between the world coordinate system and the image plane;

[0044] The second establishment module is used to establish a three-dimensional finite element model of the simulation object based on the geometric characteristics and material properties of the simulation object, and the three-dimensional coordinates of each finite element node are based on the world coordinate system;

[0045] The first calculation module is used to construct a visibility classification matrix that matches the image resolution, calculate and sort the depth values ​​of the finite element nodes in the camera coordinate system, and update the visibility classification matrix based on the occlusion of each finite element node to distinguish the visible part of the foreground object from the background area;

[0046] The second calculation module is used to solve the three-dimensional physical displacement of the finite element nodes of the simulation object under load, map the three-dimensional physical displacement to the two-dimensional image coordinate system, calculate the two-dimensional pixel displacement of each finite element node, and interpolate to generate a global pixel displacement field;

[0047] A migration module is used to map the original image pixels of the visible part of the simulated object to the deformed sub-pixel positions according to the global pixel displacement field, and to distribute the pixel brightness of the original image to adjacent pixels according to the area ratio to achieve the migration of the original image brightness;

[0048] The background processing module is used to correct the deformed image after brightness migration, and perform brightness processing on the background area of ​​the corrected deformed image according to the visibility classification matrix and the pixel brightness of the original image to obtain a simulated deformed image.

[0049] The third object of the present invention can be achieved by adopting the following technical solutions:

[0050] A computer device includes a processor and a memory for storing a program executable by the processor. When the processor executes the program stored in the memory, the above-mentioned fall monitoring method is implemented.

[0051] The fourth object of the present invention can be achieved by adopting the following technical solutions:

[0052] A storage medium stores a program, and when the program is executed by a processor, the above-mentioned fall monitoring method is implemented.

[0053] The present invention has the following beneficial effects compared to the prior art:

[0054] This invention combines the traditional finite element method with computer vision technology to establish a two-way data channel between finite element calculations and real scenes. In combination with multiple image processing algorithms, it realizes mechanical deformation augmented reality simulation that integrates geometric features, visual attributes and real background environments, improves the ability to generate high-quality simulated images, enhances the applicability of the finite element method in real scenes, and creates a new paradigm for real-scene finite element applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying any creative work.

[0056] Figure 1 This is a schematic diagram of the mechanical deformation augmented reality simulation method based on pixel-guided finite element according to Example 1 of the present invention.

[0057] Figure 2 This is a flow chart of a mechanical deformation augmented reality simulation method based on pixel-guided finite element according to embodiment 1 of the present invention.

[0058] Figure 3 This is a layout diagram of the loading device of Example 1 of the present invention.

[0059] Figure 4 This is a flowchart of object imaging visibility analysis according to embodiment 1 of the present invention.

[0060] Figure 5 (a)~ Figure 5 (c) is a schematic diagram of the sub-pixel brightness migration method according to Example 1 of the present invention.

[0061] Figure 6 (a)~ Figure 6 (c) is a schematic diagram of the image background brightness correction processing according to Example 1 of the present invention.

[0062] Figure 7 (a)~ Figure 7 (i) is a comparison diagram of the simulated deformed image obtained in Example 1 of the present invention, the original image, and the real deformed image.

[0063] Figure 8 This is a comparison diagram of the MSE of the simulated deformed image obtained in Example 1 of the present invention, the original image, and the real deformed image.

[0064] Figure 9 This is a structural block diagram of a mechanical deformation augmented reality simulation system based on pixel-guided finite elements according to embodiment 2 of the present invention.

[0065] Figure 10 This is a structural block diagram of a computer device according to embodiment 3 of the present invention. DETAILED DESCRIPTION

[0066] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0067] Example 1:

[0068] like Figure 1 and Figure 2As shown, this embodiment provides a mechanical deformation augmented reality simulation method based on pixel-guided finite element. The method is described by taking the bending deformation simulation of 6061 aluminum sheet as an example, and specifically includes the following steps:

[0069] S201, obtaining multi-angle images of the simulation object in an undeformed state, using the Zhang Zhengyou calibration method to identify the camera's intrinsic parameter matrix and extrinsic parameter matrix, and establishing a bidirectional geometric mapping relationship between the world coordinate system and the image plane.

[0070] This embodiment is realized by constructing a mechanical deformation augmented reality simulation architecture, which includes a loading device, an image acquisition module and a computing platform. The loading device is arranged as follows: Figure 3 As shown, a cantilever beam structure is used with one end fixed and a 1.5 kg counterweight suspended at the other end. The counterweight suspension point is 1 cm away from the free end to generate a pure bending load. Image acquisition is performed using a smartphone with a resolution of 4032×3024 fixed to a tripod. At the same time, a 50 mm gauge length electronic extensometer is installed in the middle of the specimen to measure the elastic modulus of the aluminum sheet and provide mechanical parameters for finite element calculations. The software level of the computing platform integrates the Abaqus finite element solver and the mechanical deformation augmented reality simulation technology based on pixel-guided finite elements. The finite element model is meshed using C3D8 units. The simulation object of this embodiment is an aluminum sheet specimen with a size of 25 mm × 1.8 mm × 150 mm. The characters "6+3=9" are marked in the middle of the specimen to enhance deformation visualization.

[0071] Before the simulated object is deformed, use the smartphone camera to capture multiple images containing a checkerboard calibration plate at different orientations and angles. The calibration plate must cover the entire image plane and ensure that the checkerboard plane is parallel to the surface of the simulated object or at a known angle to provide sufficient 2D-3D spatial correspondence points.

[0072] Based on Zhang Zhengyou's calibration method, the calibration plate is used as the reference plane, and the spacing between the checkerboard corner points is defined as a known physical size. Then, the Harris corner detection algorithm is used to extract the pixel coordinates of the corner points of the calibration plate, and their corresponding coordinates in the physical world coordinate system are recorded. Finally, a nonlinear optimization algorithm is used to minimize the reprojection error and iteratively solve the camera's intrinsic and extrinsic parameter matrices.

[0073] After the calibration is completed, the camera intrinsic parameter matrix and extrinsic parameter matrix are output to establish a projection mapping model from the physical world to the image plane. That is, a bidirectional geometric mapping relationship between the world coordinate system and the image plane is established to provide geometric constraints for the subsequent bidirectional calculation of the pixel displacement field, as shown in the following formula:

[0074]

[0075] Where s is the camera scale factor at the imaging point, (x, y, z) is the three-dimensional point in the world coordinate system, (u, v) is the corresponding two-dimensional image pixel coordinate, K is the camera intrinsic parameter matrix, R and T are the rotation matrix and translation vector respectively, and the expressions of K, R and T are:

[0076]

[0077] T=[T x ,T y ,T z ] T (4)

[0078] Among them, f u 、f v is the camera axial focal length, is the image axis skew parameter, (u i ,v0) is the coordinate of the optical center of the image coordinate system, α, β, γ are Euler angles, (T x ,T y ,T z ) is the camera's world coordinate position.

[0079] S202: Based on the geometric features and material properties of the simulation object, a three-dimensional finite element model of the simulation object is established.

[0080] This embodiment establishes a three-dimensional finite element model of the simulation object based on its geometric features and material properties, and imports finite element node coordinate data in an initial undeformed state. The three-dimensional coordinates of each finite element node are based on the world coordinate system.

[0081] S203: Construct a visibility classification matrix that matches the image resolution, calculate and sort the depth values ​​of the finite element nodes in the camera coordinate system, and update the visibility classification matrix based on the occlusion conditions of each finite element node to distinguish the visible part of the foreground object from the background area.

[0082] like Figure 4 As shown in Figure 5, the three-dimensional object imaging visibility problem is converted into a two-dimensional pixel occlusion problem, and a classification matrix V consistent with the image resolution is constructed, where the area with a classification value of 1 represents the background, and the area with a classification value of 0 represents the visible part of the simulation object. The initial values ​​are all set to 1. The finite element node coordinates (x, y, z) are converted into camera relative coordinates (X, Y, Z) through matrix transformation, and the depth value of the finite element node is calculated. The depth value calculation formula is shown in Formula (5). The Z-axis coordinate represents the depth from the point to the camera optical center. The smaller the value, the closer it is to the camera.

[0083]

[0084] All finite element nodes are sorted in ascending order by depth value, and the finite element nodes closest to the camera are processed first to form a depth-first processing queue. The sorted finite element node queue is then traversed. First, the visibility of the finite element node is determined. The finite element node q(x, y, z) is converted to q(u, v) using formula (1). If the value of the current node q in the classification matrix V is the initial value 1, indicating that the node is not blocked and belongs to the visible part of the simulation object, the current finite element node is marked as a visible node and added to the visible node list, and the visible node classification value is updated to 0. If the value of the current node q in the classification matrix V is 0, indicating that the node is blocked by the visible part and belongs to the invisible part of the simulation object, it is marked as an invisible node and the next node is processed.

[0085] For nodes classified as visible, they need to be connected to other visible nodes to expand the visible area. If the finite element where the current visible node is located contains at least two visible nodes q i If (i = 1, 2, ...) (excluding the current finite element node), a minimum enclosing convex polygon D is constructed based on the 2D projection coordinates of the current finite element node and other visible finite element nodes. The pixel area covered by the convex polygon is marked as visible, and the classification value of this area is updated to 0. Through iterative updates, based on the classification values ​​of each finite element node, a complete visibility classification matrix is ​​ultimately output to distinguish the visible parts of foreground objects from the background areas, providing a pixel-level mask for subsequent brightness migration.

[0086] S204, solving the three-dimensional physical displacement of the finite element nodes of the simulation object under load, mapping the three-dimensional physical displacement to a two-dimensional image coordinate system, calculating the two-dimensional pixel displacement of each finite element node, and interpolating to generate a global pixel displacement field.

[0087] This embodiment uses finite element analysis software (Abaqus finite element solver) to perform force analysis on the simulation object and solve the three-dimensional physical displacement of the finite element nodes of the simulation object under load. Assuming that the number of finite element nodes is n (i=1, 2, ..., n), the displacement vector of each finite element node is output:

[0088] D xyz =[Δx1,Δy1,Δz1,…,Δx n ,Δy n ,Δz n ] T (6)

[0089] Based on the pinhole imaging model, the three-dimensional physical displacement of the finite element node is converted into a two-dimensional pixel displacement in the image coordinate system. When the structure undergoes real deformation, the image point (u, v) moves to The pixel displacement expression is:

[0090]

[0091] Among them, s, is the proportional factor before and after deformation, Δx, Δy, Δz are the corresponding physical displacements, based on formula (7), s and The relationship can be expressed as:

[0092]

[0093] in, is the third row element of R.

[0094] make Rewrite equation (7) as:

[0095]

[0096] Where H = KR.

[0097] For each finite element unit, the three-dimensional physical displacement of each finite element node can be expressed as pixel displacement:

[0098]

[0099] Wherein, the subscript i represents the i-th finite element node of the finite element unit.

[0100] The pixel displacement vector of the finite element node is as follows (11):

[0101] D uv =[Δu1,Δv1,0,…,Δu n ,Δv n ,0] T (11)

[0102] The initial pixel coordinate vector of the finite element node is as follows (12):

[0103] C uv =[u1,v1,1,…,u n ,v n ,1] T (12)

[0104] Introduce the conversion matrix Q and the initial influence matrix P:

[0105]

[0106] Rewrite Equation (10) into the vector relationship between the physical displacement of the finite element node and the pixel displacement:

[0107] D xyz =Q(D uv +PC uv ) (15)

[0108] In finite element analysis, the physical displacement of any point within a known finite element unit can be determined by the physical displacement of the finite element node and the shape function:

[0109] d xyz =[Δx,Δy,Δz] T =N xyz D xyz (16)

[0110] Among them, N xyz is the shape function matrix of this type of finite element.

[0111] By combining equations (9), (15), and (16), we can obtain the displacement field expression of the simulated object in the image coordinate system when it is deformed. The displacement of any pixel in the two-dimensional image can be accurately solved by interpolation:

[0112]

[0113] Where (u, v) is the initial pixel coordinate of any point in the finite element unit, is the modified shape function matrix.

[0114] In summary, after solving the three-dimensional physical displacement of the finite element nodes of the simulation object under load, the nodal physical displacement of the finite element unit is converted into the node pixel displacement through Equation (9), and then the pixel displacement of any point in the finite element unit is interpolated through Equation (17). The global pixel displacement field is solved using the nodal pixel displacement and shape function, and the full-resolution pixel displacement field data is constructed, laying the foundation for subsequent simulation image synthesis.

[0115] S205 , mapping the original image pixels of the visible part of the simulated object to the deformed sub-pixel positions according to the global pixel displacement field, and distributing the pixel brightness of the original image to adjacent pixels according to the area ratio, thereby achieving the migration of the original image brightness.

[0116] Based on the global pixel displacement field in step S204, each pixel in the original image is mapped to a new position in the deformed image. New visual content is generated through brightness diffusion and superposition. The basic principle of brightness migration can be expressed as follows:

[0117]

[0118] Among them, L and Represent the brightness fields of the original image and the deformed image, respectively. The brightness value of each pixel consists of three RGB channels ranging from 0 to 255. Each channel is calculated using Equation (18) and subsequent brightness-related equations to ensure color consistency before and after brightness migration. The brightness of each pixel is mapped according to its new position after deformation, so that all pixels of the original image are gradually converted to generate the brightness field of the new deformed image.

[0119] Adopting the area-weighted allocation strategy: Since the pixel displacement affected by structural deformation is at the sub-pixel level, the original image pixel may be located between four adjacent whole pixel positions after deformation, such as Figure 5 As shown in (a), a single original pixel may contribute to four different pixels. To maintain the consistency of image clarity and overall brightness, the brightness of the original image pixel will be distributed to adjacent pixels according to the area ratio, as shown in Figure 5 (b) shown.

[0120] The contribution of the brightness value L of the original image pixel to the brightness of the adjacent pixels can be expressed as:

[0121]

[0122] Among them, r u and r v is the remaining sub-pixel displacement after deducting the integer pixel displacement in the deformed image, w i It is the brightness weight factor allocated according to the area ratio.

[0123] Due to the existence of sub-pixel deformation, each pixel migrated to the deformed image may correspond to multiple pixels in the original image, so the pixel brightness assigned according to the weight is superimposed on the brightness field of the deformed image, such as Figure 5 As shown in (c), the brightness value of each pixel in the deformed image is the sum of the brightness contributions of all related pixels in the original image.

[0124]

[0125] Where M represents the total number of original image pixels that contribute to the pixels migrated to the deformed image.

[0126] The brightness value of each pixel in the deformed image is obtained by summing the contribution values ​​of all relevant original pixels through formula (20), forming a complete and continuous high-quality deformed brightness field, which accurately reflects the changes in visual features during the structural deformation process.

[0127] S206 , correcting the deformed image after brightness migration, performing brightness processing on the background area of ​​the corrected deformed image according to the visibility classification matrix and the pixel brightness of the original image, to obtain a simulated deformed image.

[0128] Under uniform lighting conditions, the surface texture and color of an object should remain stable before and after deformation. However, changes in pixel distribution can lead to jagged edges and color distortion. For example, when a structure expands under tension or contracts under compression, both the average pixel brightness and the overall image brightness are affected. Therefore, a weighted summation brightness correction mechanism is designed to modify the brightness field based on brightness distribution weights, ensuring consistent image brightness before and after deformation.

[0129] For rigid body deformation, the sum of the brightness weights of each pixel after pixel brightness migration should be equal to 1. Analysis shows that the weight of the compressed area exceeds 1, while the weight of the stretched area is less than 1. To correct color distortion, the brightness weight of each pixel is normalized to keep the total brightness of the image constant, and a threshold filter is set to remove edge jaggedness, making the corrected image more realistic:

[0130]

[0131] Among them, L b Represents the background pixel brightness, ε is the filter threshold for edge jaggedness removal, and 0.2 is recommended.

[0132] Since the average pixel brightness and the overall image brightness change due to structural bending and deformation, an image correction technique based on brightness distribution weights is adopted. First, the brightness distribution weight of each pixel is normalized by formula (21) to eliminate the brightness deviation caused by excessive weight in the compressed area and insufficient weight in the stretched area, ensuring that the overall brightness of the image is conserved before and after deformation. At the same time, an edge filtering algorithm with a threshold of 0.2 is introduced to eliminate the edge jaggedness generated during the migration process and make the color transition smooth and natural.

[0133] In the background brightness processing stage, a hierarchical restoration strategy is implemented based on the visibility classification matrix: after obtaining the corrected image, according to the visibility classification matrix, for the persistent visible background that is not blocked in both the original image and the corrected image, the persistent visible background of the corrected image is made to directly inherit the pixel brightness of the original image, such as Figure 6 As shown in (a), the background brightness (indicated by orange) is directly transferred to the corresponding pixel position of the deformed image. For the background that was blocked in the original image and newly revealed in the modified image, Figure 6 For the exposed blank pixels in (b), the pixel brightness of the newly exposed background of the corrected image is obtained by interpolating the neighboring area except the visible part of the object:

[0134]

[0135] Wherein, N represents the total number of background pixels participating in the interpolation around the blank pixel to be assigned.

[0136] The brightness interpolation starts from the boundary between the unoccluded and newly visible background areas, and fills and expands from the outside to the inside layer by layer until all newly exposed pixels obtain valid brightness values, such as Figure 6 As shown in (c), if there are insufficient valid reference pixels in the neighborhood of a certain pixel, the repair is temporarily suspended and recalculated after the surrounding pixels are filled until all newly exposed background areas are repaired.

[0137] This embodiment finally obtains an image of the aluminum sheet specimen bending deformation simulation, Figure 7 (a)~ Figure 7(c) shows the original image of the aluminum sheet, the real deformed image, and the simulated deformed image, respectively. Figure 7 (d)~ Figure 7 (f) is the image overlay after setting 50% transparency. Figure 7 (g)~ Figure 7 (i) shows the brightness difference of the image pairs. The MSE index of each image pair is calculated by formula (23). The MSE index quantifies the brightness difference between the real deformed image and the simulated image. The smaller the value, the higher the similarity and the more accurate the simulation of mechanical deformation. Figure 8 The MSE comparison results of each image pair are shown.

[0138]

[0139] in, and Represent the brightness values ​​of the real deformed image and the simulated deformed image at pixel (i, j), N u ×N v Indicates the image resolution.

[0140] The results show that the outline of the aluminum sheet and the character markings in the simulated deformed image basically overlap with the real deformed image, and the brightness field error between the two is reduced by about 75% compared with the original image, proving that the simulated image synthesized by the method of this embodiment is natural and realistic, accurately presenting the structural deformation and background fusion; in addition, the entire process from finite element calculation to simulated image generation in this embodiment takes about 40 seconds, indicating that the method of this embodiment can significantly improve efficiency while maintaining visual effects and deformation quality, effectively capture the visual and mechanical properties of objects, and achieve coordinated integration with the scene.

[0141] It should be noted that although the method operations of the above embodiments are described in a particular order, this does not require or imply that the operations must be performed in that particular order, or that all of the illustrated operations must be performed to achieve the desired results. Rather, the depicted steps may be performed in a different order. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into a single step, and / or a single step may be broken down into multiple steps.

[0142] Example 2:

[0143] like Figure 9 As shown, this embodiment provides a mechanical deformation augmented reality simulation system based on pixel-guided finite elements. The system includes a first establishment module 901, a second establishment module 902, a first calculation module 903, a second calculation module 904, a migration module 905, and a background processing module 906. The specific functions of each module are as follows:

[0144] The first establishing module 901 is used to obtain multi-angle images of the simulated object in an undeformed state, identify the camera intrinsic parameter matrix and extrinsic parameter matrix using the Zhang Zhengyou calibration method, and establish a bidirectional geometric mapping relationship between the world coordinate system and the image plane;

[0145] The second establishing module 902 is used to establish a three-dimensional finite element model of the simulation object based on the geometric characteristics and material properties of the simulation object, where the three-dimensional coordinates of each finite element node are based on the world coordinate system;

[0146] The first calculation module 903 is used to calculate and sort the depth values ​​of the finite element nodes in the camera coordinate system, and update the visibility classification matrix according to the occlusion of each finite element node to distinguish the visible part of the foreground object from the background area;

[0147] The second calculation module 904 is used to solve the three-dimensional physical displacement of the finite element nodes of the simulation object under load, map the three-dimensional physical displacement to a two-dimensional image coordinate system, calculate the two-dimensional pixel displacement of each finite element node, and interpolate to generate a global pixel displacement field;

[0148] A migration module 905 is configured to map the original image pixels of the visible portion of the simulated object to the deformed sub-pixel positions based on the global pixel displacement field, and to distribute the pixel brightness of the original image to adjacent pixels according to their area ratios, thereby achieving the migration of the original image brightness.

[0149] The background processing module 906 is used to correct the deformed image after brightness migration, and perform brightness processing on the background area of ​​the corrected deformed image according to the visibility classification matrix and the pixel brightness of the original image to obtain a simulated deformed image.

[0150] It should be noted that the system provided in this embodiment is only illustrated by the division of the above-mentioned functional modules. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure can be divided into different functional modules to complete all or part of the functions described above.

[0151] Example 3:

[0152] This embodiment provides a computer device, such as Figure 10As shown, it includes a processor 1002, a memory, an input device 1003, a display device 1004, and a network interface 1005 connected via a system bus 1001. The processor is used to provide computing and control capabilities. The memory includes a non-volatile storage medium 1006 and an internal memory 1007. The non-volatile storage medium 1006 stores an operating system, a computer program, and a database. The internal memory 1007 provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. When the processor 1002 executes the computer program stored in the memory, the template matching gesture recognition method of the above-mentioned embodiment 1 is implemented as follows:

[0153] Acquire multi-angle images of the simulated object in its undeformed state, use the Zhang Zhengyou calibration method to identify the camera's intrinsic and extrinsic parameter matrices, and establish a bidirectional geometric mapping relationship between the world coordinate system and the image plane; establish a three-dimensional finite element model of the simulated object based on its geometric features and material properties, with the three-dimensional coordinates of each finite element node based on the world coordinate system; construct a visibility classification matrix that matches the image resolution, calculate the depth values ​​of the finite element nodes in the camera coordinate system and sort them; update the visibility classification matrix based on the occlusion of each finite element node to distinguish the visible part of the foreground object from the background area; solve The three-dimensional physical displacement of the finite element nodes of the simulated object under load is mapped to a two-dimensional image coordinate system, the two-dimensional pixel displacement of each finite element node is calculated, and the global pixel displacement field is generated by interpolation. The original image pixels of the visible part of the simulated object are mapped to the sub-pixel positions after deformation based on the global pixel displacement field, and the pixel brightness of the original image is distributed to adjacent pixels according to the area ratio to achieve the migration of the original image brightness. The deformed image after brightness migration is corrected, and the background area of ​​the corrected deformed image is brightness processed according to the visibility classification matrix and the pixel brightness of the original image to obtain a simulated deformed image.

[0154] Example 4:

[0155] This embodiment provides a storage medium, which is a computer-readable storage medium and stores a computer program. When the computer program is executed by a processor, the template matching gesture recognition method of the above embodiment 1 is implemented as follows:

[0156] Acquire multi-angle images of the simulated object in its undeformed state, use the Zhang Zhengyou calibration method to identify the camera's intrinsic and extrinsic parameter matrices, and establish a bidirectional geometric mapping relationship between the world coordinate system and the image plane; establish a three-dimensional finite element model of the simulated object based on its geometric features and material properties, with the three-dimensional coordinates of each finite element node based on the world coordinate system; construct a visibility classification matrix that matches the image resolution, calculate the depth values ​​of the finite element nodes in the camera coordinate system and sort them; update the visibility classification matrix based on the occlusion of each finite element node to distinguish the visible part of the foreground object from the background area; solve The three-dimensional physical displacement of the finite element nodes of the simulated object under load is mapped to a two-dimensional image coordinate system, the two-dimensional pixel displacement of each finite element node is calculated, and the global pixel displacement field is generated by interpolation. The original image pixels of the visible part of the simulated object are mapped to the sub-pixel positions after deformation based on the global pixel displacement field, and the pixel brightness of the original image is distributed to adjacent pixels according to the area ratio to achieve the migration of the original image brightness. The deformed image after brightness migration is corrected, and the background area of ​​the corrected deformed image is brightness processed according to the visibility classification matrix and the pixel brightness of the original image to obtain a simulated deformed image.

[0157] It should be noted that the computer-readable storage medium of the present embodiment may be a computer-readable signal medium or a computer-readable storage medium or any combination thereof. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0158] In this embodiment, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. Furthermore, in this embodiment, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries a computer-readable program. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable storage medium other than a computer-readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device. The computer program contained on a computer-readable storage medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0159] The computer readable storage medium can be written in one or more programming languages ​​or a combination thereof to execute the computer program for performing the present embodiment, including object-oriented programming languages ​​such as Java, Python, C++, and conventional procedural programming languages ​​such as C or similar programming languages. The program can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect via the Internet).

[0160] In summary, the present invention combines the traditional finite element method with computer vision technology to establish a two-way data channel between finite element calculations and real scenes. Combined with a variety of image processing algorithms, it realizes mechanical deformation augmented reality simulation that integrates geometric features, visual attributes and real background environments, improves the ability to generate high-quality simulated images, enhances the applicability of the finite element method in real scenes, and creates a new paradigm for the application of real-scene finite element methods.

[0161] The foregoing description is merely a preferred embodiment of the present invention. It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention. Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive. The scope of the present invention is defined by the appended claims, not the foregoing description, and all variations that come within the meaning and range of equivalents of the claims are intended to be included within the present invention.

Claims

1. A mechanical deformation augmented reality simulation method based on pixel-guided finite element, characterized in that: The method comprises: Acquire multi-angle images of the simulated object in its undeformed state, use the Zhang Zhengyou calibration method to identify the camera's intrinsic and extrinsic parameter matrices, and establish a bidirectional geometric mapping relationship between the world coordinate system and the image plane; Based on the geometric features and material properties of the simulation object, a three-dimensional finite element model of the simulation object is established, and the three-dimensional coordinates of each finite element node are based on the world coordinate system; Construct a visibility classification matrix that matches the image resolution, calculate and sort the depth values ​​of the finite element nodes in the camera coordinate system, and update the visibility classification matrix based on the occlusion of each finite element node to distinguish the visible part of the foreground object from the background area. Solve the 3D physical displacement of the finite element nodes of the simulation object under load, map the 3D physical displacement to the 2D image coordinate system, calculate the 2D pixel displacement of each finite element node, and interpolate to generate the global pixel displacement field; Map the original image pixels of the visible part of the simulated object to the deformed sub-pixel positions according to the global pixel displacement field, and distribute the pixel brightness of the original image to adjacent pixels according to the area ratio to achieve the migration of the original image brightness; The deformed image after brightness migration is corrected, and the background area of ​​the corrected deformed image is brightness processed according to the visibility classification matrix and the pixel brightness of the original image to obtain a simulated deformed image.

2. The mechanical deformation augmented reality simulation method according to claim 1, characterized in that: The Zhang Zhengyou calibration method is used to identify the camera's intrinsic parameter matrix and extrinsic parameter matrix, and establish a bidirectional geometric mapping relationship between the world coordinate system and the image plane, specifically including: Based on Zhang Zhengyou's calibration method, a calibration plate is used as a reference plane, and the spacing between the checkerboard corners is defined as a known physical size. The calibration plate covers the entire image plane and ensures that the checkerboard plane is parallel to the surface of the simulated object or at a known angle. Use the Harris corner detection algorithm to extract the pixel coordinates of the corner points of the calibration plate and record the corresponding coordinates of the pixel coordinates in the world coordinate system; The reprojection error is minimized by a nonlinear optimization algorithm, the intrinsic and extrinsic parameter matrices of the camera are iteratively solved, and a bidirectional geometric mapping relationship between the world coordinate system and the image plane is established.

3. The mechanical deformation augmented reality simulation method according to claim 1, characterized in that: The depth values ​​of the finite element nodes in the camera coordinate system are calculated and sorted, and the visibility classification matrix is ​​updated according to the occlusion of each finite element node to distinguish the visible part of the foreground object from the background area. Specifically, the following steps are performed: The finite element node coordinates are converted into camera coordinates through matrix transformation, and the depth values ​​of all finite element nodes are calculated, and all finite element nodes are arranged in ascending order according to the depth values; Traverse the sorted finite element node queue. If the current finite element node is not blocked, mark the current finite element node as a visible node and add it to the visible node list. Update the visible node classification value to 0. If the current finite element node is blocked, mark it as invisible and continue processing the next node. If the current finite element node is visible, and the finite element unit where the current finite element node is located contains at least two visible nodes other than the current finite element node, then a minimum enclosing convex polygon is constructed based on the two-dimensional projection coordinates of the current finite element node and other visible nodes, and the pixel area covered by the convex polygon is marked as visible, and the classification value of the area is updated to 0; Through iterative updating, the visibility classification matrix is ​​updated according to the classification value of each finite element node to distinguish the visible part of the foreground object from the background area.

4. The mechanical deformation augmented reality simulation method according to claim 1, characterized in that: The method of solving the three-dimensional physical displacement of the finite element nodes of the simulation object under load, mapping the three-dimensional physical displacement to a two-dimensional image coordinate system, calculating the two-dimensional pixel displacement of each finite element node, and interpolating to generate a global pixel displacement field specifically includes: Perform stress analysis on the simulation object and solve the three-dimensional physical displacement of the finite element nodes of the simulation object under load; Based on the pinhole imaging model, the three-dimensional physical displacement of the finite element nodes is converted into two-dimensional pixel displacement in the image coordinate system, and the global pixel displacement field is generated through shape function interpolation.

5. The mechanical deformation augmented reality simulation method according to claim 1, characterized in that: The method of mapping the original image pixels of the visible part of the simulated object to the deformed sub-pixel positions according to the global pixel displacement field and distributing the pixel brightness of the original image to adjacent pixels according to the area ratio to achieve the migration of the original image brightness specifically includes: Based on the global pixel displacement field, each pixel in the original image is mapped to a new position in the deformed image, and new visual content is generated through brightness diffusion and superposition; Assign brightness weight factors to adjacent pixels according to their area ratios, distribute the pixel brightness of the original image to the adjacent pixels based on the weight factors, and superimpose the distributed pixel brightness on the brightness field of the deformed image.

6. The mechanical deformation augmented reality simulation method according to claim 1, characterized in that: The correcting of the deformed image after brightness shift specifically includes: For the transformed image after migration, the brightness weight factor of each pixel in the brightness field is normalized to keep the total brightness of the image constant, and a threshold filter is set to remove edge jaggedness.

7. The mechanical deformation augmented reality simulation method according to claim 1, characterized in that: The brightness processing of the background area of ​​the corrected deformed image according to the visibility classification matrix and the pixel brightness of the original image specifically includes: According to the visibility classification matrix, for the persistent visible background that is not occluded in both the original image and the corrected deformed image, the persistent visible background of the corrected deformed image is made to directly inherit the pixel brightness of the original image. For the background that is occluded in the original image but newly revealed in the corrected deformed image, the pixel brightness of the newly revealed background in the corrected deformed image is obtained by interpolation of the neighboring area excluding the visible part of the object.

8. A mechanical deformation augmented reality simulation system based on pixel-guided finite element, characterized in that: The system comprises: The first establishment module is used to obtain multi-angle images of the simulated object in the undeformed state, use the Zhang Zhengyou calibration method to identify the camera's intrinsic parameter matrix and extrinsic parameter matrix, and establish a bidirectional geometric mapping relationship between the world coordinate system and the image plane; The second establishment module is used to establish a three-dimensional finite element model of the simulation object based on the geometric characteristics and material properties of the simulation object, and the three-dimensional coordinates of each finite element node are based on the world coordinate system; The first calculation module is used to construct a visibility classification matrix that matches the image resolution, calculate and sort the depth values ​​of the finite element nodes in the camera coordinate system, and update the visibility classification matrix based on the occlusion of each finite element node to distinguish the visible part of the foreground object from the background area; The second calculation module is used to solve the three-dimensional physical displacement of the finite element nodes of the visible part of the simulation object under load, map the three-dimensional physical displacement to the two-dimensional image coordinate system, calculate the two-dimensional pixel displacement of each finite element node, and interpolate to generate a global pixel displacement field; A migration module is used to map the original image pixels of the simulated object to the deformed sub-pixel positions according to the global pixel displacement field, and to distribute the pixel brightness of the original image to adjacent pixels according to their area ratio to achieve the migration of the original image brightness; The background processing module is used to correct the deformed image after brightness migration, and perform brightness processing on the background area of ​​the corrected deformed image according to the visibility classification matrix and the pixel brightness of the original image to obtain a simulated deformed image.

9. A computer device comprising a processor and a memory for storing a program executable by the processor, characterized in that: When the processor executes the program stored in the memory, the mechanical deformation augmented reality simulation method according to any one of claims 1 to 7 is implemented.

10. A storage medium storing a program, characterized in that: When the program is executed by a processor, the mechanical deformation augmented reality simulation method according to any one of claims 1 to 7 is implemented.

Images