A Method for Object Material Recognition Based on Tapping Sound Simulation and Deep Learning

Abstract

The present application relates to a kind of object material identification method based on knocking sound simulation and deep learning, utilize the model generated to carry out the knocking sound simulation of given knock point, then combine real knock sound and train and compare, and finally analyze and obtain the material of target object.The method includes two parts of deep learning model training and material identification and classification, wherein, deep learning model training includes five steps of constructing training material library, object model construction, actual knock sound recording, knocking sound simulation and model training;Material identification and classification include four steps of target object model construction, target actual knock sound recording, target knock sound simulation and model identification and classification.Compared with prior art, the present application has the advantages of avoiding a large number of actual knock sound collection work, high accuracy of material identification result etc..

Description

Technical Field

[0001] This invention relates to the field of material recognition technology, and in particular to a method for recognizing object materials based on tapping sound simulation and deep learning. Background Technology

[0002] In recent years, the development of Augmented Reality (AR) and Virtual Reality (VR) has led to numerous applications in fields such as education and entertainment, resulting in a surge in people's demand for and expectations of the virtual world. The key to current development lies in defining and strengthening the connection between the real and virtual worlds, thereby enabling customization for different users and fostering a sense of belonging. However, simply generating models through scanning and importing them into virtual scenes hides some problems: scanned objects lack material information, which limits user interaction to a few simple operations that don't involve physical simulation, such as translation, rotation, and copying. Furthermore, due to the lack of physical simulation, these interactions lack corresponding sound effects. All these issues contribute to a sense of unreality and significantly diminish the user experience.

[0003] To address the aforementioned problem of detecting or analyzing the material of objects, numerous institutions both domestically and internationally have explored various approaches, which can be broadly categorized into the following areas: vision-based methods, resonance spectroscopy methods, wireless signal-based methods, tapping methods, and other methods.

[0004] 1. Vision-based methods: These methods utilize deep learning techniques to analyze images of objects to determine their material composition. However, surface decorations or patterns often interfere with material analysis, making it difficult to achieve ideal results.

[0005] 2. Resonance spectroscopy: Commonly used to identify the material and composition of liquids, it utilizes the acoustic properties of liquids and attempts to find the natural frequencies of the target material. This helps in identifying liquids because the natural frequencies depend on the liquid's composition. However, this method cannot be used to identify the material of solids and therefore does not meet our needs.

[0006] 3. Wireless signal-based methods: These include methods based on ultra-wideband radar. For example, SiWa designed a handheld device based on pulsed ultra-wideband (IR-UWB) radar. This device slides across a wall to emit signals and simultaneously receives the reflected signals. Deep learning processing is then used to identify the material composition (e.g., steel reinforcement, PVC) and condition (e.g., corrosion) of objects within different wall types (e.g., concrete, drywall). Similarly, there are studies using X-ray correlation methods to analyze material characteristics of objects. However, wireless signal-based methods require additional hardware, which doesn't align with our previously envisioned simple XR application scenarios.

[0007] 4. Tapping Method: Tapping involves gently tapping a test object to generate sound, which is then recorded to identify the object's material. Since the elasticity of an object and the internal friction determined by its material and structure affect the sound frequency, deep learning techniques can be used to analyze the tapping sound to obtain the material of the 3D model. In the tapping method, Giordano et al. simply analyzed the correlation between the tapping sound and four given materials (iron, glass, wood, and plastic) based on the physical properties of the materials, and then achieved identification and classification between two groups (iron-glass and wood-plastic). However, this method has a limited scope and poor performance. While Hamontree et al. expanded the scope of material identification, this approach requires not only tapping sound but also the use of a perturbation observer to record the tapping force response, and a robot to tap the object, which does not meet our expected environmental requirements. Researchers like Luo achieved object material identification solely through deep learning analysis of tapping sounds. However, this method requires using the same tapping object (such as a marker) to tap different target objects, and demands a high number of taps (120 taps per object). Furthermore, this approach cannot identify the material of objects with new shapes. Meanwhile, some researchers have applied sound simulation technology to the tapping method. Sterling et al. used simulated sounds as supplementary data to work with depth cameras to identify the shape and material of transparent, enclosed, or highly reflective objects. However, this method simultaneously feeds both object shape and simulated sound into the deep learning model, resulting in an inability to effectively identify the material of newly created shapes. Researchers like Zhang simulated the process of an object's free fall and multiple impacts with the ground, analyzing both real and simulated sounds to derive multiple physical properties such as material and fall height. However, this method requires simulating the entire process from the start of free fall to the object's final rest, incurring excessive computational overhead. It is also unsuitable for materials where free fall experiments are inconvenient, such as common glass and ceramics. Jin et al. designed a fully differentiable sound simulation algorithm with trainable parameters, including material parameters. They then compared the simulated results with real knocking sounds and backpropagated the comparison results to optimize the parameters of the simulation algorithm. The material parameters obtained after training are the inferred material parameters. However, due to limitations in application scenarios, the recognition accuracy is not high in some scenarios.

[0008] 5. Other methods, such as those based on the principle of heat conduction (using temperature tactile sensors to identify materials with low heat absorption coefficients), are difficult to promote and apply because they require additional equipment and have a limited range of materials that can be identified. Summary of the Invention

[0009] The purpose of this invention is to provide a material identification method based on sound simulation and deep learning. This method utilizes common everyday objects and selects a light tapping method as the foundation, ensuring that the equipment requirements are easily met and the recognition range is sufficient to cover common everyday items. The method employs a deep learning model to identify and classify the material of the object being tapped based on the tapping sound. Sound simulation technology is introduced to generate simulated tapping sounds, which are then applied to both model training and model classification. This meets the needs of the intended scenario, effectively expanding the range of shapes that can be identified while reducing the workload of collecting tapping sounds, thus handling the material identification of objects with various shapes.

[0010] The objective of this invention can be achieved through the following technical solutions:

[0011] A method for object material recognition based on tapping sound simulation and deep learning includes the following steps:

[0012] S1, Deep learning model training;

[0013] S1.1, Constructing a training material library: Select several materials according to the target recognition range to form a material library; select several objects of different shapes within each material to form a training set;

[0014] S1.2, Object Model Construction: Construct 3D models of objects in the training set by scanning;

[0015] S1.3, Recording actual tapping sounds: Select several tapping points on the objects in the training set, record the positions of the tapping points, and tap lightly on the tapping points to obtain actual tapping sounds;

[0016] S1.4, Impact Sound Simulation: Assign the corresponding material to the scanned 3D model, and obtain the vibration mode and transmission diagram of the object by calculating the modal and sound transfer function; input the position of the impact point, and generate simulated impact sound based on the impact point and the vibration mode and transmission diagram of the object.

[0017] S1.5, Model Training: The deep learning model is trained by using actual and simulated knocking sounds as inputs and material as outputs.

[0018] S2, Material Identification and Classification;

[0019] S2.1, Target object model construction: Construct a 3D model of the target object by scanning;

[0020] S2.2 Recording of actual striking sound of the target: Select at least one striking point on the target object, record the position of the striking point, and lightly tap the striking point to obtain the actual striking sound of the target;

[0021] S2.3, Target Impact Sound Simulation: Assign each material in the material library to the 3D model of the scanned target object, and obtain the corresponding vibration mode and transmission diagram by calculating the modal and acoustic transfer function; input the position of the impact point, and generate the target simulated impact sound corresponding to each material based on the impact point and the corresponding vibration mode and transmission diagram.

[0022] S2.4, Model Recognition and Classification: The actual striking sound of the target and the simulated striking sound of the target corresponding to various materials are input into the deep learning model to obtain the material recognition result of the target object.

[0023] In the actual recording of the striking sound, several striking points are evenly distributed.

[0024] The vibration mode is obtained through modal calculation:

[0025] For a damped object with multiple degrees of freedom, its vibration differential function is written as:

[0026]

[0027] Where M is the mass matrix, C is the damping matrix, K is the stiffness matrix, and u is the displacement matrix. It is a velocity matrix. It is the acceleration matrix;

[0028] The surface mesh of the 3D model is converted into a tetrahedral mesh using the isosurface filling method, and the mass matrix M and stiffness matrix K are calculated using the converted tetrahedral mesh.

[0029] The damping matrix is ​​calculated using the Rayleigh damping approximation, where Rayleigh damping is a composite function of the mass matrix and the stiffness matrix, in the form: C = αM + βK, where α and β are pre-set weighting parameters;

[0030] Substituting Rayleigh damping into the vibration differential function yields...

[0031] The vibration modes of the model are calculated based on the vibration differential function as follows:

[0032]

[0033] in It is the eigenvector of the differential equation, n is the number of independent vibration modes within the frequency range that the human ear can hear, Y is the modal amplitude, and ω is the eigenvector. d θ is the modal frequency, and θ is the modal phase.

[0034] The sound transmission map is obtained by calculating the sound transfer function to obtain the air pressure distribution map at the actual sound sampling location. The specific steps include:

[0035] The Neumann boundary conditions are used to calculate the acceleration transmitted from the vibration of an object's surface to the air. The Neumann boundary conditions are as follows:

[0036]

[0037] Among them, a n (x,t) is the surface acceleration, ρ is the density of the medium, Γ is the surface area of ​​the object, and p is the sound pressure. It is the normal vector of the boundary at position x;

[0038] The wave propagation problem is handled using a 3D wave function, where the sound pressure p(x,t) satisfies the following equation:

[0039]

[0040] Where c represents the speed of sound in air. It is the Laplace operator;

[0041] Meanwhile, the following harmonic pressure equation is adopted: p(x,t)=p(x)e +iωt ;

[0042] Substituting the harmonic pressure equation into the equation that the sound pressure must satisfy, we obtain the Helmholtz equation as follows:

[0043]

[0044] in, ω is the frequency corresponding to the mode, and λ is the wavelength corresponding to the mode;

[0045] The sound transmission diagram is obtained by solving p(x) using the Neumann boundary conditions and the Helmholtz equations.

[0046] The finite-difference time-domain method is used for rapid calculation of acoustic transfer maps, specifically including the following steps:

[0047] Wrap the object in a regular mesh, discretize the object, and derive the following formula for the pressure update over time on the mesh:

[0048]

[0049] in, ρ is the sound pressure at point (i,j,k) at time mτ, where h is the grid size, τ is the time step, c represents the speed of sound in air, and α is a constant coefficient controlling air viscosity damping. It is the Laplace operator, and the formula is as follows:

[0050]

[0051] Setting absorbing boundary conditions at the mesh edges can reduce the impact of false reflections;

[0052] By rasterizing the object's surface according to a mesh, a new form of the von Neumann boundary condition is obtained as follows:

[0053]

[0054] Where, x b It is a point on the surface of the object, x g It is a surface point on the grid after rasterization, x r It is the reflection point, x r =x g +2(x b -x g ), p r It is x r The sound pressure at p g It is x g The sound pressure at that location, l = |x r -x g |,a n ρ is the surface acceleration, ρ is the density of the medium, and O is a higher-order small term representing the error term.

[0055] The factors related to surface acceleration are listed and calculated in a package. The results of each factor are linearly superimposed to obtain the surface acceleration. Substituting this into a new form of Neumann boundary conditions yields the acoustic transmission diagram.

[0056] The process of listing and encapsulating calculations of factors related to surface acceleration includes one or more of the following: point / regional sound source merging calculation, modal vibration calculation, acceleration noise calculation, water foam calculation, and finite element calculation of thin-shell objects.

[0057] The method for generating simulated impact sounds based on the impact point, corresponding vibration mode, and sound transmission diagram is as follows: Input the location of the impact point and the impact force data; calculate the vibration state of the object's surface from the vibration mode and sound transmission diagram; then calculate the air pressure state around the actual impact sound acquisition location by combining the sound transmission diagram; and finally linearly combine the vibration state and air pressure state to generate simulated impact sounds.

[0058]

[0059] Where x represents the actual location where the striking sound was collected, I j The sound transmission intensity at the actual sound sampling location under mode j is represented by the absolute value of the sound pressure p, q j It represents the amplitude of vibration on the surface of an object.

[0060] The deep learning model employs a domain adversarial transfer network, which includes a feature extractor, a label predictor, and a domain classifier.

[0061] During the training process of the deep learning model, for the input actual knocking sound and simulated knocking sound, the deep learning model uses a feature extractor to extract the sound features of the two respectively, then inputs the sound features into a label predictor to obtain the material label, and inputs the sound features into a domain classifier to obtain the data domain.

[0062] In the model recognition and classification steps, before material recognition, the parameters of the feature extractor are fixed, and the simulated knocking sounds corresponding to various materials are input into the feature extractor to extract the simulated sound features. The extracted simulated sound features and their corresponding materials are then used to fine-tune the label predictor.

[0063] Compared with the prior art, the present invention has the following beneficial effects:

[0064] 1. This invention introduces sound simulation technology to generate simulated knocking sounds, and simultaneously applies these simulated sounds to both model training and model classification / recognition. On the one hand, using simulated sounds reduces the model's need for a large amount of real knocking sound training data, and also reduces the repetitive and tedious knocking and sound recording required by users, saving researchers' and users' time and effort. On the other hand, because both simulated and real knocking sounds contain the same object shape information, applying them together to both deep learning model training and model classification / recognition allows the model to overcome shape limitations, enabling it to recognize the material of objects with new shapes, and maintaining high recognition accuracy in recognizing the material of objects of various shapes.

[0065] 2. This invention uses both simulated sound and real knocking sound to train the deep learning model. Through the training of the domain adversarial transfer network, the common features of the two sounds can be effectively extracted. At the same time, during online recognition, the model is fine-tuned by generating the target simulated sound signal, which ensures the accuracy and robustness of the recognition. Attached Figure Description

[0066] Figure 1 This is a flowchart of the method of the present invention;

[0067] Figure 2 This is a flowchart illustrating the simulation of a striking sound in one embodiment;

[0068] Figure 3 This is a flowchart illustrating offline training of a deep learning model in one embodiment;

[0069] Figure 4 This is a flowchart illustrating online recognition using a deep learning model in one embodiment. Detailed Implementation

[0070] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0071] This embodiment provides a method for object material recognition based on tapping sound simulation and deep learning. It uses a light tapping method as a foundation and introduces sound simulation technology to generate simulated tapping sounds. This reduces the repetitive and tedious process of multiple taps and sound recordings during training data collection and also addresses the material recognition of newly added shapes. Additionally, a deep learning model, Domain Adversarial Neural Networks (DANN), is introduced to eliminate the influence of experimental environmental factors (such as air pressure differences) that are irrelevant to material recognition and differ between real and simulated tapping sounds. This extracts common features between simulated and actual tapping sounds, allowing simulated sounds to be applied seamlessly to network training and applications, just like real tapping sounds.

[0072] Specifically, such as Figure 1 As shown, it includes the following steps:

[0073] S1, Deep learning model training:

[0074] S1.1, Constructing a training material library: Select several materials to form a material library based on the target recognition range; within each material, select several objects of different shapes to form a training set. Compared to other tapping recognition methods that need to consider the influence of multiple shapes, this invention does not rely on the diversity of training object shapes, and recommends selecting only about three shapes.

[0075] S1.2, Object Model Construction: Construct 3D models of objects in the training set by scanning.

[0076] 3D models of objects can be constructed using any method, including but not limited to manual modeling, 3D scanner scanning, and mobile phone camera scanning. In this embodiment, an existing 3D scanning app (Spectre3D on the Android platform) is used to scan the target object and generate the corresponding 3D model. The model file type is .obj. The specific steps are as follows:

[0077] S1.2.1 Place the object stably, leaving enough space around it for the scanner to walk around.

[0078] S1.2.2, The scanner opens the scanning app and enters the scanning process. Holding the phone, the scanner walks around the object 4-5 times, ensuring that there are three perspectives: eye level, top view, and bottom view. At the same time, the object must be completely within the camera's field of view.

[0079] S1.2.3 After shooting, wait for the APP to process and generate a 3D model. The model type is .obj file. Then send the model file to the computer.

[0080] S1.3, Recording actual tapping sounds: Select several evenly distributed tapping points on the objects in the training set, record the positions of the tapping points, and perform light tapping on the tapping points to obtain actual tapping sounds.

[0081] In this embodiment, a light tap can be made using a corner of the phone casing at the tapping point, and the sound produced by the tap can be collected using a microphone (such as the phone's built-in microphone). In specific implementation, a sampling frequency of 44100Hz can be used, but is not limited to.

[0082] S1.4, Impact Sound Simulation: Assign the corresponding material to the scanned 3D model, calculate the vibration mode and transmission diagram of the object by calculating the modal and sound transfer function; input the position of the impact point, and generate simulated impact sound based on the impact point and the vibration mode and transmission diagram of the object.

[0083] like Figure 2 As shown, the process of generating a simulated striking sound includes the following steps:

[0084] S1.4.1, Import the corresponding parameters of the material for this object;

[0085] S1.4.2, Vibration modes are obtained through modal calculation:

[0086] This embodiment uses the traditional linear modal analysis method to calculate the modes of the object. For a multi-degree-of-freedom damped object, its vibration differential function is written as:

[0087]

[0088] Where M is the mass matrix, C is the damping matrix, K is the stiffness matrix, and u is the displacement matrix. It is a velocity matrix. It is the acceleration matrix.

[0089] To obtain the mass matrix and stiffness matrix of the object, the surface mesh of the 3D model is converted into a tetrahedral mesh using the isosurface filling method, and the mass matrix M and stiffness matrix K are calculated using the converted tetrahedral mesh.

[0090] Since the actual damping matrix is ​​very complex, this embodiment uses Rayleigh damping to approximate the final result. Rayleigh damping is a composite function of the mass matrix and the stiffness matrix, in the form of: C = αM + βK, where α and β are preset weight parameters.

[0091] Substituting Rayleigh damping into the vibration differential function yields...

[0092] The vibration modes of the model are calculated based on the vibration differential function as follows:

[0093]

[0094] in It is the eigenvector of the differential equation, n is the number of independent vibration modes within the frequency range that the human ear can hear, Y is the modal amplitude, and ω is the eigenvector. d θ is the modal frequency, and θ is the modal phase.

[0095] S1.4.3, Obtain the air pressure distribution map at the actual sound acquisition location by calculating the sound transfer function, and obtain the sound transfer map. The specific steps include:

[0096] The Neumann boundary conditions are used to calculate the acceleration transmitted from the vibration of an object's surface to the air. The Neumann boundary conditions are as follows:

[0097]

[0098] Among them, a n (x,t) is the surface acceleration, ρ is the density of the medium, Γ is the surface area of ​​the object, and p is the sound pressure. It is the normal vector of the boundary at position x.

[0099] The wave propagation problem is handled using a 3D wave function, where the sound pressure p(x,t) satisfies the following equation:

[0100]

[0101] Where c represents the speed of sound in air. It is the Laplace operator.

[0102] Meanwhile, the following harmonic pressure equation is adopted: p(x,t)=p(x)e +iωt ;

[0103] Substituting the harmonic pressure equation into the equation that the sound pressure must satisfy, we obtain the Helmholtz equation as follows:

[0104]

[0105] in, ω is the frequency corresponding to the mode, and λ is the wavelength corresponding to the mode;

[0106] The sound transmission diagram is obtained by solving p(x) using the Neumann boundary conditions and the Helmholtz equations.

[0107] S1.4.4, Since the computational load of sound transmission in S1.4.3 is extremely large, this embodiment uses the Finite-Difference Time-Domain (FDTD) method to perform fast calculation of the sound transmission map using a GPU. Specifically, it includes the following steps:

[0108] S1.4.4.1, by wrapping the object with a regular mesh and discretizing the object, the following formula is derived for the pressure updating over time on the mesh:

[0109]

[0110] in, ρ is the sound pressure at point (i,j,k) at time mτ, where h is the grid size, τ is the time step, c represents the speed of sound in air, and α is a constant coefficient controlling air viscosity damping. It is the Laplace operator, and the formula is as follows:

[0111]

[0112] S1.4.4.2, setting absorbing boundary conditions (ABCs) at the mesh edges to reduce the effect of false reflections;

[0113] S1.4.4.3, the object surface is rasterized according to a mesh, resulting in the following new form of the von Neumann boundary condition:

[0114]

[0115] Where, x b It is a point on the surface of the object, x g It is a surface point on the grid after rasterization, x r It is the reflection point, x r =x g +2(x b -x g ), p r It is x r The sound pressure at p g It is x g The sound pressure at that location, l = |x r -x g |,a n ρ is the surface acceleration, ρ is the density of the medium, and O is a higher-order small term representing the error term.

[0116] S1.4.4.4 In order to quickly calculate the surface acceleration, the factors related to the surface acceleration are listed and encapsulated for calculation, including but not limited to: point / region sound source merging calculation, modal vibration calculation, acceleration noise calculation, water foam calculation, finite element calculation of thin shell objects, one or more of these factors. The results of each factor are linearly superimposed to obtain the surface acceleration, and the new form of Neumann boundary conditions is substituted to obtain the sound transmission diagram.

[0117] S1.4.5, Generating simulated impact sound based on the impact point, corresponding vibration mode, and sound transmission diagram: Input the location of the impact point and the impact force data. Based on the object's vibration mode and sound transmission diagram, calculate the vibration state of the object's surface from the vibration mode. Then, combine the sound transmission diagram to calculate the air pressure state around the actual impact sound acquisition location. Linearly combine the vibration state and air pressure state to generate simulated impact sound.

[0118]

[0119] Where x represents the actual location where the striking sound was collected, I j The sound transmission intensity at the actual sound sampling location under mode j is represented by the absolute value of the sound pressure p, q j It represents the amplitude of vibration on the surface of an object.

[0120] S1.5, Model Training: The deep learning model is trained by using actual and simulated knocking sounds as inputs and material as outputs.

[0121] In this embodiment, the deep learning model uses a Domain Adversarial Transfer Network (DANN), which includes a feature extractor, a label predictor, and a domain classifier.

[0122] like Figure 3 As shown, during the training process of the deep learning model, after grouping and pairing the input actual and simulated knocking sounds and performing data preprocessing, the deep learning model uses a feature extractor to extract the sound features of both. These sound features are then input into a label predictor to obtain material labels, and into a domain classifier to obtain the domain to which the data belongs. The domain classifier is designed to achieve domain adversarial training, outputting a prediction of whether the input features come from simulated or real sounds—that is, a domain label. The feature extractor, during training, tries to extract features that the domain classifier cannot recognize, thus creating an adversarial process. Through adversarial training, the feature extractor aims to extract as many common features as possible from both simulated and actual signals.

[0123] S2, Material Identification and Classification:

[0124] S2.1, Target object model construction: Construct a 3D model of the target object by scanning; the implementation process of this step can refer to step S1.2, and use an existing 3D scanning APP (Spectre3D on the Android platform) to scan the target object and generate the corresponding 3D model. The model file type is .obj.

[0125] S2.2 Recording of actual striking sound of the target: Select at least one striking point on the target object, record the position of the striking point, and lightly strike the striking point to obtain the actual striking sound of the target.

[0126] The implementation process of this step can refer to step S1.3. However, this step does not require collecting the sound of multiple tapping points as in training. It is only necessary to select any number (greater than or equal to 1) of tapping points and collect the sound.

[0127] S2.3, Target Impact Sound Simulation: Assign each material from the material library to the 3D model of the scanned target object, and obtain the corresponding vibration mode and transmission diagram by calculating the modal and acoustic transfer function; input the position of the impact point, and generate the target simulated impact sound corresponding to each material based on the impact point and the corresponding vibration mode and transmission diagram.

[0128] S2.4, Model Recognition and Classification: The actual striking sound of the target and the simulated striking sound of the target corresponding to various materials are input into the deep learning model to obtain the material recognition result of the target object.

[0129] like Figure 4 As shown, a trained deep learning model is applied, and the parameters of the feature extractor are fixed. Simulated impact sounds corresponding to various materials are preprocessed and input into the feature extractor to extract simulated sound features. These features, along with their corresponding materials, are then used to fine-tune the label predictor. After fine-tuning, the actual impact sounds are input into the feature extractor to obtain actual sound features. The actual sound features and all simulated sound features are then input into the fine-tuned label predictor, which outputs the material identification result of the target object.

[0130] In summary, this invention employs sound simulation to supplement the impact sound dataset, while simultaneously allowing the training results to take shape factors into account, enabling its application to the recognition of materials from objects of various shapes. Furthermore, by using both simulated and real impact sounds for training the deep learning model, the simulated sound can be applied to the training and application of the network model as effectively as real impact sounds, thereby improving the model's recognition accuracy.

[0131] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A method for identifying material of an object based on knocking sound simulation and deep learning, characterized in that, Comprise the following steps: S1, deep learning model training; S1.1, construct training material library: according to the target recognition range selected several kinds of material constitutes material library; Each kind of material in the selection of several different shape object constitutes training set; S1.2, object model construction: through scanning the 3D model of object in training set is constructed; S1.3, actual knocking sound recording: in the object of training set selects several knocking points, records the position of knocking point, and performs light knocking on the knocking point, obtains the actual knocking sound; S1.4, knocking sound simulation: the corresponding material is given to the 3D model obtained by scanning, and the vibration mode and transfer function of the object are obtained by calculating the modal and sound transfer function; Input the position of the knocking point, generate the simulation knocking sound according to the knocking point and the vibration mode and transfer function of the object; Specifically, the air pressure distribution map of the actual knocking sound collection position is obtained by calculating the sound transfer function, the sound transfer function is obtained, the finite difference time domain method is used for fast calculation of the sound transfer function, which comprises the following steps: The object is wrapped with a regular grid, and the object is discretized to obtain the following formula for updating the pressure on the grid over time: wherein, is the sound pressure at the grid (x, y) point and time t, i , j , k ) and time t, h is the grid cell size, is the time step, c represents the speed of sound in air, is a constant coefficient that controls the air viscosity damping, is the Laplacian operator, which is formulated as follows:​ The absorbing boundary condition is set at the edge of the grid to reduce the influence of false reflection; The object surface is rasterized according to the grid to obtain a new form of Neumann boundary condition as follows: wherein is a point on the surface of the object, x g is a point on the surface of the object after gridding on a grid, x r is a reflection point, x r x g + 2( x b x g ), p r is x r the sound pressure at p g is x g the sound pressure at l | x r x g |, is the surface acceleration, is the density of the medium, is a high order small term, representing an error term;​​​ List and encapsulate the factors related to surface acceleration, linearly superimpose the results of each factor to obtain the surface acceleration, and substitute the surface acceleration into the new form of Neumann boundary condition to obtain the sound transfer function; S1.5, model training: taking the actual knocking sound and the simulation knocking sound as the input of the deep learning model, and taking the material as the output of the model, the model training is carried out; S2, material identification and classification; S2.1, target object model construction: through scanning the 3D model of target object is constructed; S2.2, target actual knocking sound recording: at least one knocking point is selected on the target object, the position of the knocking point is recorded, and light knocking is performed on the knocking point to obtain the target actual knocking sound; S2.3, target knocking sound simulation: each kind of material in the material library is respectively given to the 3D model of the target object obtained by scanning, and the corresponding vibration mode and transfer function are obtained by calculating the modal and sound transfer function; Input the position of the knocking point, generate the target simulation knocking sound corresponding to each kind of material according to the knocking point and the corresponding vibration mode and transfer function; S2.4, model identification and classification: the target actual knocking sound and the target simulation knocking sound corresponding to multiple materials are input into the deep learning model to obtain the material identification result of the target object. 2.The object material recognition method based on tap sound simulation and deep learning according to claim 1, characterized in that, In the actual knocking sound recording, the several knocking points are uniformly distributed. 3.The object material recognition method based on tap sound simulation and deep learning according to claim 1, characterized in that, The vibration mode is obtained by modal calculation: For a multi-degree-of-freedom damping object, its vibration differential function is written as: wherein, M is a mass matrix, C is a damping matrix, K is a stiffness matrix, u is a displacement matrix, is a velocity matrix, is an acceleration matrix; The surface mesh of the 3D model is converted into a tetrahedral mesh using the isosurface filling method, and the mass matrix is ​​calculated using the converted tetrahedral mesh. M and stiffness matrix K ; The damping matrix is calculated using Rayleigh damping approximation, which is a function of both the mass matrix and the stiffness matrix, and is given by: where, , is a pre-set weight parameter; Substituting the Rayleigh damping into the vibration differential function gives ; According to the vibration differential function, the vibration mode of the model is calculated as follows: wherein is the eigenvector of the differential equation, n is the number of independent vibration modes in the frequency range audible to the human ear, is the modal amplitude, is the modal frequency, is the modal phase.

4. The object material recognition method based on knocking sound simulation and deep learning according to claim 1, characterized in that, The air pressure distribution map of the actual knocking sound collection position is obtained by calculating the sound transfer function, the sound transfer function is obtained, the finite difference time domain method is used for fast calculation of the sound transfer function, which comprises the following steps: The calculation of the acceleration of the vibration transmission of the surface of the object to the air is performed using Neumann boundary conditions as follows: wherein is the surface acceleration, is the density of the medium, is the surface area of the object, is the sound pressure, is the normal vector of the boundary at position x ; The wave propagation problem is handled using 3D wave functions, where the sound pressure satisfies the following equation: wherein c represents the speed of sound in air, is the Laplacian operator; At the same time, the harmonic pressure equation is as follows: ; The harmonic pressure equation is substituted into the equation that the sound pressure needs to satisfy to obtain the Helmholtz equation as follows: wherein , ω is the frequency corresponding to the mode, λ is the wavelength corresponding to the mode; The sound transmission map is obtained by solving the Neumann boundary condition and the Helmholtz equation p x .​ 5.The object material recognition method based on tap sound simulation and deep learning according to claim 1, characterized in that, The listing and encapsulating the factors related to the surface acceleration include one or more of point / area sound source merging calculation, modal vibration calculation, accelerated noise calculation, bubble foam calculation, and finite element calculation of thin shell objects. 6.The object material recognition method based on tap sound simulation and deep learning according to claim 1, characterized in that, The method for generating the simulated knocking sound according to the knocking point, the corresponding vibration mode and the transfer diagram comprises the following steps: inputting the position of the knocking point and the knocking force data, calculating the vibration state of the surface of the object from the vibration mode according to the vibration mode and the sound transfer diagram of the object, combining the sound transfer diagram to calculate the air pressure state around the actual knocking sound collection position, and linearly combining the vibration state and the air pressure state to generate the simulated knocking sound. : wherein, x representing the actual tapping sound collection position, I j representing the modal j sound transmission strength at the actual tapping sound collection position, taking the absolute value of the sound pressure, p q j representing the amplitude of the object surface vibration.​ 7. The object material recognition method based on knocking sound simulation and deep learning according to claim 1, characterized in that, The deep learning model adopts a domain adversarial transfer network, and includes a feature extractor, a label predictor, and a domain classifier. 8.The object material recognition method based on tap sound simulation and deep learning according to claim 7, characterized in that, In the training process of the deep learning model, for the input actual knocking sound and the simulation knocking sound, the deep learning model extracts the sound features of the two respectively by using the feature extractor, inputs the sound features into the label predictor to obtain the material label, and inputs the sound features into the domain classifier to obtain the domain to which the data belongs. 9.The object material recognition method based on tap sound simulation and deep learning according to claim 7, characterized in that, In the model identification and classification step, before the material identification, the parameters of the feature extractor are fixed, the target simulation knocking sound corresponding to multiple materials is input into the feature extractor, the simulation sound features are extracted, and the simulation sound features and the corresponding materials are used to fine-tune the label predictor.

Images