Object material identification method based on knocking sound simulation and deep learning

Abstract

The invention relates to an object material identification method based on knocking sound simulation and deep learning, and the method comprises the steps: carrying out the knocking sound simulation of a given knocking point through a generated model, carrying out the training and comparison through combining with the real knocking sound, and finally carrying out the analysis to obtain the material of a target object. The method comprises two parts of deep learning model training and material identification and classification, and the deep learning model training comprises five steps of training material library construction, object model construction, actual knocking sound recording, knocking sound simulation and model training; the material identification and classification comprises four steps of target object model construction, target actual knocking sound recording, target knocking sound simulation and model identification and classification. Compared with the prior art, the method has the advantages of avoiding collection work of a large amount of actual knocking sounds, being high in material recognition result accuracy and the like.

Description

Technical Field

[0001] The present invention relates to the technical field of material recognition, and in particular to a method for object material recognition based on knocking sound simulation and deep learning. Background Art

[0002] In recent years, the development of augmented reality and virtual reality has brought about a large number of applications in fields such as education and entertainment, which has also led to a growing demand and expectation for the virtual world. The key point of current development is how to define and strengthen the connection between the real world and the virtual world, so as to achieve customization for different users and bring a sense of belonging to users. However, the interaction of only generating models through scanning and importing them into virtual scenes hides some problems: the objects generated by scanning do not contain material information, and the lack of material information will result in users being able to interact with the virtual objects only through a few simple operations that do not involve physical simulation, such as translation, rotation, copying, etc., and because of the lack of physical simulation, these interactions will not be accompanied by matching sound effects. All of these problems will give users a sense of unreality and greatly reduce the user experience.

[0003] In order to solve the above problem, that is, to detect or analyze the material of an object, many institutions at home and abroad have explored and studied various solutions, which can be mainly divided into the following directions: vision-based methods, resonance spectroscopy methods, wireless signal-based methods, tapping methods, and other methods.

[0004] 1. Vision-based method: Use deep learning technology to analyze the image of the object to obtain the material. However, the decoration or pattern on the surface of the object often interferes with the material analysis, so it is difficult to obtain the ideal analysis effect.

[0005] 2. Resonance spectroscopy method: It is often used to identify the material and composition of liquids. It uses the acoustic properties of liquids and tries to find the natural frequency of the target object material, which helps to identify liquids because the natural frequency depends on the composition of the liquid. However, this cannot be used to identify the material of solids and cannot meet our needs.

[0006] 3. Methods based on wireless signals: Among them are methods based on ultra-wideband radar. For example, SiWa has designed a handheld device based on pulse ultra-wideband (IR-UWB) radar. It slides on the wall and transmits signals, receives reflected signals at the same time, and then uses deep learning processing to identify the material (such as steel bars, PVC) and condition (such as whether it is corroded) of objects inside different walls (such as concrete walls, dry walls, etc.). Similarly, there are studies on the material characterization of objects through X-ray related methods. However, methods based on wireless signals require additional hardware equipment, which is inconsistent with the simple XR application scenarios we had imagined before.

[0007] 4. Light-touch tapping method: Light-touch tapping refers to the process of tapping the test object to produce sound, and the sound of tapping is recorded to identify the material of the object. Since the elasticity of the object and the internal friction determined by its material and structure affect the frequency of the sound, deep learning technology can be used to analyze the tapping sound to obtain the material of the 3D model. In the light-touch tapping method, Giordano and other researchers simply analyzed the association between the tapping sound and four given materials (iron, glass, wood, and plastic) based on the physical properties of the material, and then achieved the recognition and classification effect between the two groups (iron-glass and wood-plastic). The scope of application is small and the effect is not good. Although Hamontree and other researchers have expanded the scope of material recognition, this scheme not only requires tapping sounds to identify the material of the object, but also requires the use of a disturbance observer to record the tapping force response, and also requires a robot to tap the object, which cannot meet our expected environmental requirements. Luo et al. only used deep learning to analyze the knocking sound to achieve the recognition of the material of the object. However, it requires the same knocking object (such as a marker) to knock on different target objects, and requires a large number of knocking times (120 knocking sounds are recorded for each object). In addition, this solution cannot recognize the material of objects with new shapes. At the same time, some researchers have applied sound simulation technology to the light touch knocking method. Sterling et al. used the simulated sound as supplementary data to cooperate with the depth camera to identify the shape and material of transparent, closed or highly reflective objects. However, this method uses the object shape and simulated sound as training data into the deep learning model at the same time, resulting in the inability to effectively recognize the material of objects with new shapes. Zhang et al. simulated the process of an object hitting the ground multiple times after free fall, and analyzed the real sound and simulated sound of the object hitting the ground during the whole process to obtain multiple physical properties of the object, such as material and drop height. However, this method needs to simulate the entire process from the beginning of free fall to the final rest of the object. The physical calculation overhead is too large, and it cannot be applied to some materials that are not convenient for free fall experiments, such as common glass and ceramics. Researchers such as Jin designed a sound simulation algorithm that is fully differentiable and contains trainable parameters. Material parameters are also trainable parameters. The simulated results are compared with the real knocking sounds, and the comparison results are back-propagated to optimize the parameters of the simulation algorithm. The material parameters obtained after training are the inferred material parameters. However, due to the limitations of usage scenarios, the recognition accuracy is not high in some scenarios.

[0008] 5. Other methods, such as those based on the principle of heat conduction (using temperature tactile sensors to identify materials with low heat absorption coefficients), are difficult to promote and apply because they require additional equipment and have large limitations on the materials that can be identified. Summary of the invention

[0009] The purpose of the present invention is to select a light-touch tapping method as the basic direction, under the premise that common equipment in daily life environment can be used for material identification, and the equipment requirements are easy to meet and the identification range is sufficient to cover common objects in daily life. A method for object material identification based on knocking sound simulation and deep learning is provided, and a deep learning model is used to identify and classify the material of the knocked object according to the knocking sound. Sound simulation technology is introduced to generate simulated knocking sounds, and the simulated sounds are simultaneously applied to both model training and model classification and recognition to meet the needs of the expected scenarios, that is, it can effectively expand the range of recognized shapes while reducing the workload of collecting knocking sounds, and cope with the material identification of objects of various shapes.

[0010] The purpose of the present invention can be achieved by the following technical solutions:

[0011] A method for object material recognition based on knocking sound simulation and deep learning, comprising the following steps:

[0012] S1, deep learning model training;

[0013] S1.1, build a training material library: select several materials to form a material library according to the target recognition range; select several objects of different shapes in each material to form a training set;

[0014] S1.2, object model construction: construct 3D models of objects in the training set by scanning;

[0015] S1.3, actual knocking sound recording: select several knocking points on the objects in the training set, record the positions of the knocking points, and lightly knock on the knocking points to obtain the actual knocking sound;

[0016] S1.4, knocking sound simulation: assign the corresponding material to the scanned 3D model, calculate the modal and acoustic transfer function to obtain the vibration mode and transfer diagram of the object; input the position of the knocking point, and generate the simulated knocking sound according to the knocking point and the vibration mode and transfer diagram of the object;

[0017] S1.5, model training: use the actual knocking sound and the simulated knocking sound as the input of the deep learning model, and use the material as the output of the model to perform model training;

[0018] S2, material identification and classification;

[0019] S2.1, target object model construction: construct a 3D model of the target object by scanning;

[0020] S2.2, recording the actual knocking sound of the target: select at least one knocking point on the target object, record the position of the knocking point, and lightly knock on the knocking point to obtain the actual knocking sound of the target;

[0021] S2.3, target knocking sound simulation: assign each material in the material library to the 3D model of the scanned target object, and obtain the corresponding vibration mode and transfer diagram by calculating the modal and acoustic transfer function; input the position of the knocking point, and generate the target simulated knocking sound corresponding to each material according to the knocking point and the corresponding vibration mode and transfer diagram;

[0022] S2.4, model recognition and classification: The actual knocking sound of the target and the simulated knocking sounds of the target corresponding to various materials are input into the deep learning model to obtain the material recognition result of the target object.

[0023] In the actual knocking sound recording, a plurality of knocking points are evenly distributed.

[0024] The vibration mode is obtained by modal calculation:

[0025] For a multi-degree-of-freedom damped object, its vibration differential function is written as:

[0026]

[0027] Among them, M is the mass matrix, C is the damping matrix, K is the stiffness matrix, u is the displacement matrix, is the velocity matrix, is the acceleration matrix;

[0028] The surface mesh of the 3D model is converted into a tetrahedral mesh using the isosurface filling method, and the mass matrix M and stiffness matrix K are calculated using the converted tetrahedral mesh;

[0029] The damping matrix is ​​calculated using Rayleigh damping approximation, which is a composite function of the mass matrix and the stiffness matrix in the following form: C = αM + βK, where α and β are preset weight parameters;

[0030] Substituting Rayleigh damping into the vibration differential function, we get

[0031] The vibration mode of the model calculated according to the vibration differential function is as follows:

[0032]

[0033] in is the eigenvector of the differential equation, n is the number of independent vibration modes within the frequency range audible to the human ear, Y is the modal amplitude, ω d is the modal frequency and θ is the modal phase.

[0034] The air pressure distribution diagram of the actual knocking sound collection position is obtained by calculating the sound transfer function, and the sound transfer diagram is obtained, which specifically includes the following steps:

[0035] Neumann boundary conditions are used to calculate the acceleration of the vibration on the surface of the object transmitted to the air. The Neumann boundary conditions are as follows:

[0036]

[0037] Among them, a n (x, t) is the surface acceleration, ρ is the density of the medium, Γ is the surface area of ​​the object, p is the sound pressure, is the normal vector of the boundary at position x;

[0038] The 3D wave function is used to deal with the wave propagation problem, where the sound pressure p(x, t) satisfies the following equation:

[0039]

[0040] Where c is the speed of sound in air, is the Laplace operator;

[0041] At the same time, the following harmonic pressure equation is used: p(x,t)=p(x)e +iωt ;

[0042] Substituting the harmonic pressure equation into the equation that needs to be satisfied for sound pressure, the Helmholtz equation is obtained as follows:

[0043]

[0044] in, ω is the frequency corresponding to the mode, λ is the wavelength corresponding to the mode;

[0045] The Neumann boundary condition and the Helmholtz equation are used to solve p(x), which is the sound transfer diagram.

[0046] The finite difference time domain method is used to quickly calculate the sound transfer map, which specifically includes the following steps:

[0047] Wrap the object with a regular grid and discretize it, and the formula for updating the pressure on the grid over time is as follows:

[0048]

[0049] in, is the sound pressure at the grid point (i,j,k) and time mτ, h is the grid size, τ is the time step, c is the speed of sound in air, α is the constant coefficient controlling the air viscosity damping, is the Laplace operator, and the formula is as follows:

[0050]

[0051] Set absorbing boundary conditions at the edge of the grid to reduce the impact of false reflections;

[0052] The surface of the object is gridded according to the grid, and the new form of the Neumann boundary condition is obtained as follows:

[0053]

[0054] Among them, x b is a point on the surface of the object, x g is the surface point on the grid after rasterization, x r is the reflection point, x r =x g +2(x b -x g ), p r is x r The sound pressure at g is x g The sound pressure at l = |x r -x g |, a n is the surface acceleration, ρ is the density of the medium, O is a high-order small quantity term, representing the error term;

[0055] The factors related to the surface acceleration are listed and packaged for calculation, and the results of each factor are linearly superimposed to obtain the surface acceleration, which is then substituted into the new form of the Neumann boundary condition to obtain the sound transfer diagram.

[0056] The factors related to the surface acceleration are listed and packaged for calculation, including one or more of: point / regional sound source merging calculation, modal vibration calculation, acceleration noise calculation, water foam calculation, and finite element calculation of thin shell objects.

[0057] The method for generating a simulated knocking sound according to the knocking point and the corresponding vibration mode and transfer diagram is as follows: input the position of the knocking point and the knocking force data, calculate the vibration state of the object surface from the vibration mode according to the vibration mode and the sound transfer diagram of the object, and then calculate the air pressure state around the actual knocking sound collection position in combination with the sound transfer diagram, and linearly combine the vibration state and the air pressure state to generate a simulated knocking sound:

[0058]

[0059] Among them, x represents the actual tapping sound collection position, I j represents the sound transmission intensity at the actual knocking sound collection position under mode j, taking the absolute value of the sound pressure p, q j Represents the amplitude of the vibration of the object's surface.

[0060] The deep learning model adopts a domain adversarial transfer network, including a feature extractor, a label predictor, and a domain classifier.

[0061] During the training process of the deep learning model, for the actual knocking sound and the simulated knocking sound input, the deep learning model uses a feature extractor to extract the sound features of the two respectively, inputs the sound features into a label predictor to obtain a material label, and inputs the sound features into a domain classifier to obtain the domain to which the data belongs.

[0062] In the model recognition and classification step, before material recognition, the parameters of the feature extractor are fixed, the target simulated knocking sounds corresponding to a variety of materials are input into the feature extractor, the simulated sound features are extracted, and the extracted simulated sound features and their corresponding materials are used to fine-tune the label predictor.

[0063] Compared with the prior art, the present invention has the following beneficial effects:

[0064] 1. The present invention introduces sound simulation technology to generate simulated knocking sounds, and simultaneously applies the simulated sounds to both model training and model classification and recognition. On the one hand, the use of simulated sounds can reduce the model's demand for a large amount of real knocking sound training data, and also reduce the user's repeated and boring multiple knocking and sound recording during actual use, saving the time and energy of researchers and users; on the other hand, because both the simulated knocking sounds and the real knocking sounds contain the same object shape information, after investing in the two links of deep learning model training and model classification and recognition together, the model is freed from shape restrictions, so that it can recognize the material of objects of new shapes, and can be applied to the recognition of the material of objects of various shapes and maintain a high recognition accuracy.

[0065] 2. The present invention uses both simulated sounds and real knocking sounds to train the deep learning model. Through the training of the domain adversarial transfer network, the common features of the two sounds can be effectively extracted. At the same time, when performing online recognition, the model is fine-tuned by generating target simulated sound signals, thereby ensuring the accuracy and robustness of recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 is a flow chart of the method of the present invention;

[0067] Figure 2 A flowchart of a knocking sound simulation in one embodiment;

[0068] Figure 3 A flowchart of offline training of a deep learning model in an embodiment;

[0069] Figure 4 A flowchart of online recognition of a deep learning model in an embodiment. DETAILED DESCRIPTION

[0070] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0071] This embodiment provides an object material recognition method based on knocking sound simulation and deep learning. It is based on the light touch knocking method and introduces sound simulation technology to generate simulated knocking sounds to reduce the repetitive and boring multiple knocking and sound recording when collecting training data. It is also used to cope with the material recognition of newly added shapes. In addition, a deep learning model is also introduced: Domain Adversarial Neural Networks, which is used to eliminate the influence of experimental environmental factors (such as air pressure differences) that are different between real knocking sounds and simulated sounds and are irrelevant to material recognition, and to extract the common features of simulated sounds and actual knocking sounds, so that the simulated sounds can be applied to network training and applications without difference together with real knocking sounds.

[0072] Specifically, Figure 1 As shown, the following steps are included:

[0073] S1, deep learning model training:

[0074] S1.1, build a training material library: select several materials to form a material library according to the target recognition range; select several objects of different shapes from each material to form a training set. Compared with other tap recognition methods that need to consider the influence of multiple shapes, the present invention does not rely on the diversity of training object shapes, and it is recommended to select only three or so shapes.

[0075] S1.2, object model construction: construct 3D models of objects in the training set by scanning.

[0076] The 3D model of an object can be constructed by any method, including but not limited to manual modeling, 3D scanner scanning, mobile phone camera scanning, etc. In this embodiment, a 3D scanning APP (Spectre3D on the Android platform) available on the market is used to scan the target object and generate the corresponding 3D model. The model file type is .obj. The specific steps are as follows:

[0077] S1.2.1, Place the object in a stable position, leaving enough space for the scanner to move around.

[0078] S1.2.2. The scanner opens the scanning app and enters the scanning process. The scanner holds the phone and walks around the object for 4-5 times. The scanner must take pictures from three perspectives: horizontal, downward, and upward. The object must be completely within the camera's field of view.

[0079] S1.2.3, after shooting, wait until the APP processes and generates a 3D model, the model type is .obj file, and then send the model file to the computer.

[0080] S1.3, actual knocking sound recording: select several evenly distributed knocking points on the objects in the training set, record the positions of the knocking points, and knock lightly on the knocking points to obtain the actual knocking sound.

[0081] In this embodiment, a corner of the mobile phone housing can be used but is not limited to tapping the tapping point with a relatively light force, and a microphone (such as a built-in microphone of the mobile phone) can be used to collect the sound generated by the tapping. In specific implementation, a collection frequency of 44100 Hz can be used but is not limited to.

[0082] S1.4, knocking sound simulation: assign the corresponding material to the scanned 3D model, and obtain the vibration mode and transfer diagram of the object by calculating the modal and acoustic transfer function; input the position of the knocking point, and generate a simulated knocking sound based on the knocking point and the vibration mode and transfer diagram of the object.

[0083] like Figure 2 As shown, the process of generating the simulated knocking sound includes the following steps:

[0084] S1.4.1, import the corresponding parameters of the material of the object;

[0085] S1.4.2, obtain the vibration mode through modal calculation:

[0086] This embodiment uses the traditional linear modal analysis method to calculate the mode of the object. For a multi-degree-of-freedom damped object, its vibration differential function is written as:

[0087]

[0088] Among them, M is the mass matrix, C is the damping matrix, K is the stiffness matrix, u is the displacement matrix, is the velocity matrix, is the acceleration matrix.

[0089] In order to obtain the mass matrix and stiffness matrix of the object, the surface mesh of the 3D model is converted into a tetrahedral mesh using the isosurface filling method, and the mass matrix M and stiffness matrix K are calculated using the converted tetrahedral mesh;

[0090] Since the actual damping matrix is ​​very complex, this embodiment uses Rayleigh Damping to approximate the final result. Rayleigh Damping is a composite function of the mass matrix and the stiffness matrix, and is in the following form: C = αM + βK, where α and β are preset weight parameters;

[0091] Substituting Rayleigh damping into the vibration differential function, we get

[0092] The vibration mode of the model calculated according to the vibration differential function is as follows:

[0093]

[0094] in is the eigenvector of the differential equation, n is the number of independent vibration modes within the frequency range audible to the human ear, Y is the modal amplitude, ω d is the modal frequency and θ is the modal phase.

[0095] S1.4.3, obtaining the air pressure distribution diagram of the actual knocking sound collection position by calculating the sound transfer function to obtain the sound transfer diagram, specifically comprising the following steps:

[0096] Neumann boundary conditions are used to calculate the acceleration of the vibration on the surface of the object transmitted to the air. The Neumann boundary conditions are as follows:

[0097]

[0098] Among them, a n (x, t) is the surface acceleration, ρ is the density of the medium, Γ is the surface area of ​​the object, p is the sound pressure, is the normal vector of the boundary at position x.

[0099] The 3D wave function is used to deal with the wave propagation problem, where the sound pressure p(x, t) satisfies the following equation:

[0100]

[0101] Where c is the speed of sound in air, is the Laplace operator.

[0102] At the same time, the following harmonic pressure equation is used: p(x,t)=p(x)e +iωt ;

[0103] Substituting the harmonic pressure equation into the equation that needs to be satisfied for sound pressure, the Helmholtz equation is obtained as follows:

[0104]

[0105] in, ω is the frequency corresponding to the mode, λ is the wavelength corresponding to the mode;

[0106] The Neumann boundary condition and the Helmholtz equation are used to solve p(x), which is the sound transfer diagram.

[0107] S1.4.4, since the amount of calculation of sound transfer in S1.4.3 is extremely large, this embodiment uses the Finite-Difference Time-Domain (FDTD) method to quickly calculate the sound transfer map using a GPU, specifically including the following steps:

[0108] S1.4.4.1, wrap the object with a regular grid, discretize the object, and get the following formula for the pressure update over time on the grid:

[0109]

[0110] in, is the sound pressure at the grid point (i,j,k) and time mτ, h is the grid size, τ is the time step, c is the speed of sound in air, α is the constant coefficient controlling the air viscosity damping, is the Laplace operator, and the formula is as follows:

[0111]

[0112] S1.4.4.2, set absorbing boundary conditions (ABCs) at the edges of the grid to reduce the impact of false reflections;

[0113] S1.4.4.3, grid the surface of the object according to the grid, and obtain the new form of Neumann boundary condition as follows:

[0114]

[0115] Among them, x b is a point on the surface of the object, x g is the surface point on the grid after rasterization, x r is the reflection point, x r =x g +2(x b -x g ), p r is x r The sound pressure at g is x g The sound pressure at l = |x r -x g |, a n is the surface acceleration, ρ is the density of the medium, O is a high-order small quantity term, representing the error term;

[0116] S1.4.4.4, in order to quickly calculate the surface acceleration, the factors related to the surface acceleration are listed and packaged for calculation, including but not limited to: one or more of the following: point / area sound source combined calculation, modal vibration calculation, acceleration noise calculation, water foam calculation, and finite element calculation of thin shell objects. The results of each factor are linearly superimposed to obtain the surface acceleration, which is substituted into the new form of the Neumann boundary condition to obtain the sound transfer diagram.

[0117] S1.4.5, generate simulated knocking sound according to the knocking point and the corresponding vibration mode and transfer diagram: input the position of the knocking point and the knocking force data, calculate the vibration state of the object surface from the vibration mode according to the vibration mode and sound transfer diagram of the object, and then calculate the air pressure state around the actual knocking sound collection position in combination with the sound transfer diagram, and linearly combine the vibration state and the air pressure state to generate the simulated knocking sound:

[0118]

[0119] Among them, x represents the actual tapping sound collection position, I j represents the sound transmission intensity at the actual knocking sound collection position under mode j, taking the absolute value of the sound pressure p, q j Represents the amplitude of the vibration of the object's surface.

[0120] S1.5, model training: use the actual knocking sound and the simulated knocking sound as the input of the deep learning model, and use the material as the output of the model to perform model training.

[0121] In this embodiment, the deep learning model adopts a domain adversarial transfer network (DANN), including a feature extractor, a label predictor and a domain classifier.

[0122] like Figure 3 As shown in the figure, during the training process of the deep learning model, after the input actual knocking sound and simulated knocking sound are grouped and paired and the data is preprocessed, the deep learning model uses the feature extractor to extract the sound features of the two respectively, inputs the sound features into the label predictor to obtain the material label, and inputs the sound features into the domain classifier to obtain the domain to which the data belongs. The domain classifier is used to achieve domain confrontation, and outputs a prediction of whether the input feature comes from simulated sound or real sound, that is, the domain label. The feature extractor tries to extract features that the domain classifier cannot recognize during the training process, thus forming a confrontation. Through confrontation training, the feature extractor is made to extract the common features of the simulated signal and the actual signal as much as possible.

[0123] S2, material identification and classification:

[0124] S2.1, target object model construction: construct a 3D model of the target object by scanning; the implementation process of this step can refer to step S1.2, and also use the existing 3D scanning APP on the market (Spectre3D on the Android platform) to scan the target object and generate the corresponding 3D model. The model file type is .obj.

[0125] S2.2, recording the actual knocking sound of the target: select at least one knocking point on the target object, record the position of the knocking point, and lightly knock on the knocking point to obtain the actual knocking sound of the target.

[0126] The implementation process of this step can refer to step S1.3, but this step does not need to collect the knocking sounds of multiple knocking points like in training. It only needs to select any number (greater than or equal to 1) of knocking points to knock and collect the sounds.

[0127] S2.3, target knocking sound simulation: assign each material in the material library to the 3D model of the scanned target object, and obtain the corresponding vibration mode and transfer diagram by calculating the modal and acoustic transfer function; input the position of the knocking point, and generate the target simulated knocking sound corresponding to each material according to the knocking point and the corresponding vibration mode and transfer diagram.

[0128] S2.4, model recognition and classification: The actual knocking sound of the target and the simulated knocking sounds of the target corresponding to various materials are input into the deep learning model to obtain the material recognition result of the target object.

[0129] like Figure 4 As shown in the figure, the trained deep learning model is applied, the parameters of the feature extractor are fixed, and the simulated knocking sounds corresponding to various materials are input into the feature extractor after data preprocessing to extract the simulated sound features. The extracted simulated sound features and their corresponding materials are used to fine-tune the label predictor. After fine-tuning, the actual knocking sound of the target is input into the feature extractor to obtain the actual sound features. The actual sound features and all simulated sound features are input into the fine-tuned label predictor, and the material recognition result of the target object is output.

[0130] In summary, the present invention uses a sound simulation method to supplement the percussion sound data set, and at the same time, the training results can take into account the shape factor at the same time, and can be applied to the recognition of materials of objects of various shapes. In addition, the simulated sound and the real percussion sound are used at the same time to train the deep learning model, so that the simulated sound can be effectively applied to the training and application of the network model as the real percussion sound, thereby improving the recognition accuracy of the model.

[0131] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning, or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A method for object material recognition based on knocking sound simulation and deep learning, characterized in that: The following steps are involved: S1, deep learning model training; S1.1, build a training material library: select several materials to form a material library according to the target recognition range; select several objects of different shapes in each material to form a training set; S1.2, object model construction: construct 3D models of objects in the training set by scanning; S1.3, actual knocking sound recording: select several knocking points on the objects in the training set, record the positions of the knocking points, and lightly knock on the knocking points to obtain the actual knocking sound; S1.4, knocking sound simulation: assign the corresponding material to the scanned 3D model, calculate the modal and acoustic transfer function to obtain the vibration mode and transfer diagram of the object; input the position of the knocking point, and generate the simulated knocking sound according to the knocking point and the vibration mode and transfer diagram of the object; S1.5, model training: use the actual knocking sound and the simulated knocking sound as the input of the deep learning model, and use the material as the output of the model to perform model training; S2, material identification and classification; S2.1, target object model construction: construct a 3D model of the target object by scanning; S2.2, recording the actual knocking sound of the target: select at least one knocking point on the target object, record the position of the knocking point, and lightly knock on the knocking point to obtain the actual knocking sound of the target; S2.3, target knocking sound simulation: assign each material in the material library to the 3D model of the scanned target object, and obtain the corresponding vibration mode and transfer diagram by calculating the modal and acoustic transfer function; input the position of the knocking point, and generate the target simulated knocking sound corresponding to each material according to the knocking point and the corresponding vibration mode and transfer diagram; S2.4, model recognition and classification: The actual knocking sound of the target and the simulated knocking sounds of the target corresponding to various materials are input into the deep learning model to obtain the material recognition result of the target object.

2. The object material recognition method based on knocking sound simulation and deep learning according to claim 1, characterized in that: In the actual knocking sound recording, a plurality of knocking points are evenly distributed.

3. The object material recognition method based on knocking sound simulation and deep learning according to claim 1, characterized in that: The vibration mode is obtained by modal calculation: For a multi-degree-of-freedom damped object, its vibration differential function is written as: Among them, M is the mass matrix, C is the damping matrix, K is the stiffness matrix, u is the displacement matrix, is the velocity matrix, is the acceleration matrix; The surface mesh of the 3D model is converted into a tetrahedral mesh using the isosurface filling method, and the mass matrix M and stiffness matrix K are calculated using the converted tetrahedral mesh; The damping matrix is ​​calculated using Rayleigh damping approximation, which is a composite function of the mass matrix and the stiffness matrix in the following form: C = αM + βK, where α and β are preset weight parameters; Substituting Rayleigh damping into the vibration differential function, we get The vibration mode of the model calculated according to the vibration differential function is as follows: in is the eigenvector of the differential equation, n is the number of independent vibration modes within the frequency range audible to the human ear, Y is the modal amplitude, ω d is the modal frequency and θ is the modal phase.

4. The object material recognition method based on knocking sound simulation and deep learning according to claim 1, characterized in that: The air pressure distribution diagram of the actual knocking sound collection position is obtained by calculating the sound transfer function, and the sound transfer diagram is obtained, which specifically includes the following steps: Neumann boundary conditions are used to calculate the acceleration of the vibration on the surface of the object transmitted to the air. The Neumann boundary conditions are as follows: Among them, a n (x, t) is the surface acceleration, ρ is the density of the medium, Γ is the surface area of ​​the object, p is the sound pressure, is the normal vector of the boundary at position x; The 3D wave function is used to deal with the wave propagation problem, where the sound pressure p(x, t) satisfies the following equation: Where c is the speed of sound in air, is the Laplace operator; At the same time, the following harmonic pressure equation is used: p(x,t)=p(x)e +iωt ; Substituting the harmonic pressure equation into the equation that needs to be satisfied for sound pressure, the Helmholtz equation is obtained as follows: in, ω is the frequency corresponding to the mode, λ is the wavelength corresponding to the mode; The Neumann boundary condition and the Helmholtz equation are used to solve p(x), which is the sound transfer diagram.

5. The object material recognition method based on knocking sound simulation and deep learning according to claim 4 is characterized in that: The finite difference time domain method is used to quickly calculate the sound transfer map, which specifically includes the following steps: Wrap the object with a regular grid and discretize it, and the formula for updating the pressure on the grid over time is as follows: in, is the sound pressure at the grid point (i,j,k) and time mτ, h is the grid size, τ is the time step, c is the speed of sound in air, α is the constant coefficient controlling the air viscosity damping, is the Laplace operator, and the formula is as follows: Set absorbing boundary conditions at the edge of the grid to reduce the impact of false reflections; The surface of the object is gridded according to the grid, and the new form of the Neumann boundary condition is obtained as follows: Among them, x b is a point on the surface of the object, x g is the surface point on the grid after rasterization, x r is the reflection point, x r =x g +2(x b -x g ), p r is x r The sound pressure at g is x g The sound pressure at l = |x r -x g |, a n is the surface acceleration, ρ is the density of the medium, O is a high-order small quantity term, representing the error term; The factors related to the surface acceleration are listed and packaged for calculation, and the results of each factor are linearly superimposed to obtain the surface acceleration, which is then substituted into the new form of the Neumann boundary condition to obtain the sound transfer diagram.

6. The object material recognition method based on knocking sound simulation and deep learning according to claim 5, characterized in that: The factors related to the surface acceleration are listed and packaged for calculation, including one or more of: point / regional sound source merging calculation, modal vibration calculation, acceleration noise calculation, water foam calculation, and finite element calculation of thin shell objects.

7. The object material recognition method based on knocking sound simulation and deep learning according to claim 1, characterized in that: The method for generating a simulated knocking sound according to the knocking point and the corresponding vibration mode and transfer diagram is as follows: input the position of the knocking point and the knocking force data, calculate the vibration state of the object surface from the vibration mode according to the vibration mode and the sound transfer diagram of the object, and then calculate the air pressure state around the actual knocking sound collection position in combination with the sound transfer diagram, and linearly combine the vibration state and the air pressure state to generate a simulated knocking sound: Among them, x represents the actual tapping sound collection position, I j represents the sound transmission intensity at the actual knocking sound collection position under mode j, taking the absolute value of the sound pressure p, q j Represents the amplitude of the vibration of the object's surface.

8. The object material recognition method based on knocking sound simulation and deep learning according to claim 1, characterized in that: The deep learning model adopts a domain adversarial transfer network, including a feature extractor, a label predictor, and a domain classifier.

9. The object material recognition method based on knocking sound simulation and deep learning according to claim 8, characterized in that: During the training process of the deep learning model, for the actual knocking sound and the simulated knocking sound input, the deep learning model uses a feature extractor to extract the sound features of the two respectively, inputs the sound features into a label predictor to obtain a material label, and inputs the sound features into a domain classifier to obtain the domain to which the data belongs.

10. The object material recognition method based on knocking sound simulation and deep learning according to claim 8, characterized in that: In the model recognition and classification step, before material recognition, the parameters of the feature extractor are fixed, the target simulated knocking sounds corresponding to a variety of materials are input into the feature extractor, the simulated sound features are extracted, and the extracted simulated sound features and their corresponding materials are used to fine-tune the label predictor.

Images