Visual and auditory cross-modal object material retrieval method and system

Abstract

The invention provides a visual and auditory cross-modal object material retrieval method and system, and relates to the technical field of artificial intelligence. The visual and auditory cross-modal object material retrieval system comprises a data acquisition unit for acquiring visual images containing objects made of various materials and corresponding audio data; feature extraction: feature extraction is performed on the visual and auditory data, visual features include but are not limited to colors, textures and shapes, and auditory features include but are not limited to spectrum features and time domain features; feature fusion: designing a fusion algorithm, and effectively fusing visual and auditory features to form cross-modal feature representation; and retrieving a model, wherein the model can learn a mapping relationship between the cross-modal features and the material of the object. By combining visual and auditory information, the system can more comprehensively understand the characteristics of the object, the visual information can provide the appearance characteristics of the object, the auditory information can provide the dynamic characteristics about the material of the object, and the fusion is beneficial to improving the accuracy of material identification.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a visual and auditory cross-modal object material retrieval method and system. Background Art

[0002] In daily life, people not only use vision to identify objects, but also use other senses such as hearing, touch and smell. When we hear the rumbling of thunder, it may indicate that a heavy rain is coming; when buying watermelons, we judge whether the watermelons are ripe by judging the sound produced by knocking on them; the "smell" in the Chinese medicine diagnosis of "look, smell, ask, and cut" and the Western medicine uses a stethoscope to listen to the patient's heartbeat to judge the patient's physical condition, etc., are all applications of sound in our daily production and life.

[0003] In the field of computer vision and auditory processing, the recognition of object materials is an important research topic, which has wide applications in many fields such as robot navigation, virtual reality, augmented reality, and intelligent monitoring. Traditional object material recognition methods mainly rely on single-modal information, for example, using only visual images or only audio signals. However, single-modal information is often limited by specific environmental conditions and perception limitations, resulting in low recognition accuracy and limited generalization ability.

[0004] For example, in the visual field, the appearance features of an object (such as color, texture, shape) may change greatly due to factors such as changes in lighting and differences in viewing angles, making it difficult for material recognition methods that rely solely on visual information to cope with complex and changing real-world environments. In the auditory field, although sound signals can provide the dynamic characteristics of object materials, the use of auditory information alone also faces problems such as noise interference and the complexity of sound signals. In addition, although existing cross-modal learning methods have solved the limitations of single modal information to a certain extent, these methods often focus on specific types of modal combinations and do not fully consider the complementarity of visual and auditory information. In addition, these methods still have deficiencies in feature fusion, model design, and user interaction, which limits their performance and user experience in practical applications. Therefore, technicians in this field provide a visual and auditory cross-modal object material retrieval method and system to solve the problems raised in the above background technology. Summary of the invention

[0005] 1. Technical issues to be resolved

[0006] In view of the deficiencies in the prior art, the present invention provides a visual and auditory cross-modal object material retrieval method and system, which solves the problems of feature fusion, model design and user interaction in the prior art.

[0007] (II) Technical solution

[0008] To achieve the above objectives, the present invention is implemented through the following technical solutions: a visual and auditory cross-modal object material retrieval system, comprising:

[0009] Data collection, collecting visual images and corresponding audio data of objects of various materials;

[0010] Feature extraction: extracting features from visual and auditory data respectively. Visual features include but are not limited to color, texture, and shape, and auditory features include but are not limited to spectrum features and time domain features.

[0011] Feature fusion: design a fusion algorithm to effectively fuse visual and auditory features to form a cross-modal feature representation;

[0012] Retrieval model: build a deep learning-based retrieval model that can learn the mapping relationship between cross-modal features and object materials;

[0013] Retrieval and feedback: Implement a retrieval interface where users input or upload visual and auditory data, the system outputs material retrieval results, and optimizes the model based on user feedback;

[0014] Experiments and evaluation, use indicators such as accuracy, recall, F1 score to evaluate model performance. At the same time, user research is conducted to evaluate the actual usage effect and user experience of the system.

[0015] Preferably, the data collection comprises the following steps:

[0016] Visual data collection:

[0017] S1. Select the object and determine the type of material of the object to be collected, such as metal, wood, plastic and fabric;

[0018] S2. Set up the environment, set different lighting conditions and shooting angles for each object to simulate the diversity in the real world;

[0019] S3. Image acquisition: Use a high-definition video camera or camera to capture the image of the object in a controlled environment to ensure that the image is clear and covers all parts of the object;

[0020] S4. Image preprocessing: preprocess the collected images, including resizing, cropping and normalization, to adapt to the subsequent feature extraction algorithm. The formula for image normalization is:

[0021]

[0022] Among them, I is the original image, I min and I max are the minimum and maximum pixel values ​​in the image, respectively;

[0023] Auditory data collection:

[0024] S1. Select objects, same as visual data collection;

[0025] S2. Set up the environment and ensure that the background noise is as low as possible to reduce interference with the sound data;

[0026] S3. Sound collection, using a high-fidelity microphone to record the sounds of objects in different interactive modes, including knocking, rubbing or squeezing;

[0027] S4. Sound preprocessing: preprocessing the recorded sound, including noise reduction, segmentation and normalization. The formula for sound normalization is:

[0028]

[0029] Among them, S is the original sound signal, S mean and S std are the mean and standard deviation of the signal respectively;

[0030] Data synchronization and tagging:

[0031] S1. Synchronous acquisition, ensuring that visual and auditory data are collected synchronously to facilitate subsequent cross-modal analysis;

[0032] S2. Data labeling: Detailed labeling of the collected data, including the material of the object, the environmental conditions of collection, and the interaction method.

[0033] Preferably, the feature extraction comprises the following steps:

[0034] Visual feature extraction:

[0035] S1. Image preprocessing, including image scaling, cropping and normalization, to adapt to the subsequent feature extraction model;

[0036] S2. Edge detection, use Canny edge detection to extract image edge information. The specific calculation formula is:

[0037] Gaussian smoothing:

[0038]

[0039] Where σ is the standard deviation of the Gaussian kernel, and x and y are the coordinates of the position within the kernel;

[0040] Gradient calculation:

[0041]

[0042] Among them, G x and G y are the gradients of the image in the x and y directions respectively;

[0043] S3. Texture feature extraction, using gray level co-occurrence matrix to extract texture features, and the calculation formula is:

[0044]

[0045] Where I(p,q) is the pixel value at position (p,q) in the image, i and j are grayscale values, θ is the direction, d is the distance, Δx and Δy are the offsets calculated based on the direction and distance;

[0046] S4. Color feature extraction, using color histogram, color moment or color aggregation vector to extract color information;

[0047] S5. Deep learning model extraction, using pre-trained convolutional neural network models to extract high-level visual features;

[0048] Auditory feature extraction:

[0049] S1. Audio preprocessing, including noise reduction, segmentation, and normalization, to prepare the audio signal;

[0050] S2. Spectral feature extraction, using short-time Fourier transform to extract the spectral features of the audio signal, and the formula is:

[0051]

[0052] Where x[n] is the discrete time signal, ω[n] is the discrete form of the window function, m is the discrete time index, k is the discrete frequency index, and N is the number of DFT points;

[0053] S3. Time domain feature extraction, including zero crossing rate, energy and amplitude;

[0054] S4. Deep learning model extraction uses a pre-trained recurrent neural network to extract high-level features of the audio signal, and the calculation formula is:

[0055] h t =f(Ux t +Wh t-1 +b)

[0056] y t =g(Vh t +c)

[0057] Among them, h t is the hidden state at time step t, y t is the output at time step t, x tis the input at time step t, U is the weight matrix from input to hidden state, W is the weight matrix from previous hidden state to current hidden state, V is the weight matrix from hidden state to output, b and c are bias terms, f is the activation function, usually nonlinear activation function is used in hidden layer, g is the activation function of output layer, depending on the task, it may be linear function, softmax function.

[0058] Preferably, the feature fusion comprises the following steps:

[0059] S1. Feature extraction, extracting features from visual and auditory data respectively;

[0060] S2. Feature alignment, ensuring that visual and auditory features are aligned in time or space;

[0061] S3. Feature fusion strategy selection, early fusion: directly concatenate features of different modalities; late fusion: independently process and make decisions on the features of each modality, and then combine the results; intermediate layer fusion: feature fusion is performed in the intermediate layers of the deep learning model, which usually involves designing a shared network structure in which features of different modalities are combined in an intermediate layer;

[0062] S4. Fusion model design: design the fusion model according to the selected fusion strategy;

[0063] S5. Train the fusion model and use the labeled data to train the fusion model.

[0064] Preferably, the retrieval model comprises the following steps:

[0065] S1. Model architecture design: design a deep learning model that can process the fused features and perform material classification or regression;

[0066] S2. Loss function selection, select the appropriate loss function according to the task type;

[0067] S3. Optimization algorithm selection: select an optimization algorithm to minimize the loss function. Commonly used optimization algorithms include stochastic gradient descent (SGD), Adam, and RMSprop.

[0068] S4. Model training, using labeled data to train the model;

[0069] S5. Model evaluation and tuning: Use the validation set to evaluate model performance and adjust the model structure or parameters based on the evaluation results.

[0070] Preferably, the retrieval and feedback comprises the following steps:

[0071] S1. User input processing: Users upload or input visual and auditory data through the interface, and pre-process the user input data, such as image scaling and audio noise reduction, to match the model input requirements;

[0072] S2. Feature extraction and fusion: extract features from the processed visual and auditory data respectively, and fuse the extracted features to form a cross-modal feature representation;

[0073] S3. Retrieval model processing, inputting the fused features into the trained retrieval model, and the model outputs the material retrieval results, which are usually the classification labels of the materials or the continuous values ​​of the material attributes;

[0074] S4. Result display: display the material results output by the model to the user in a user-friendly manner;

[0075] S5. User feedback collection: Provide a feedback mechanism so that users can evaluate the accuracy of search results. The feedback can be a simple rating or a more detailed description.

[0076] S6. Model optimization: adjust model parameters or structure based on user feedback to improve retrieval accuracy and user satisfaction.

[0077] Preferably, the experiment and evaluation include the following steps:

[0078] S1. Experimental design, data set preparation, ensuring there is a test set containing labeled data to evaluate model performance and set experimental parameters;

[0079] S2. Model evaluation: Use appropriate performance indicators to evaluate the model. You can use accuracy, recall, F1 score, precision, and area under the ROC curve. Construct a confusion matrix to analyze the classification performance of the model, especially for multi-class classification problems.

[0080] S3. The performance index calculation formula includes:

[0081] Accuracy:

[0082]

[0083] Recall:

[0084]

[0085] Accuracy:

[0086]

[0087] F1 score:

[0088]

[0089] S4. Statistical significance test, use statistical tests to determine significant differences in model performance;

[0090] S5. Error analysis: analyzing cases where the model makes incorrect predictions to understand the model’s weaknesses and potential directions for improvement;

[0091] S6. Model optimization: adjust the model structure, parameters or training strategy based on the evaluation results.

[0092] 9. A method for using a visual and auditory cross-modal object material retrieval system, comprising the following steps:

[0093] S1. Data collection, collecting visual images of objects of various materials and corresponding audio data;

[0094] S2. Feature extraction: extracting features from visual and auditory data respectively. Visual features include but are not limited to color, texture, and shape. Auditory features include but are not limited to spectrum features and time domain features.

[0095] S3. Feature fusion: design a fusion algorithm to effectively fuse visual and auditory features to form a cross-modal feature representation;

[0096] S4. Retrieval model: build a deep learning-based retrieval model that can learn the mapping relationship between cross-modal features and object materials;

[0097] S5. Retrieval and feedback: implement a retrieval interface where users input or upload visual and auditory data, the system outputs material retrieval results, and optimizes the model based on user feedback;

[0098] S6. Experiments and evaluation, use indicators such as accuracy, recall, F1 score to evaluate model performance. At the same time, conduct user research to evaluate the actual usage effect and user experience of the system.

[0099] (III) Beneficial effects

[0100] The present invention provides a method and system for visual and auditory cross-modal object material retrieval, which has the following beneficial effects:

[0101] 1. In the present invention, by combining visual and auditory information, the system can understand the characteristics of an object more comprehensively. Visual information can provide the appearance characteristics of an object, such as color, texture, and shape, while auditory information can provide dynamic characteristics of the object's material, such as knocking or friction sounds. This fusion helps to improve the accuracy of material recognition.

[0102] 2. In the present invention, since the system learns both visual and auditory features, it may have better generalization capabilities for material recognition in different environments and conditions, which means that the system can still maintain high recognition performance when faced with new, unseen data.

[0103] 3. In the present invention, the technical solution can adapt to different application scenarios, such as robot navigation, virtual reality, augmented reality, etc., to provide richer interactive experience and more accurate environmental perception capabilities for these fields, and provide a user-friendly retrieval interface. Users can easily upload or input visual and auditory data, and the system outputs material retrieval results. This interactive method is intuitive and easy to use. BRIEF DESCRIPTION OF THE DRAWINGS

[0104] Figure 1 It is a structural schematic diagram of the overall process of the present invention. DETAILED DESCRIPTION

[0105] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0106] Embodiment 1:

[0107] like Figure 1 , an embodiment of the present invention provides a visual and auditory cross-modal object material retrieval system, comprising:

[0108] Data collection, collecting visual images and corresponding audio data of objects of various materials;

[0109] Feature extraction: extracting features from visual and auditory data respectively. Visual features include but are not limited to color, texture, and shape, and auditory features include but are not limited to spectrum features and time domain features.

[0110] Feature fusion: design a fusion algorithm to effectively fuse visual and auditory features to form a cross-modal feature representation;

[0111] Retrieval model: build a deep learning-based retrieval model that can learn the mapping relationship between cross-modal features and object materials;

[0112] Retrieval and feedback: Implement a retrieval interface where users input or upload visual and auditory data, the system outputs material retrieval results, and optimizes the model based on user feedback;

[0113] Experiments and evaluation, use indicators such as accuracy, recall, F1 score to evaluate model performance. At the same time, user research is conducted to evaluate the actual usage effect and user experience of the system.

[0114] Data collection includes the following steps:

[0115] Visual data collection:

[0116] S1. Select the object and determine the type of material of the object to be collected, such as metal, wood, plastic and fabric;

[0117] S2. Set up the environment, set different lighting conditions and shooting angles for each object to simulate the diversity in the real world;

[0118] S3. Image acquisition: Use a high-definition video camera or camera to capture the image of the object in a controlled environment to ensure that the image is clear and covers all parts of the object;

[0119] S4. Image preprocessing: preprocess the collected images, including resizing, cropping and normalization, to adapt to the subsequent feature extraction algorithm. The formula for image normalization is:

[0120]

[0121] Among them, I is the original image, I min and I max are the minimum and maximum pixel values ​​in the image, respectively;

[0122] Auditory data collection:

[0123] S1. Select objects, same as visual data collection;

[0124] S2. Set up the environment and ensure that the background noise is as low as possible to reduce interference with the sound data;

[0125] S3. Sound collection, using a high-fidelity microphone to record the sounds of objects in different interactive modes, including knocking, rubbing or squeezing;

[0126] S4. Sound preprocessing: preprocessing the recorded sound, including noise reduction, segmentation and normalization. The formula for sound normalization is:

[0127]

[0128] Among them, S is the original sound signal, S mean and S std are the mean and standard deviation of the signal respectively;

[0129] Data synchronization and tagging:

[0130] S1. Synchronous acquisition, ensuring that visual and auditory data are collected synchronously to facilitate subsequent cross-modal analysis;

[0131] S2. Data labeling: Detailed labeling of the collected data, including the material of the object, the environmental conditions of collection, and the interaction method.

[0132] Feature extraction includes the following steps:

[0133] Visual feature extraction:

[0134] S1. Image preprocessing, including image scaling, cropping and normalization, to adapt to the subsequent feature extraction model;

[0135] S2. Edge detection, use Canny edge detection to extract image edge information. The specific calculation formula is:

[0136] Gaussian smoothing:

[0137]

[0138] Where σ is the standard deviation of the Gaussian kernel, and x and y are the coordinates of the position within the kernel;

[0139] Gradient calculation:

[0140]

[0141]

[0142] Among them, G x and G y are the gradients of the image in the x and y directions respectively;

[0143] S3. Texture feature extraction, using gray level co-occurrence matrix to extract texture features, and the calculation formula is:

[0144]

[0145] Where I(p,q) is the pixel value at position (p,q) in the image, i and j are grayscale values, θ is the direction, d is the distance, Δx and Δy are the offsets calculated based on the direction and distance;

[0146] S4. Color feature extraction, using color histogram, color moment or color aggregation vector to extract color information;

[0147] S5. Deep learning model extraction, using pre-trained convolutional neural network models to extract high-level visual features;

[0148] Auditory feature extraction:

[0149] S1. Audio preprocessing, including noise reduction, segmentation, and normalization, to prepare the audio signal;

[0150] S2. Spectral feature extraction, using short-time Fourier transform to extract the spectral features of the audio signal, and the formula is:

[0151]

[0152] Where x[n] is the discrete time signal, ω[n] is the discrete form of the window function, m is the discrete time index, k is the discrete frequency index, and N is the number of DFT points;

[0153] S3. Time domain feature extraction, including zero crossing rate, energy and amplitude;

[0154] S4. Deep learning model extraction uses a pre-trained recurrent neural network to extract high-level features of the audio signal, and the calculation formula is:

[0155] h t =f(Ux t +Wh t-1 +b)

[0156] y t =g(Vh t +c)

[0157] Among them, h t is the hidden state at time step t, y t is the output at time step t, x t is the input at time step t, U is the weight matrix from input to hidden state, W is the weight matrix from previous hidden state to current hidden state, V is the weight matrix from hidden state to output, b and c are bias terms, f is the activation function, usually nonlinear activation function is used in hidden layer, g is the activation function of output layer, depending on the task, it may be linear function, softmax function.

[0158] Feature fusion includes the following steps:

[0159] S1. Feature extraction, extracting features from visual and auditory data respectively;

[0160] S2. Feature alignment, ensuring that visual and auditory features are aligned in time or space;

[0161] S3. Feature fusion strategy selection, early fusion: directly concatenate features of different modalities; late fusion: independently process and make decisions on the features of each modality, and then combine the results; intermediate layer fusion: feature fusion is performed in the intermediate layers of the deep learning model, which usually involves designing a shared network structure in which features of different modalities are combined in an intermediate layer;

[0162] S4. Fusion model design: design the fusion model according to the selected fusion strategy;

[0163] S5. Train the fusion model and use the labeled data to train the fusion model.

[0164] The retrieval model consists of the following steps:

[0165] S1. Model architecture design: design a deep learning model that can process the fused features and perform material classification or regression;

[0166] S2. Loss function selection, select the appropriate loss function according to the task type;

[0167] S3. Optimization algorithm selection: select an optimization algorithm to minimize the loss function. Commonly used optimization algorithms include stochastic gradient descent (SGD), Adam, and RMSprop.

[0168] S4. Model training, using labeled data to train the model;

[0169] S5. Model evaluation and tuning: Use the validation set to evaluate model performance and adjust the model structure or parameters based on the evaluation results.

[0170] Retrieval and feedback includes the following steps:

[0171] S1. User input processing: Users upload or input visual and auditory data through the interface, and pre-process the user input data, such as image scaling and audio noise reduction, to match the model input requirements;

[0172] S2. Feature extraction and fusion: extract features from the processed visual and auditory data respectively, and fuse the extracted features to form a cross-modal feature representation;

[0173] S3. Retrieval model processing, inputting the fused features into the trained retrieval model, and the model outputs the material retrieval results, which are usually the classification labels of the materials or the continuous values ​​of the material attributes;

[0174] S4. Result display: display the material results output by the model to the user in a user-friendly manner;

[0175] S5. User feedback collection: Provide a feedback mechanism so that users can evaluate the accuracy of search results. The feedback can be a simple rating or a more detailed description.

[0176] S6. Model optimization: adjust model parameters or structure based on user feedback to improve retrieval accuracy and user satisfaction.

[0177] The experiment and evaluation include the following steps:

[0178] S1. Experimental design, data set preparation, ensuring there is a test set containing labeled data to evaluate model performance and set experimental parameters;

[0179] S2. Model evaluation: Use appropriate performance indicators to evaluate the model. You can use accuracy, recall, F1 score, precision, and area under the ROC curve. Construct a confusion matrix to analyze the classification performance of the model, especially for multi-class classification problems.

[0180] S3. The performance index calculation formula includes:

[0181] Accuracy:

[0182]

[0183] Recall:

[0184]

[0185] Accuracy:

[0186]

[0187] F1 score:

[0188]

[0189] S4. Statistical significance test, use statistical tests to determine significant differences in model performance;

[0190] S5. Error analysis: analyzing cases where the model makes incorrect predictions to understand the model’s weaknesses and potential directions for improvement;

[0191] S6. Model optimization: adjust the model structure, parameters or training strategy based on the evaluation results.

[0192] 10. A method for using a visual and auditory cross-modal object material retrieval system, comprising the following steps:

[0193] S1. Data collection, collecting visual images of objects of various materials and corresponding audio data;

[0194] S2. Feature extraction: extracting features from visual and auditory data respectively. Visual features include but are not limited to color, texture, and shape. Auditory features include but are not limited to spectrum features and time domain features.

[0195] S3. Feature fusion: design a fusion algorithm to effectively fuse visual and auditory features to form a cross-modal feature representation;

[0196] S4. Retrieval model: build a deep learning-based retrieval model that can learn the mapping relationship between cross-modal features and object materials;

[0197] S5. Retrieval and feedback: implement a retrieval interface where users input or upload visual and auditory data, the system outputs material retrieval results, and optimizes the model based on user feedback;

[0198] S6. Experiments and evaluation, use indicators such as accuracy, recall, F1 score to evaluate model performance. At the same time, conduct user research to evaluate the actual usage effect and user experience of the system.

[0199] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A visual and auditory cross-modal object material retrieval system, characterized by: include: Data collection, collecting visual images and corresponding audio data of objects of various materials; Feature extraction: extracting features from visual and auditory data respectively. Visual features include but are not limited to color, texture, and shape, and auditory features include but are not limited to spectrum features and time domain features. feature Fusion: design a fusion algorithm to effectively fuse visual and auditory features to form a cross-modal feature representation; Retrieval model: build a deep learning-based retrieval model that can learn the mapping relationship between cross-modal features and object materials; Retrieval and feedback: Implement a retrieval interface where users input or upload visual and auditory data, the system outputs material retrieval results, and optimizes the model based on user feedback; Experiments and evaluation, use indicators such as accuracy, recall, F1 score to evaluate model performance. At the same time, user research is conducted to evaluate the actual usage effect and user experience of the system.

2. The visual and auditory cross-modal object material retrieval system according to claim 1, characterized in that: The data collection comprises the following steps: Visual Data Collection: S1. Select the object and determine the type of material of the object to be collected, such as metal, wood, plastic and fabric; S2. Set up the environment, set different lighting conditions and shooting angles for each object to simulate the diversity in the real world; S3. Image acquisition: Use a high-definition video camera or camera to capture the image of the object in a controlled environment to ensure that the image is clear and covers all parts of the object; S4. Image preprocessing: preprocess the collected images, including resizing, cropping and normalization, to adapt to the subsequent feature extraction algorithm. The formula for image normalization is: Among them, I is the original image, I min and I max are the minimum and maximum pixel values ​​in the image, respectively; Auditory data collection: S1. Select objects, same as visual data collection; S2. Set up the environment and ensure that the background noise is as low as possible to reduce interference with the sound data; S3. Sound collection, using a high-fidelity microphone to record the sounds of objects in different interactive modes, including knocking, rubbing or squeezing; S4. Sound preprocessing: preprocess the recorded sound, including noise reduction, segmentation and normalization. The formula for sound normalization is: Among them, S is the original sound signal, S mean and S std are the mean and standard deviation of the signal respectively; Data synchronization and tagging: S1. Synchronous acquisition, ensuring that visual and auditory data are collected synchronously to facilitate subsequent cross-modal analysis; S2. Data labeling: Detailed labeling of the collected data, including the material of the object, the environmental conditions of collection, and the interaction method.

3. The visual and auditory cross-modal object material retrieval system according to claim 1, characterized in that: The feature extraction comprises the following steps: Visual feature extraction: S1. Image preprocessing, including image scaling, cropping and normalization, to adapt to the subsequent feature extraction model; S2. Edge detection, use Canny edge detection to extract image edge information. The specific calculation formula is: Gaussian smoothing: Where σ is the standard deviation of the Gaussian kernel, and x and y are the coordinates of the position within the kernel; Gradient calculation: Among them, G x and G y are the gradients of the image in the x and y directions respectively; S3. Texture feature extraction, using gray level co-occurrence matrix to extract texture features, and the calculation formula is: Where I(p, q) is the pixel value at position (p, q) in the image, i and j are grayscale values, θ is the direction, d is the distance, Δx and Δy are the offsets calculated based on the direction and distance; S4. Color feature extraction, using color histogram, color moment or color aggregation vector to extract color information; S5. Deep learning model extraction, using pre-trained convolutional neural network models to extract high-level visual features; Auditory feature extraction: S1. Audio preprocessing, including noise reduction, segmentation, and normalization, to prepare the audio signal; S2. Spectral feature extraction, using short-time Fourier transform to extract the spectral features of the audio signal, and the formula is: Where x[n] is the discrete time signal, ω[n] is the discrete form of the window function, m is the discrete time index, k is the discrete frequency index, and N is the number of DFT points; S3. Time domain feature extraction, including zero crossing rate, energy and amplitude; S4. Deep learning model extraction uses a pre-trained recurrent neural network to extract high-level features of the audio signal, and the calculation formula is: h t =f(Ux t +Wh t-1 +b) y t =g(Vh t +c) Among them, h t is the hidden state at time step t, y t is the output at time step t, x t is the input at time step t, U is the weight matrix from input to hidden state, W is the weight matrix from previous hidden state to current hidden state, V is the weight matrix from hidden state to output, b and c are bias terms, f is the activation function, usually nonlinear activation function is used in hidden layer, g is the activation function of output layer, depending on the task, it may be linear function, softmax function.

4. The visual and auditory cross-modal object material retrieval system according to claim 1, characterized in that: The feature fusion comprises the following steps: S1. Feature extraction, extracting features from visual and auditory data respectively; S2. Feature alignment, ensuring that visual and auditory features are aligned in time or space; S3. Feature fusion strategy selection, early fusion: directly concatenate features of different modalities; late fusion: independently process and make decisions on the features of each modality, and then combine the results; intermediate layer fusion: feature fusion is performed in the intermediate layers of the deep learning model, which usually involves designing a shared network structure in which features of different modalities are combined in an intermediate layer; S4. Fusion model design: design the fusion model according to the selected fusion strategy; S5. Train the fusion model and use the labeled data to train the fusion model.

5. The visual and auditory cross-modal object material retrieval system according to claim 1, characterized in that: The retrieval model includes the following steps: S1. Model architecture design: design a deep learning model that can process the fused features and perform material classification or regression; S2. Loss function selection, select the appropriate loss function according to the task type; S3. Optimization algorithm selection: select an optimization algorithm to minimize the loss function. Commonly used optimization algorithms include stochastic gradient descent (SGD), Adam, and RMSprop. S4. Model training, using labeled data to train the model; S5. Model evaluation and tuning: Use the validation set to evaluate model performance and adjust the model structure or parameters based on the evaluation results.

6. The visual and auditory cross-modal object material retrieval system according to claim 1, characterized in that: The retrieval and feedback comprises the following steps: S1. User input processing: Users upload or input visual and auditory data through the interface, and pre-process the user input data, such as image scaling and audio noise reduction, to match the model input requirements; S2. Feature extraction and fusion: extract features from the processed visual and auditory data respectively, and fuse the extracted features to form a cross-modal feature representation; S3. Retrieval model processing, inputting the fused features into the trained retrieval model, and the model outputs the material retrieval results, which are usually the classification labels of the materials or the continuous values ​​of the material attributes; S4. Result display: display the material results output by the model to the user in a user-friendly manner; S5. User feedback collection: Provide a feedback mechanism so that users can evaluate the accuracy of search results. The feedback can be a simple rating or a more detailed description. S6. Model optimization: adjust model parameters or structure based on user feedback to improve retrieval accuracy and user satisfaction.

7. The visual and auditory cross-modal object material retrieval system according to claim 1, characterized in that: The experiment and evaluation include the following steps: S1. Experimental design, data set preparation, ensuring there is a test set containing labeled data to evaluate model performance and set experimental parameters; S2. Model evaluation: Use appropriate performance indicators to evaluate the model. You can use accuracy, recall, F1 score, precision, and area under the ROC curve. Construct a confusion matrix to analyze the classification performance of the model, especially for multi-class classification problems. S3. The performance index calculation formula includes: Accuracy: Recall: Accuracy: F1 score: S4. Statistical significance test, use statistical tests to determine significant differences in model performance; S5. Error analysis: analyzing cases where the model makes incorrect predictions to understand the model’s weaknesses and potential directions for improvement; S6. Model optimization: adjust the model structure, parameters or training strategy based on the evaluation results.

8. A method for using a visual and auditory cross-modal object material retrieval system, characterized by: The following steps are involved: S1. Data collection, collecting visual images of objects of various materials and corresponding audio data; S2. Feature extraction: extracting features from visual and auditory data respectively. Visual features include but are not limited to color, texture, and shape. Auditory features include but are not limited to spectrum features and time domain features. S3. Feature fusion: design a fusion algorithm to effectively fuse visual and auditory features to form a cross-modal feature representation; S4. Retrieval model: build a deep learning-based retrieval model that can learn the mapping relationship between cross-modal features and object materials; S5. Retrieval and feedback: implement a retrieval interface where users input or upload visual and auditory data, the system outputs material retrieval results, and optimizes the model based on user feedback; S6. Experiments and evaluation, use indicators such as accuracy, recall, F1 score to evaluate model performance. At the same time, conduct user research to evaluate the actual usage effect and user experience of the system.

Images