A lithology judgment method and device based on audio-visual fusion recognition

Abstract

The application discloses a lithology judgment method and device based on audio-visual fusion recognition, and belongs to the technical field of artificial intelligence. Through synchronous acquisition and unified time sequence synchronization of three modes of images, spectra and voiceprints, millisecond-level collaborative acquisition of multi-modal data is realized. Then, space alignment, feature extraction and deep fusion at the feature level and the decision level are performed, and a multi-modal Transformer network is constructed for lithology classification. In combination with a semi-supervised training strategy and edge quantization deployment, real-time and efficient recognition is realized. The application effectively solves the problems of low recognition accuracy, sensitivity to complex environments and lack of multi-source information fusion decision mechanism caused by limited information dimension of a single mode, significantly improves the precision and robustness of lithology recognition, and realizes on-site real-time decision and continuous self-adaptive optimization of the model.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to a method and apparatus for lithology determination based on audiovisual fusion recognition. Background Technology

[0002] With the rapid development of infrastructure construction, resource exploration and development, and geological engineering disaster prevention and control, lithology identification, as a fundamental and key technology in the field of geological science, directly affects the efficiency of mineral resource exploration, the safety factor of geotechnical engineering design, and the reliability of disaster early warning through its accuracy and efficiency. Lithology identification, through comprehensive analysis of the material composition, structure, and physical and mechanical characteristics of rock samples, provides crucial information for studying the distribution patterns of underground rock strata, assessing mineral resource reserves, evaluating the stability of tunnel surrounding rock, and formulating slope protection plans, playing an irreplaceable and vital role in national economic and social development.

[0003] Lithology identification methods mainly fall into three categories: traditional visual identification, chemical elemental analysis, and physical field detection. Traditional visual identification relies on the experience of geological professionals, judging lithology by observing the rock's color, texture, grain size, and fracture morphology. Chemical elemental analysis obtains the rock's chemical composition through methods such as spectral analysis and X-ray fluorescence testing, thereby establishing a mapping relationship between element content and lithology. Physical field detection utilizes the characteristics of seismic wave propagation, the acoustic response characteristics of rocks, or differences in electromagnetic induction response to infer lithology.

[0004] Existing lithology identification methods generally suffer from the following significant drawbacks: First, traditional identification methods require highly skilled operators, and experienced geological professionals are scarce and require long training periods, resulting in low efficiency and high labor costs in lithology identification. Second, chemical elemental analysis and physical field detection methods require lengthy processes such as sample collection, laboratory testing, and data processing, with identification cycles ranging from several days to several weeks, making it difficult to meet the needs of real-time on-site decision-making. Third, single-modal identification technologies can only acquire a very limited number of information dimensions. When the lithologies to be identified are similar or when there are complex conditions such as weathering or water content on the rock surface, the accuracy of identification methods based on a single image, single spectrum, or single acoustic signature drops significantly, and misjudgments may even occur. Fourth, existing technologies lack a fusion decision-making mechanism for multi-source heterogeneous data. Each identification modality is independent and has high information redundancy, making it impossible to achieve complementary advantages, resulting in overall identification accuracy and robustness that are difficult to meet the high-precision application requirements under complex geological conditions. The aforementioned technical problems severely restrict the widespread application of lithology identification technology in rapid on-site exploration, intelligent geological surveys, and unmanned exploration operations, and urgently need to be systematically solved through deep integration of multimodal information fusion and artificial intelligence technology. Summary of the Invention

[0005] To achieve the above objectives, this application provides the following technical solution: The purpose of this invention is to provide a lithology identification method and device based on audiovisual fusion recognition, so as to effectively solve the technical problems of low identification accuracy, poor real-time performance, sensitivity to complex environments, and lack of multi-source information fusion decision-making mechanism in existing lithology identification technologies.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A lithology determination method based on audiovisual fusion recognition includes the following steps: S1. Synchronous acquisition steps: Multimodal data is acquired synchronously at the millisecond level through image acquisition, spectral acquisition, acoustic signature acquisition, and unified time-series synchronization. S2, Multi-source data fusion steps, including spatial alignment, feature extraction, feature-level fusion and decision-level fusion, to achieve deep fusion of image, spectral and voiceprint features; S3, the artificial intelligence identification step, uses a multimodal Transformer network for lithology classification and combines a semi-supervised training strategy and edge quantization deployment to achieve real-time and efficient identification.

[0007] Preferably, step S1 specifically includes the following steps: S11. Image acquisition: An industrial camera is set up directly above the rock surface to capture at least three frames of images before and after the moment of hammering, in order to obtain information on rock surface texture, chipping, and debris distribution. S12. Spectral acquisition: A visible-near-infrared spectrometer is set obliquely on the rock surface to simultaneously acquire the rock surface reflectance spectrum at the moment of hammering, covering the preset waveband. S13. Acoustic print acquisition: A wideband microphone and a three-axis accelerometer are set on the rock surface to simultaneously record the sound pressure waveform and acceleration waveform at the moment of hammering. S14. Timing synchronization: Based on the trigger pulse output by the piezoelectric sensor built into the hammer, image, spectrum, and acoustic data are stamped with a unified timestamp to achieve millisecond-level synchronization.

[0008] Preferably, in step S11, the industrial camera uses a global shutter CMOS image sensor with a pixel count not less than a preset value, works with a fixed-focus lens at a preset working distance, and covers the rock surface to be measured area with a corresponding field of view, and the pixel resolution meets the requirements for identifying fine textures on the rock surface. The camera is equipped with a ring-shaped LED light source, which consists of a high color rendering index LED array. It features independent dimming in multiple quadrants and uses preset frequency PWM modulation to avoid flickering with image exposure. A polarizing optical element is added in front of the light source to reduce specular reflection interference.

[0009] Preferably, in step S12, the visible-near-infrared spectrometer adopts a MEMS scanning grating structure, has a preset resolution and a preset signal-to-noise ratio, a field-of-view collimating lens is installed at the front end of the fiber optic probe, the working distance corresponds to the measurement spot diameter and is coaxially arranged with the image center area, and a laser pointer is placed next to the probe for on-site alignment.

[0010] Preferably, in step S13, the wideband microphone is a free-field capacitor microphone with preset sensitivity and preset dynamic range. The triaxial accelerometer is of the IEPE type, with preset sensitivity and preset bandwidth. Both are acquired by a synchronous acquisition card at a sampling rate no less than the preset rate to ensure that data is not lost in the event of a sudden trigger.

[0011] Preferably, in step S14, the trigger pulse is converted into a voltage signal by a charge amplifier and then sent to the acquisition card. It is then distributed to each acquisition module via TTL. The total delay of the trigger chain is less than the preset microsecond value, which is much smaller than the image exposure time and spectral integration time, and can be ignored. The data acquisition card uses a temperature-compensated crystal oscillator as its clock source. The frequency stability meets the preset specifications. It synchronizes with the reference clock source once during the first power-on each day, and the accumulated drift meets the real-time requirements.

[0012] Preferably, step S2 specifically includes the following steps: S21. Spatial alignment: Establish the mapping relationship between image pixel coordinates and spectral field of view spatial coordinates based on checkerboard joint calibration. S22. Feature extraction: The image is processed by a deep convolutional neural network to extract the first-dimensional deep features, the spectrum is processed by a one-dimensional convolutional neural network to extract the second-dimensional features, and the voiceprint is processed by a time-frequency transformation combined with a two-dimensional convolutional neural network to extract the third-dimensional features. S23, Feature-level fusion, which concatenates three-modal features in the channel dimension to obtain multimodal fused features; S24. Decision-level fusion: The soft probabilities output by the three modes are summed by dynamic weights to obtain the final lithology probability distribution.

[0013] Preferably, in step S22, the image feature extraction adopts a deep residual network structure with a preset output dimension to extract visual features such as rock surface texture, color distribution, and collapse morphology. Spectral feature extraction employs a one-dimensional convolutional neural network to extract spectral features such as the position and depth of mineral characteristic absorption peaks. Voiceprint feature extraction employs filter-based time-frequency transformation to extract acoustic features, followed by extraction of voiceprint time-frequency spectrum features via a two-dimensional convolutional neural network.

[0014] Preferably, in step S23, the feature-level fusion concatenates the first-dimensional feature, the second-dimensional feature, and the third-dimensional feature in the channel dimension to form a fused feature vector. The fused feature dimension is the sum of the feature dimensions of each modality, providing rich cross-modal information for the subsequent multimodal Transformer network.

[0015] Preferably, in step S24, the decision-level fusion adopts a dynamic weighting mechanism. The weights are dynamically adjusted according to the quality assessment results of each modal feature. When the quality of a certain modal feature deteriorates due to environmental interference, its weight is automatically reduced, thereby achieving robust multimodal decision fusion.

[0016] Preferably, step S3 specifically includes the following steps: S31. Construct a multimodal Transformer network, with fused features as input and lithology category and confidence level as output; S32. Joint training with cross-entropy loss and center loss is adopted, and unlabeled field samples are expanded through a semi-supervised strategy. S33. After quantization, the model is deployed on the edge computing unit, and the single recognition latency meets the real-time requirements. S34. When the confidence level is lower than the preset threshold, manual review is triggered, and the review results are updated with the model parameters through incremental learning to achieve closed-loop optimization.

[0017] Preferably, in step S31, the multimodal Transformer network adopts a multi-layer encoder structure, each layer containing a multi-head self-attention mechanism and a feedforward network, the number of self-attention heads meets the requirements of multimodal feature interaction, and the hidden layer dimension meets the feature representation capability; The network introduces modality location encoding and modality type embedding to represent the spatial positional relationship and modality identity information of each modality feature, respectively, thereby enhancing the network's ability to model multimodal temporal relationships.

[0018] Preferably, in step S32, the joint training strategy simultaneously optimizes the classification loss and the metric loss. The classification loss ensures that the network learns the correct lithology category mapping, and the metric loss ensures that features of the same type of lithology are compactly distributed in the embedding space and features of different types are far apart. The semi-supervised strategy utilizes a large number of unlabeled field samples for pseudo-label training, and ensures the quality of pseudo-labels through a confidence screening mechanism, thereby enabling the model to adapt to complex field environments.

[0019] Preferably, in step S33, the model quantization adopts a preset precision quantization method. After quantization, the model size is greatly reduced and the inference speed is significantly improved. It is deployed on an embedded AI computing platform with GPU resources that meet the computing power requirements. The power consumption is controlled within a preset range, and the overall battery life meets the requirements of long-term on-site operation.

[0020] Preferably, in step S34, the incremental learning method, while retaining the original knowledge, incorporates the results of manual verification into the model through weight importance assessment and regularization constraints, enabling the model to continuously learn new lithological characteristics, while avoiding catastrophic forgetting of learned knowledge, and ensuring the model's long-term adaptive capability under complex geological conditions.

[0021] According to a second aspect of the present invention, the present invention claims protection for a lithology determination device based on audiovisual fusion recognition, for performing the aforementioned lithology determination method based on audiovisual fusion recognition, the device comprising: a force hammer module, having a built-in piezoelectric force sensor, and outputting trigger pulses and real-time force value signals; The image module includes an industrial camera, a ring light source, and a trigger interface; the spectral module includes a miniature spectrometer, a fiber optic collimator, and a trigger interface. The voiceprint module includes a wideband microphone, a three-axis accelerometer, and a synchronous acquisition card; The edge computing module has a built-in quantized multimodal Transformer model; the display and interaction module displays the recognition results and explanatory diagrams in real time on the touch screen. The power supply and housing are integrated into a portable case structure.

[0022] Compared with existing technologies, the advantages of this invention are as follows: This invention achieves millisecond-level synchronous acquisition of rock surface visual morphology, mineral spectra, and hammer-impact acoustic responses through a three-modal synchronous acquisition architecture of image, spectrum, and acoustic signature. This effectively solves the technical problem of decreased recognition accuracy under similar or complex lithological conditions due to the limited dimensions of single-modal information. First, the three-modal complementary mechanism synchronously acquires lithological feature information from visual, spectral, and acoustic dimensions. Each modal feature mutually verifies and complements the others, significantly improving recognition accuracy. Second, the hammer-triggered mechanism ensures consistent acquisition conditions each time, and the unified temporal reference eliminates the time misalignment problem of multimodal data, reducing human error. Third, the multimodal Transformer network maps fused features to a high-dimensional embedding space, and through a contrastive learning mechanism, compactly clusters similar lithological features and clearly separates dissimilar features, effectively improving the model's discriminative ability. Finally, the real-time edge inference capability allows the system to make immediate decisions on-site without waiting for background analysis, significantly improving operational efficiency.

[0023] Furthermore, by employing a deep residual network for image feature extraction, this network effectively alleviates the problem of deep network training degradation through the residual connection mechanism. It can extract the texture features, fracture morphology features, and debris distribution features of the deep rock surface. These features contain key visual information for lithological discrimination and provide rich image dimensional representations for subsequent fusion decisions.

[0024] Furthermore, by employing a one-dimensional convolutional neural network for spectral feature extraction, this network can automatically learn mineral characteristics such as the position, depth, and width of spectral absorption peaks, achieving accurate characterization of different mineral combinations and effectively solving the problem of traditional spectral analysis relying on manual feature engineering.

[0025] Furthermore, by employing filter-based time-frequency transformation combined with a two-dimensional convolutional neural network for acoustic feature extraction, this method converts sound pressure waveforms and acceleration waveforms into time-frequency spectrum representations. The two-dimensional convolutional network then extracts the frequency distribution features, attenuation characteristics, and resonance peak information of the hammer impact acoustics. These acoustic features are closely related to physical and mechanical parameters such as lithological hardness, density, and elastic modulus, providing unique acoustic dimensional information for lithological identification.

[0026] Furthermore, by cascading image features, spectral features, and voiceprint features in the channel dimension to form fused features, this feature-level fusion method preserves the integrity of the original information of each modality feature, avoids the loss of modality-specific information that may be lost in early fusion, and realizes dynamic interaction and information supplementation of cross-modal features through the self-attention mechanism of Transformer.

[0027] Furthermore, by employing a semi-supervised learning strategy to expand the unlabeled field samples, this strategy utilizes the model's prediction confidence to filter high-quality pseudo-labeled samples, transforming unlabeled data into valuable training resources. This significantly expands the scale of training data, enabling the model to learn the distribution of lithological characteristics in complex field environments, effectively improving the model's generalization ability and adaptability to non-laboratory conditions.

[0028] Furthermore, by employing an incremental learning method to achieve closed-loop optimization of the model, this method identifies key parameters based on their weight importance and applies regularization constraints after the feedback of the manual review results. This enables the model to maintain its ability to identify existing lithological categories while incrementally learning new knowledge, thus achieving long-term adaptive learning of the system.

[0029] Furthermore, by quantizing and deploying the model on an edge computing unit, which has embedded AI acceleration capabilities, inference latency meets the real-time requirements of the field, and power consumption control meets the battery life requirements of portable devices, the system can independently complete lithology identification tasks without a network connection, making it suitable for various complex field environments such as tunnel faces, open-pit mines, and underground drilling. Attached Figure Description

[0030] Figure 1 This is a flowchart illustrating the process of a lithology determination method based on audiovisual fusion recognition, which is claimed in an embodiment of the present invention. Figure 2This is a second flowchart of a lithology determination method based on audiovisual fusion recognition, which is claimed in an embodiment of the present invention. Figure 3 The third flowchart of a lithology determination method based on audiovisual fusion recognition, which is claimed in an embodiment of the present invention, is shown below. Figure 4 The fourth flowchart of a lithology determination method based on audiovisual fusion recognition, which is claimed in an embodiment of the present invention, is shown in the figure below. Detailed Implementation

[0031] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0032] The terms "first," "second," and "third" in this application are for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first," "second," or "third" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified. All directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of this application are only used to explain the relative positional relationships and movements between components in a specific orientation (as shown in the figures). If the specific orientation changes, the directional indications also change accordingly. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.

[0033] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0034] According to a first embodiment of the present invention, the present invention claims protection for a lithology determination method based on audiovisual fusion recognition, referring to... Figure 1 This includes the following steps: S1. Synchronous acquisition steps: Multimodal data is acquired synchronously at the millisecond level through image acquisition, spectral acquisition, acoustic signature acquisition, and unified time-series synchronization. S2, Multi-source data fusion steps, including spatial alignment, feature extraction, feature-level fusion and decision-level fusion, to achieve deep fusion of image, spectral and voiceprint features; S3, the artificial intelligence identification step, uses a multimodal Transformer network for lithology classification and combines a semi-supervised training strategy and edge quantization deployment to achieve real-time and efficient identification.

[0035] Preferred, refer to Figure 2 Step S1 specifically includes the following steps: S11. Image acquisition: An industrial camera is set up directly above the rock surface to capture at least three frames of images before and after the moment of hammering, in order to obtain information on rock surface texture, chipping, and debris distribution. S12. Spectral acquisition: A visible-near-infrared spectrometer is set obliquely on the rock surface to simultaneously acquire the rock surface reflectance spectrum at the moment of hammering, covering the preset waveband. S13. Acoustic print acquisition: A wideband microphone and a three-axis accelerometer are set on the rock surface to simultaneously record the sound pressure waveform and acceleration waveform at the moment of hammering. S14. Timing synchronization: Based on the trigger pulse output by the piezoelectric sensor built into the hammer, image, spectrum, and acoustic data are stamped with a unified timestamp to achieve millisecond-level synchronization.

[0036] Preferably, in step S11, the industrial camera uses a global shutter CMOS image sensor with a pixel count not less than a preset value, works with a fixed-focus lens at a preset working distance, and covers the rock surface to be measured area with a corresponding field of view, and the pixel resolution meets the requirements for identifying fine textures on the rock surface. The camera is equipped with a ring-shaped LED light source, which consists of a high color rendering index LED array. It features independent dimming in multiple quadrants and uses preset frequency PWM modulation to avoid flickering with image exposure. A polarizing optical element is added in front of the light source to reduce specular reflection interference.

[0037] Preferably, in step S12, the visible-near-infrared spectrometer adopts a MEMS scanning grating structure, has a preset resolution and a preset signal-to-noise ratio, a field-of-view collimating lens is installed at the front end of the fiber optic probe, the working distance corresponds to the measurement spot diameter and is coaxially arranged with the image center area, and a laser pointer is placed next to the probe for on-site alignment.

[0038] Preferably, in step S13, the wideband microphone is a free-field capacitor microphone with preset sensitivity and preset dynamic range. The triaxial accelerometer is of the IEPE type, with preset sensitivity and preset bandwidth. Both are acquired by a synchronous acquisition card at a sampling rate no less than the preset rate to ensure that data is not lost in the event of a sudden trigger.

[0039] Preferably, in step S14, the trigger pulse is converted into a voltage signal by a charge amplifier and then sent to the acquisition card. It is then distributed to each acquisition module via TTL. The total delay of the trigger chain is less than the preset microsecond value, which is much smaller than the image exposure time and spectral integration time, and can be ignored. The data acquisition card uses a temperature-compensated crystal oscillator as its clock source. The frequency stability meets the preset specifications. It synchronizes with the reference clock source once during the first power-on each day, and the accumulated drift meets the real-time requirements.

[0040] Preferred, refer to Figure 3 Step S2 specifically includes the following steps: S21. Spatial alignment: Establish the mapping relationship between image pixel coordinates and spectral field of view spatial coordinates based on checkerboard joint calibration. S22. Feature extraction: The image is processed by a deep convolutional neural network to extract the first-dimensional deep features, the spectrum is processed by a one-dimensional convolutional neural network to extract the second-dimensional features, and the voiceprint is processed by a time-frequency transformation combined with a two-dimensional convolutional neural network to extract the third-dimensional features. S23, Feature-level fusion, which concatenates three-modal features in the channel dimension to obtain multimodal fused features; S24. Decision-level fusion: The soft probabilities output by the three modes are summed by dynamic weights to obtain the final lithology probability distribution.

[0041] Preferably, in step S22, the image feature extraction adopts a deep residual network structure with a preset output dimension to extract visual features such as rock surface texture, color distribution, and collapse morphology. Spectral feature extraction employs a one-dimensional convolutional neural network to extract spectral features such as the position and depth of mineral characteristic absorption peaks. Voiceprint feature extraction employs filter-based time-frequency transformation to extract acoustic features, followed by extraction of voiceprint time-frequency spectrum features via a two-dimensional convolutional neural network.

[0042] Preferably, in step S23, the feature-level fusion concatenates the first-dimensional feature, the second-dimensional feature, and the third-dimensional feature in the channel dimension to form a fused feature vector. The fused feature dimension is the sum of the feature dimensions of each modality, providing rich cross-modal information for the subsequent multimodal Transformer network.

[0043] Preferably, in step S24, the decision-level fusion adopts a dynamic weighting mechanism. The weights are dynamically adjusted according to the quality assessment results of each modal feature. When the quality of a certain modal feature deteriorates due to environmental interference, its weight is automatically reduced, thereby achieving robust multimodal decision fusion.

[0044] Preferred, refer to Figure 4 Step S3 specifically includes the following steps: S31. Construct a multimodal Transformer network, with fused features as input and lithology category and confidence level as output; S32. Joint training with cross-entropy loss and center loss is adopted, and unlabeled field samples are augmented through a semi-supervised strategy. S33. After quantization, the model is deployed on the edge computing unit, and the single recognition latency meets the real-time requirements. S34. When the confidence level is lower than the preset threshold, manual review is triggered, and the review results are updated with the model parameters through incremental learning to achieve closed-loop optimization.

[0045] Preferably, in step S31, the multimodal Transformer network adopts a multi-layer encoder structure, each layer containing a multi-head self-attention mechanism and a feedforward network, the number of self-attention heads meets the requirements of multimodal feature interaction, and the hidden layer dimension meets the feature representation capability; The network introduces modality location encoding and modality type embedding to represent the spatial positional relationship and modality identity information of each modality feature, respectively, thereby enhancing the network's ability to model multimodal temporal relationships.

[0046] Preferably, in step S32, the joint training strategy simultaneously optimizes the classification loss and the metric loss. The classification loss ensures that the network learns the correct lithology category mapping, and the metric loss ensures that features of the same type of lithology are compactly distributed in the embedding space and features of different types are far apart. The semi-supervised strategy utilizes a large number of unlabeled field samples for pseudo-label training, and ensures the quality of pseudo-labels through a confidence screening mechanism, thereby enabling the model to adapt to complex field environments.

[0047] Preferably, in step S33, the model quantization adopts a preset precision quantization method. After quantization, the model size is greatly reduced and the inference speed is significantly improved. It is deployed on an embedded AI computing platform with GPU resources that meet the computing power requirements. The power consumption is controlled within a preset range, and the overall battery life meets the requirements of long-term on-site operation.

[0048] Preferably, in step S34, the incremental learning method, while retaining the original knowledge, incorporates the results of manual verification into the model through weight importance assessment and regularization constraints, enabling the model to continuously learn new lithological characteristics, while avoiding catastrophic forgetting of learned knowledge, and ensuring the model's long-term adaptive capability under complex geological conditions.

[0049] In this embodiment, the system uses an integrated portable explosion-proof box as a carrier, integrating an image acquisition module, a spectrum acquisition module, a voiceprint acquisition module, an edge computing module, and a display interaction module. It achieves millisecond-level synchronous acquisition and real-time intelligent recognition of multimodal data through a force hammer triggering mechanism. The mechanical structure, hardware configuration, data acquisition process, feature extraction algorithm, fusion decision mechanism, and artificial intelligence recognition process of each module will be described in detail below in the order of method steps.

[0050] Regarding the synchronous acquisition step, this invention employs a multi-sensor collaborative architecture. The image acquisition sub-step is implemented as follows: an industrial camera is positioned directly above the rock surface. This camera uses a global shutter CMOS image sensor with a 1.1-inch target area, 20 million effective pixels, and a pixel size of 3.45 micrometers. The camera's frame rate can be set to a fixed value within the range of 120 to 500 frames per second. The camera works with a 25mm fixed-focus lens at a working distance of 300mm. At this working distance, the camera's field of view is 120mm x 80mm, effectively covering the area to be measured on the rock surface. The pixel resolution is approximately 0.05mm per pixel, meeting the accuracy requirements for identifying fine textures on the rock surface. The camera connects to the edge computing module via a gigabit Ethernet interface, and the image data is transmitted to the computing module in raw RAW format for subsequent processing. The camera is also equipped with a ring-shaped LED light source, consisting of an array of 64 high color rendering index (CRI) LEDs (6500K). The light source is divided into four quadrants with independently adjustable brightness and employs a 20kHz PWM modulation method to avoid flicker interference with image exposure. A polarizing optical element is installed in front of the light source to reduce high-light interference caused by specular reflection from the rock surface. Within the time window before and after the hammer impact, the camera acquires at least three frames in a preset continuous capture mode, thus comprehensively recording information on changes in rock surface texture, fracture morphology, and debris distribution. The camera's operation is controlled by the edge computing module via a TTL trigger signal. This trigger signal is synchronized with the output pulse of the piezoelectric sensor built into the hammer, ensuring precise timing of image acquisition.

[0051] The spectral acquisition sub-step is implemented as follows: A visible-near-infrared spectrometer is obliquely positioned on the rock surface. This spectrometer employs a MEMS scanning grating structure with a slit width of 25 micrometers and a numerical aperture of 0.22. It achieves a spectral resolution of 3 nanometers in the 900-1700 nanometer wavelength range, with a signal-to-noise ratio greater than 1000:1. A collimating lens with a field of view of 2 degrees is installed at the front end of the spectrometer's fiber optic probe. The working distance is set to 150 millimeters, corresponding to a measurement spot diameter of approximately 5 millimeters. This spot is coaxially positioned with the center of the camera's field of view to ensure the spatial correspondence between image features and spectral features. An 850-nanometer wavelength laser pointer is placed next to the probe for quick alignment with the measurement point on the rock surface during on-site operation. The spectrometer connects to the edge computing module via a USB 3.0 interface. The acquired spectral data is stored in 32-bit floating-point format, including wavelength information and corresponding reflection intensity information. The spectrometer's integration time can be set to a fixed value within the range of 1 to 10 milliseconds, and acquisition is initiated synchronously with the hammer trigger signal.

[0052] The acoustic signature acquisition sub-step is implemented as follows: A wideband microphone and a triaxial accelerometer are placed near the rock surface to synchronously record the sound pressure waveform and acceleration wave generated by the hammer impact. The wideband microphone uses a quarter-inch free-field condenser microphone with a sensitivity of 50 mV / Pa, a dynamic range greater than 120 dB, and a frequency response range covering 20 Hz to 20 kHz, capable of completely capturing the spectral characteristics of the hammer impact sound signal. The triaxial accelerometer uses an IEPE integrated circuit piezoelectric sensor with a sensitivity of 10 mV / gravity acceleration and a bandwidth of 1 Hz to 15 kHz, capable of measuring vibration acceleration in three orthogonal directions. The microphone and accelerometer are connected to a synchronous acquisition card via a BNC coaxial cable. This acquisition card has 8 analog input channels, an analog-to-digital conversion accuracy of 24 bits, a sampling rate that can be set to 256 kilosamples per second, and a built-in 128 kilosample FIFO buffer to ensure no data loss in the event of sudden triggering. The acquisition of voiceprint data is also controlled by a hammer trigger signal. Once the trigger signal arrives, the acquisition card immediately starts acquisition and stores the data in a buffer for later reading.

[0053] The timing synchronization sub-step is implemented as follows: the timing reference of the entire system is determined by the trigger pulse output from the piezoelectric force sensor built into the hammer. The piezoelectric force sensor is a Kistler 9211B model with a range of 0 to 5 kN and a natural frequency greater than 40 kHz. When the hammer strikes, the piezoelectric sensor outputs a charge signal, which is sent to a charge amplifier via a coaxial cable and converted into a voltage signal of 0 to 10 volts. This voltage signal serves as the trigger pulse for system acquisition. After amplification, the trigger pulse is sent to the synchronization acquisition card, and then transmitted via the TTL distribution circuit of the acquisition card to the trigger input terminals of the image acquisition module, the spectrum acquisition module, and the acoustic signature acquisition module, respectively, to achieve synchronous startup of each module. The total delay of the trigger chain is less than 80 microseconds according to actual measurements. This delay is much smaller than the image exposure time and the spectrum integration time, and can be ignored, having no significant impact on the synchronization accuracy of multimodal data. The synchronization acquisition card uses a temperature-compensated crystal oscillator as the clock source, with a frequency stability of ±0.1 parts per million. It is synchronized with GPS upon first power-on each day. The PPS reference clock source is synchronized once, and the cumulative drift over 24 hours is less than 8.6 milliseconds, meeting the real-time requirements for on-site lithology identification. The timestamp accuracy is at the microsecond level, providing a precise time reference for subsequent spatial alignment and feature fusion of multi-source data.

[0054] In terms of the multi-source data fusion step, this invention adopts a two-level fusion architecture to achieve deep fusion of image, spectral, and acoustic features. The spatial alignment sub-step is implemented by establishing a mapping relationship between image pixel coordinates and spectral field-of-view spatial coordinates based on a checkerboard joint calibration method. The specific operation process is as follows: a checkerboard calibration board with a known-sized checkerboard pattern is placed on a rock surface. First, an industrial camera is used to capture images of the calibration board from multiple angles. The corner pixel coordinates of the checkerboard corners in the image are extracted using a corner detection algorithm. Simultaneously, a spectrometer is used to collect spectra from the same area of ​​the calibration board to obtain the spatial coverage of the spectral probe. Then, a pinhole camera model is used to establish the transformation relationship between image pixel coordinates and three-dimensional spatial coordinates, and the collinearity equation is used to solve for the camera intrinsic parameter matrix and distortion coefficients. Finally, a laser rangefinder is used to measure the distance and angle relationship between the camera lens plane and the spectral probe plane, establishing the extrinsic parameter rotation matrix and translation vector between the two sensors, thereby achieving accurate mapping between image pixel coordinates and spectral field-of-view spatial coordinates. The spatial alignment error is tested to be less than 1 mm, meeting the spatial correspondence accuracy requirements of multimodal feature fusion.

[0055] The feature extraction sub-step is implemented as follows: For the three modalities of data, corresponding deep neural networks are designed for feature extraction. For image data, a deep residual network structure is used for feature extraction. The main structure of this network is ResNet50, containing 50 convolutional layers and residual connections. The network input is a rock surface image captured by a camera. The image is preprocessed, resized to 224 pixels by 224 pixels, and normalized. The network extracts features through multiple convolutional layers with 3x3 and 1x1 kernels. After each residual block, the input and output features are added element-wise, effectively alleviating the gradient vanishing problem during deep network training. Finally, the fully connected layer of the network outputs a 2048-dimensional deep feature vector, which contains visual information such as rock surface texture features, color distribution features, fracture morphology features, and debris distribution features. For spectral data... For the data, a one-dimensional convolutional neural network was used for feature extraction. The input to this network was a wavelength-reflectance intensity data sequence collected by a spectrometer, with a length of 256 sampling points. The network contained multiple one-dimensional convolutional layers with kernel sizes of 3, 5, and 7, and the number of kernels increased from 32 to 128. Batch normalization and ReLU activation functions were alternated between the convolutional layers. Finally, a global average pooling layer was used to compress the feature map into a 128-dimensional feature vector, which contained spectral feature information such as the position, depth, and width of mineral characteristic absorption peaks. For the voiceprint data, a filter-based time-frequency transform combined with a two-dimensional convolutional neural network was used for feature extraction. First, the sound pressure waveform collected by the microphone and the acceleration waveform collected by the triaxial accelerometer were preprocessed, including removing the DC component, pre-emphasis filtering, and frame windowing operations. The frame length was set to 1024 sampling points, and the frame shift was set to 512 sampling points. The preprocessed signal is then decomposed through a set of bandpass filters. The filter set consists of 32 bandpass filters with center frequencies ranging from 100 Hz to 10 kHz on a logarithmic scale. Each filter outputs the energy value of its corresponding frequency band, forming a time-frequency feature matrix. Finally, the time-frequency feature matrix is ​​input into a two-dimensional convolutional neural network. This network contains four 3x3 convolutional layers with kernel numbers of 32, 64, 128, and 256, respectively. Batch normalization and ReLU activation functions are applied between the convolutional layers, and downsampling is performed using 2x2 max-pooling layers. After global average pooling, a 256-dimensional feature vector is output, containing the frequency distribution characteristics, attenuation properties, and formant information of the hammer-impact acoustics. The feature extraction process for the three modalities is executed in parallel on the GPU of the edge computing module, with a total feature extraction time of approximately 15 milliseconds.

[0056] The feature-level fusion sub-step is implemented by sequentially concatenating the feature vectors extracted from the three modalities along the channel dimension to form a multimodal fusion feature vector. Specifically, the three feature vectors are first aligned in dimension, cropping or padding the image feature vector to a uniform length. Then, the image feature vector, spectral feature vector, and voiceprint feature vector are concatenated sequentially, resulting in a fusion feature vector with dimensions of 2048 + 128 + 256, equaling 2432 dimensions. Feature-level fusion preserves the integrity of the original information of each modality's features, avoiding the loss of modality-specific information that might occur during early fusion. The fusion feature vector serves as the input to the subsequent multimodal Transformer network, providing the network with rich cross-modal information.

[0057] The decision-level fusion sub-step is implemented as follows: as an optional fusion strategy, the soft probabilities output by the three modalities are weighted and summed according to dynamic weights to obtain the final lithology probability distribution. Specifically, before the three modal features enter the Transformer network, each modal feature first passes through an independent fully connected layer, outputting soft probability values ​​for various lithologies. Simultaneously, the system evaluates the quality of each modal feature, using metrics such as feature variance, feature norm, and signal-to-noise ratio. When the quality of a modal feature deteriorates due to environmental interference, its weight is automatically reduced. The dynamic weights are calculated using an attention mechanism. After concatenating the three modal features, three weight values ​​are output through a fully connected layer. These three weight values ​​are then normalized using Softmax and weighted and summed with the corresponding soft probability values ​​to obtain the final lithology probability distribution. This decision-level fusion strategy achieves robust multimodal decision fusion, improving the system's recognition reliability in complex environments.

[0058] In terms of the artificial intelligence recognition steps, this invention employs a multimodal Transformer network for lithology classification. The multimodal Transformer network adopts a multi-layer encoder structure, specifically implemented as follows: the network contains 12 encoder layers, each containing a multi-head self-attention mechanism and a feedforward network; the multi-head self-attention mechanism contains 8 attention heads, each with a hidden layer dimension of 768, and the attention heads calculate attention weights in parallel and concatenate the outputs. The attention mechanism enables the network to dynamically learn the correlation between features of different modalities; the feedforward network contains two fully connected layers with a hidden layer dimension of 3072, and the intermediate layers use the GELU activation function. The network also introduces layer normalization for feature standardization. To enhance the network's ability to model multimodal temporal relationships, modal location encoding and modal type embedding are introduced: modal location encoding sets learnable location embeddings for image patches, spectral segments, and acoustic frames to represent the spatial positional relationship of each modality feature; modal type embedding represents the identity information of each modality, distinguishing whether the feature originates from the image, spectrum, or acoustic signature; a 128-dimensional fully connected layer is added to the network output, followed by a Softmax layer to output the probability distribution of lithology categories; the input of the multimodal Transformer network is the fused feature vector, and the output is the lithology category and its corresponding confidence score. The lithology categories that the network can identify include no fewer than 20 common rock types such as granite, basalt, sandstone, limestone, slate, and marble.

[0059] The network training employs a joint training strategy using cross-entropy loss and center loss. Specifically, cross-entropy loss ensures the network learns the correct lithology category mapping, while center loss ensures that features of similar lithologies are compactly distributed in the embedding space, while features of dissimilar lithologies are kept far apart. The training process consists of two stages: the first stage uses laboratory-calibrated data for supervised training, with a dataset containing 648 rock samples, each with 30 taps, totaling 19,440 samples, trained for 100 epochs; the second stage uses unlabeled data collected in the field for semi-supervised training, with approximately 20,000 data points. The FixMatch strategy is used to select high-quality pseudo-labeled samples based on the model's prediction confidence for training, followed by another 50 epochs. The optimizer used is AdamW, with an initial learning rate of 3 x 10⁻⁴. Cosine annealing is used to decay the learning rate to 3 x 10⁻⁶, and weight decay is set to 1 x 10⁻². Regularization strategies include DropPath probability of 0.1, SpecAugment for voiceprint data enhancement, and GridMask for image data enhancement. The trained model is saved in ONNX format for subsequent quantization deployment.

[0060] The model quantization deployment adopted the TensorRT post-training quantization method. Specifically, a calibration set of 2000 samples was used to quantize and calibrate the model, achieving an INT8 precision. After quantization, the model size was reduced from 180 megabytes to 45 megabytes, a reduction of approximately 75%. The quantized model was deployed on the NVIDIA Jetson Xavier edge computing platform, which features an 8-core ARM64 CPU, a 512-core Volta GPU, 16GB of LPDDR4 memory, and a GPU computing power of 11 TFLOPS. During model inference, the ONNX model was first imported into the TensorRT engine for layer fusion and memory optimization, and then GPU-accelerated inference was used. The single-recognition latency was reduced from 180 milliseconds before quantization to 58 milliseconds, meeting the real-time requirements of the field. The Jetson Xavier's GPU utilization was approximately 65%, with power consumption less than 15 watts. The replaceable lithium battery pack within the integrated chassis has a capacity of 100 watt-hours, providing a battery life of more than 8 hours, meeting the needs of long-term field operations.

[0061] The specific implementation of the manual review and incremental learning mechanism is as follows: When the confidence level of the network output is lower than the preset threshold of 0.8, the system triggers the manual review process. The recognition results and SHAP interpretation diagrams are displayed in real time on the touch screen of the display and interaction module. The SHAP diagram indicates which features contribute the most to the current recognition result, such as the absorption peak of the spectrum at 1400 nm or the energy distribution of the acoustic signature in the 2 to 4 kHz frequency band. On-site operators judge the correctness of the recognition results based on experience and input the review results into the system through the touch screen. The review results update the model parameters through the incremental learning method. The incremental learning adopts the EWC algorithm, namely the Elastic Weight Consolidation algorithm. This algorithm, while retaining the original knowledge, identifies key parameters by evaluating the importance of weights and applies regularization constraints, integrating the manual review results into the model, enabling the model to continuously learn new lithological features while avoiding catastrophic forgetting of learned knowledge. Each incremental learning update uses 10 to 20 review samples, and the learning rate is set to one-tenth of the original learning rate. After several incremental learning iterations, the model's recognition ability is continuously improved.

[0062] The complete technical solution of this embodiment achieves real-time intelligent lithology assessment through the above steps. The operator holds a hammer and strikes the rock surface vertically. The trigger pulse output by the piezoelectric sensor built into the hammer synchronously triggers the three acquisition modules: image, spectrum, and acoustic signature. The system completes the entire process of data acquisition, feature extraction, and fusion recognition within 60 milliseconds, outputting the lithology category and confidence level, with an accuracy rate greater than 92%. The system supports three interchangeable hammerheads: steel, nylon, and rubber. The steel head is suitable for hard rock testing (Rockwell hardness 60), the nylon head is suitable for soft rock or testing to avoid surface damage (Shore D75), and the rubber head is suitable for testing water-bearing or fragile rock surfaces (Shore A60). The system also includes built-in temperature, humidity, and vibration compensation functions. An NTC temperature probe inside the housing monitors the ambient temperature in real time, automatically activating the cooling fan when the temperature exceeds 50 degrees Celsius. A humidity sensor detects ambient humidity, automatically activating a miniature heating film to prevent lens condensation when the relative humidity exceeds 80%. The accelerometer collects background vibration signals when idle; if the vibration RMS value is greater than 0.5 times the gravitational acceleration, the data set is marked for manual verification.

[0063] In terms of system hardware architecture, the integrated explosion-proof housing is made of IP67-rated magnesium-aluminum alloy, with dimensions of 420 mm x 330 mm x 180 mm and a weight of no more than 8 kg. A quick-release transparent quartz glass window is located on the top of the housing for optical acquisition by the image and spectral modules; explosion-proof glands are located on the sides for the insertion of hammer, microphone, and accelerometer cables. The internal space of the housing is divided into three layers: the upper layer houses the industrial camera, spectrometer, LED ring light source, and heat dissipation ducts, which guide heat to the heat dissipation holes on the sides of the housing; the middle layer houses the Jetson Xavier computing core, NVMe cache card, and synchronous acquisition card, with the computing core dissipating heat through thermal pads contacting the housing's base plate; the lower layer houses a replaceable lithium battery pack, DC-DC voltage regulator module, fuses, and surge protectors, with the battery pack featuring a pluggable design for easy field replacement. All modules within the housing are connected via standardized connectors for easy installation, maintenance, and troubleshooting.

[0064] The technical effects of this embodiment are specifically reflected in the following aspects: Through the image-spectrum-acoustic print three-modal synchronous acquisition architecture, millisecond-level synchronous acquisition of rock surface visual morphology, mineral spectrum, and hammer impact acoustic response is achieved, effectively solving the technical problem of decreased recognition accuracy under similar or complex lithological conditions due to the limited dimension of single-modal information. The three-modal complementary mechanism acquires lithological feature information synchronously from three dimensions, and the features of each modality verify and complement each other, significantly improving the recognition accuracy. The hammer triggering mechanism ensures that the acquisition conditions are consistent each time, and the unified time series benchmark eliminates the time misalignment problem of multimodal data and reduces human error. The multimodal Transformer network maps the fused features to a high-dimensional embedding space, and through the contrastive learning mechanism, it makes similar lithological features compactly clustered and dissimilar features clearly separated, effectively improving the model's discrimination ability. The real-time inference capability at the edge enables the system to make immediate decisions on-site without waiting for background analysis, significantly improving operational efficiency.

[0065] According to a second embodiment of the present invention, the present invention claims protection for a lithology determination device based on audiovisual fusion recognition, used to execute the aforementioned lithology determination method based on audiovisual fusion recognition, the device comprising: a force hammer module, with a built-in piezoelectric force sensor, outputting trigger pulses and real-time force value signals; The image module includes an industrial camera, a ring light source, and a trigger interface; the spectral module includes a miniature spectrometer, a fiber optic collimator, and a trigger interface. The voiceprint module includes a wideband microphone, a three-axis accelerometer, and a synchronous acquisition card; The edge computing module has a built-in quantized multimodal Transformer model; the display and interaction module displays the recognition results and explanatory diagrams in real time on the touch screen. The power supply and housing are integrated into a portable case structure.

[0066] This embodiment is mainly aimed at downhole drilling application scenarios, integrating the lithology identification device into the top short section of the drill pipe to realize the function of real-time lithology judgment during drilling.

[0067] In terms of hardware structure, this embodiment adapts the integrated housing: the housing material is changed from magnesium-aluminum alloy to high-strength non-magnetic stainless steel to avoid magnetic interference to the measurement-while-drilling instrument in the underground magnetic field environment, while also meeting the mine explosion-proof requirements. The external dimensions of the stainless steel housing are adjusted to 380 mm x 280 mm x 150 mm, and the weight is increased to 10 kg to meet the strength requirements of underground installation. The internal module layout of the housing maintains a three-layer structure, but the installation positions of each module are optimized and reorganized according to the underground space constraints. The light source configuration of the upper layer is modified, and the visible light LED light source is replaced with a near-infrared 940 nm wavelength invisible light source to avoid light interference with the operator's vision in the limited underground space. At the same time, the near-infrared light source has stronger penetration ability to water-bearing rock surfaces, and can obtain deeper rock surface information. The fiber optic probe of the spectral module extends out of the housing through a sapphire window. The sapphire window has a diameter of 15 mm and a thickness of 5 mm, which can withstand the impact of high-pressure drilling mud underground. The surface of the window is coated with a hydrophobic coating to prevent mud contamination. The microphone of the acoustic module is installed inside a specially designed mud shield. The inside of the shield is filled with sound-absorbing material, and the vent of the shield is equipped with a high-pass filter, which can filter out mud pump noise interference with a frequency greater than 5 kHz, so that the microphone can mainly collect low-frequency sound wave signals generated by the drill bit breaking rock.

[0068] Regarding data acquisition, the synchronous acquisition timing in this embodiment maintains the same millisecond-level synchronization accuracy as in Embodiment 1, but the triggering mechanism has been adjusted. In downhole drilling scenarios, manual tapping cannot be used to stimulate the rock surface response; therefore, this embodiment uses the vibration signal generated by the drill bit breaking the rock as the trigger source. Specifically, a piezoelectric vibration sensor is installed inside the short section at the top of the drill pipe. When the drill bit cuts into the rock surface, the rock-breaking vibration signal increases sharply. The vibration sensor output signal is processed by a comparator circuit to generate a TTL trigger pulse, which simultaneously initiates image acquisition, spectral acquisition, and acoustic signature acquisition. Since the rock-breaking process is continuous, the frequency of the trigger pulse is related to the drill bit rotation speed and propulsion speed. The system sets the acquisition duration for each trigger to 500 milliseconds, and waits for the next trigger after acquisition is completed. The image acquisition module continuously acquires 10 frames of images after triggering, at a frame rate of 20 frames per second. The image content includes the morphology, grain size distribution, and fresh cross-sectional features of the rock cuttings. The spectral acquisition module continuously acquires 50 spectral lines after triggering, with a spectral integration time of 2 milliseconds. The spectrometer's operating wavelength remains from 900 nm to 1700 nm, covering the main mineral characteristic absorption peaks. The acoustic signature acquisition module continuously acquires sound pressure and acceleration waveforms for 500 milliseconds after triggering, with a sampling rate of 256 kilosamples per second.

[0069] In terms of feature extraction, this embodiment adjusts the feature extraction network accordingly based on the characteristics of drilling data. Image feature extraction still uses the ResNet50 network structure, but the preprocessing method for the input images has changed: because the cuttings are in high-speed motion during drilling, the acquired images have motion blur. Therefore, a deblurring module is added in the preprocessing stage. A deep learning-based deblurring network is used to preprocess the images before inputting them into ResNet50. The deblurring network adopts a U-Net structure, containing a 4-layer encoder and a 4-layer decoder. The encoder extracts features, and the decoder restores the clear image. During training, the network uses synthesized motion-blurred images for supervised learning, and the deblurred images are then input into the feature extraction network. The structure of the spectral feature extraction network is the same as in Embodiment 1, but it has been optimized for the spectral characteristics of drilling cuttings: drilling cuttings usually contain a large amount of fine-grained material, making their spectral characteristics more complex. Therefore, the number of convolutional kernels in the network is increased from 128 to 256 to enhance feature representation capabilities. The structure of the acoustic signature extraction network is the same as in Example 1, but it has been optimized for the drilling noise environment: an adaptive noise suppression module has been added before the time-frequency transformation. This module uses a noise estimation method based on spectral subtraction to estimate the background noise spectrum, and then subtracts the noise estimate from the noisy signal to obtain the clean signal. Noise estimation is performed in the first 100 milliseconds of each acquisition, when the drill bit has not yet cut into the rock surface or has just left the rock surface, and the acquired noise is mainly mud pump noise and mechanical noise.

[0070] In terms of fusion recognition, the fusion decision mechanism in this embodiment is basically the same as in Embodiment 1, but multi-mine adaptive processing is added to suit the characteristics of underground application scenarios. Since the rock strata distribution and lithological characteristics differ among different mines, this embodiment adds a mine feature adaptation layer after the decision-level fusion. This adaptation layer is a lightweight fully connected network. The weight parameters are rapidly fine-tuned when each mine is used for the first time. The fine-tuning data comes from known lithological samples of that mine, and the fine-tuning process can be completed in just 30 minutes. The role of the adaptation layer is to transform the general feature mapping into a lithological feature space specific to that mine, thereby improving recognition accuracy.

[0071] The technical advantages of this embodiment are as follows: Addressing the specific needs of downhole drilling applications, the system utilizes targeted designs such as a non-magnetic stainless steel casing, near-infrared light source, sapphire window, and mud shield to achieve real-time lithology identification under drilling conditions; optimization through technologies such as defuzzification networks, adaptive noise suppression, and mine feature adaptation improves identification accuracy and environmental adaptability; the system can withstand harsh environments with high pressure, high temperature, and high dust levels downhole, and can operate continuously for more than 4 hours, meeting the lithology monitoring needs of a borehole throughout its entire lifecycle.

[0072] According to the third embodiment of the present invention, for applications in open-pit mine slope inspection and geological disaster investigation, a lithology identification device is mounted on an unmanned aerial vehicle (UAV) platform to achieve a non-contact rapid lithology survey function.

[0073] In terms of hardware structure, this embodiment features a lightweight and miniaturized system design, reducing the overall weight from 8 kg to 3 kg to meet drone payload restrictions. Specific lightweight design measures include: changing the shell material to carbon fiber composite material, adjusting the dimensions to 300 mm x 200 mm x 100 mm, and weighing 1.5 kg. The camera model remains unchanged, but the bulky lens hood has been removed, with the lens directly exposed to the air and protected against dust and water through a pressure-balanced structure. A more miniaturized version of the spectrometer has been adopted, reducing the weight from 500 g to 200 g, but the spectral resolution has been reduced from 3 nm to 5 nm in exchange for smaller size. The microphone and accelerometer use MEMS integrated sensors, integrated into a 10 mm x 10 mm x 3 mm chip, weighing only 5 g. The edge computing module uses the NVIDIA Jetson Nano, which has lower computing power than the Jetson Xavier but also lower power consumption, reducing the overall weight from 500 g to 150 g. The battery capacity has been reduced from 100 Wh to 30 Wh, providing approximately 30 minutes of flight time. The image module, spectral module, and acoustic signature module are mounted on the bottom of the drone via a three-axis gimbal. The gimbal provides stabilization control in three directions: roll, pitch, and yaw, ensuring stable pointing of the acquisition modules towards the target during drone flight. The three-axis gimbal is driven by brushless motors, achieving a control accuracy of ±0.01 degrees and a response speed meeting a 100Hz closed-loop control frequency. The drone is a multi-rotor type with a maximum payload of 5 kg, a flight time of 40 minutes, and a flight speed of up to 15 meters per second.

[0074] Regarding the excitation method, this embodiment abandons the traditional hammer striking method and adopts a high-speed electromagnetic impactor to achieve non-contact excitation. The working principle of the high-speed electromagnetic impactor is as follows: the coil is energized to generate a magnetic field, which drives the iron core to strike the rock surface and produce vibration. The impactor has a stroke of 5 mm, an impact energy of 0.3 joules, and an impact frequency that can be set from 1 to 10 Hz. The impactor communicates with the UAV-borne control system via wireless remote control. After the UAV flies over to the area to be tested and hovers, the ground control personnel send an excitation command, and the impactor generates three consecutive impacts, with each impact interval of 500 milliseconds. The sound wave signal generated by the impactor is relatively weak, so the gain setting of the acoustic fingerprint acquisition module is increased by 20 dB compared to Embodiment 1. At the same time, a signal superposition function is added to superimpose the acoustic fingerprint signals of the three impacts to enhance the signal-to-noise ratio.

[0075] In terms of positioning and attitude, this embodiment adds an RTK positioning module to obtain precise coordinates of the rock surface. The RTK positioning module adopts a dual-antenna design: the main antenna is mounted on the top of the UAV for coarse positioning with sub-meter accuracy; the secondary antenna is mounted on the bottom of the gimbal for precise positioning with centimeter accuracy. The real-time heading angle of the UAV can be calculated by the phase difference between the main and secondary antennas, with an accuracy better than 0.5 degrees. The positioning data is stored in association with the lithology identification results to form a lithology distribution map with geographic coordinate information, providing data support for subsequent geological mapping and reserve calculation. The UAV is also equipped with a laser rangefinder to measure the distance between the UAV and the rock surface, and combined with camera intrinsic parameters, the actual area and characteristic scale of the rock surface can be calculated.

[0076] In terms of image acquisition, this embodiment employs a pushbroom-style working mode. The UAV flies along a set route at a speed of 5 meters per second, with the gimbal pitch axis angle set to a 45-degree downward view, the camera's frame rate set to 10 frames per second, and the camera exposure time set to 5 milliseconds. Each exposure acquires one frame, with approximately 70% overlap between adjacent frames. Multiple frames are stitched together into a continuous panoramic image of the rock surface using an image stitching algorithm. The image stitching process is executed in real-time on the edge computing module, and the resulting panoramic image has a resolution of 8000 pixels by 4000 pixels, fully preserving the detailed information of the rock surface. Spectral acquisition uses a similar pushbroom method. The spectrometer continuously acquires data during UAV flight, with the acquisition interval synchronized with the image frame rate. The acquired spectral data is spatially aligned with the image data to form a remote sensing image of the rock surface containing spectral information.

[0077] In terms of fusion recognition, the recognition algorithm in this embodiment is optimized for UAV-borne application scenarios. Due to vibrations and attitude changes during flight, the quality of data from each modality fluctuates significantly. Therefore, this embodiment adds a quality assessment and weighting module before feature-level fusion. This module calculates a quality score for the feature vector of each modality, with scoring indicators including the variance of the feature vector, the sparsity of the feature vector, and the similarity between the feature and the template. The weight of each modality feature during fusion is dynamically adjusted based on the quality score, assigning greater weight to high-quality modality features and less weight to low-quality modality features. Modality data with excessively low quality scores is discarded and does not participate in the fusion decision. After decision-level fusion, a temporal filtering module is added, using multiple consecutive recognition results from the UAV flight process for voting filtering to further improve recognition stability.

[0078] The technical advantages of this embodiment are as follows: Addressing the specific needs of UAV-based application scenarios, it achieves a non-contact rapid lithological survey function through lightweight design, electromagnetic shock excitation, RTK positioning, and push-broom data acquisition; the system can cover areas such as slopes and steep cliffs that are difficult for humans to reach, and its identification efficiency is dozens of times higher than that of manual methods; each flight can complete a lithological survey of approximately 10,000 square meters, and the resulting lithological distribution map can be directly used for geological mapping and slope stability assessment.

[0079] According to the fourth embodiment of the present invention, a reliable lithology identification function is achieved in a low temperature environment of -40 degrees Celsius to -60 degrees Celsius for the identification of freeze-thawed rocks in high-altitude areas and polar scientific research applications.

[0080] In terms of hardware structure, this embodiment features a comprehensive low-temperature adaptability modification to the system. The shell material has been changed to a low-temperature-specific titanium alloy, which maintains good toughness and does not exhibit brittleness at -60 degrees Celsius. Multiple layers of insulation jackets are installed inside the shell, using aerogel composite material with a thickness of 20 mm and a thermal conductivity of less than 0.02 W / m / Kelvin, effectively preventing external low temperatures from penetrating the shell. A miniature heating film with a power of 50 watts is also installed inside the shell, powered by the battery through a DC-DC boost module. The system operates in two modes: in standby mode, only the heating film maintains the internal shell temperature above -10 degrees Celsius, allowing the battery to support 72 hours of standby; in operating mode, the heating film operates at full power, raising the internal shell temperature to above 5 degrees Celsius, at which point all electronic components function normally. The battery used is a low-temperature lithium thionyl chloride battery, which retains 80% of its discharge capacity at -40 degrees Celsius, with a capacity of 150 Wh. The glass window of the optical component has been replaced with fused silica glass. Fused silica has better low-temperature transmittance than ordinary optical glass and an extremely low coefficient of thermal expansion, preventing the window from cracking due to temperature changes. The LED chip of the light source has been changed to a flip-chip structure, resulting in a shorter heat dissipation path and allowing for rapid heating to operating temperature at low temperatures. The sensor cable uses fluoroplastic insulation material. Fluoroplastic has better low-temperature flexibility than ordinary PVC and can remain flexible even at -60 degrees Celsius.

[0081] In terms of data acquisition, this embodiment optimized the acquisition parameters based on the physical characteristics of freeze-thawed rocks. Since the acoustic response of freeze-thawed rocks under hammer impact differs significantly from that of rocks at room temperature (indicating faster sound wave propagation speed, reduced attenuation coefficient, and a shift in resonant frequency to higher frequencies), the sampling rate of the acoustic signature acquisition module was increased from 256 kilosamples per second to 512 kilosamples per second to capture more high-frequency details. The analysis bandwidth of the acoustic signature data was expanded from 20 kHz to 30 kHz. Regarding spectral acquisition, the surface of freeze-thawed rocks is usually covered with ice film or frost. Ice film has strong absorption characteristics in the near-infrared band, which can interfere with the spectral characteristics of minerals. Therefore, polarization functionality was added to the spectral acquisition. Polarization analysis can distinguish between ice film reflected light and mineral reflected light, improving the accuracy of spectral feature extraction. In terms of image acquisition, due to the dry air and reduced atmospheric scattering in low-temperature environments, image contrast and sharpness are actually improved. Therefore, the camera exposure time can be appropriately reduced to decrease motion blur, while automatic gain control is added to compensate for possible changes in lighting.

[0082] In terms of feature extraction, this embodiment specifically trains the neural network to capture the characteristics of freeze-thawed rocks. The training dataset contains a large number of freeze-thawed rock samples collected in low-temperature environments, including frozen sandstone, frozen limestone, and frozen mudstone. During network training, a penalty term for freeze-thaw features is added to the loss function, enabling the network to learn the correlation between freeze-thaw features such as ice film reflection and ice layer thickness and lithological categories. The output dimension of the feature extraction network remains unchanged, but the network weight parameters are fine-tuned for the freeze-thaw scenario. The fine-tuning dataset contains approximately 500 freeze-thawed rock samples, and the fine-tuning process takes about 2 hours.

[0083] Regarding system self-testing, this embodiment adds a low-temperature operating adaptability self-test function. Each time the system is powered on, it first performs a self-test, including: sensor response testing, optical system focusing testing, acoustic system sensitivity testing, and computing module function testing. During the self-test, the heating film first raises the internal temperature of the housing to above 0 degrees Celsius, and then each module is powered on sequentially and performs its function test. After the self-test is completed, the system reports the self-test results. If a module fails the self-test, an error code is displayed on the screen along with maintenance suggestions. After the self-test passes, the system enters the working state. At this time, the heating film can reduce power to maintain temperature balance and save power consumption.

[0084] The technical advantages of this embodiment are as follows: For applications in extremely cold environments, reliable lithology identification is achieved at -60 degrees Celsius through technologies such as titanium alloy shells, aerogel insulation, low-temperature batteries, and heating films; the acquisition parameters and algorithms have been optimized and adjusted to address the physical characteristics of freeze-thawed rocks, ensuring that the identification accuracy is unaffected by low temperatures; the system can be used in extreme environments such as high-altitude mountainous areas and polar expeditions, expanding the application scope of this invention.

[0085] The fifth embodiment of the basic aspect provides a reliable lithology identification function for applications in seabed lithology detection and geological surveys before offshore oil and gas drilling, under high-pressure and saline-humid underwater environments.

[0086] In terms of hardware structure, this embodiment features a comprehensive modification to adapt the system to the marine environment. The shell material has been changed to titanium alloy or super duplex stainless steel, capable of withstanding underwater pressure of 30 MPa. The shell adopts a cylindrical design to enhance pressure resistance, with a diameter of 250 mm, a length of 400 mm, and a weight of 15 kg. The exterior of the shell is coated with an anti-corrosion coating, which uses a composite system of epoxy zinc-rich primer and polyurethane topcoat, resisting seawater corrosion for up to 5 years. The optical window uses an optical glass structure with a metal sealing ring. The glass is selected as high-pressure-resistant optical glass, capable of withstanding external pressure of 50 MPa. The light source uses waterproof LEDs with an IP68 protection rating, allowing continuous operation underwater. The electrical connectors use SubConn series seawater connectors, with internal oil filling to prevent seawater ingress. The system's power supply has been changed to umbilical cable power, supplied by the surface vessel via the umbilical cable. The umbilical cable also transmits data signals. The power input is 48 volts DC, with a power of 100 watts. The edge computing module can also be equipped with a surface installation mode, where a high-performance GPU server is installed in the surface control box, and only sensors and simple preprocessing circuits are installed on the underwater probe. The raw data is transmitted to the surface for processing via an umbilical cable.

[0087] Regarding data acquisition, this embodiment adjusted the acquisition parameters to suit the characteristics of the underwater environment. For image acquisition, due to severe light attenuation underwater, the camera incorporated infrared auxiliary illumination. Infrared light penetrates water several times farther than visible light. A 940nm wavelength LED with a power of 10 watts was selected as the infrared light source, matching the camera's near-infrared sensitivity. The camera is equipped with a dedicated underwater lens, which offers a wider field of view than in air, compensating for the reduced field of view caused by refraction. For spectral acquisition, the underwater spectrometer's operating wavelength was adjusted to 400nm to 900nm, as water's absorption characteristics prevent infrared light above 900nm from propagating underwater. The spectrometer's probe housing was designed with a streamlined shape to reduce water resistance, and a self-cleaning brush was installed on the probe surface to prevent marine organisms from adhering. For acoustic signature acquisition, the underwater acoustic environment differs significantly from that in air. Sound waves travel at approximately 1500 meters per second in seawater, and the attenuation characteristics of seawater differ from those in air; therefore, the acoustic signature acquisition algorithm requires corresponding compensation. The data acquisition card has been upgraded with a preamplifier whose gain can be adjusted from 20 dB to 60 dB. The gain is automatically adjusted according to the water depth to compensate for the effect of hydrostatic pressure on the sensor sensitivity.

[0088] In terms of feature extraction, this embodiment specifically trained the neural network to capture the features of underwater rocks. The training dataset contains a large number of rock samples collected in underwater environments, including seafloor basalt, seafloor sedimentary rocks, and seafloor coral reefs. An underwater feature enhancement module was added during network training. This module uses underwater image enhancement algorithms to preprocess the input images, including contrast enhancement, color correction, and dehazing, before inputting them into the ResNet50 for feature extraction. A water absorption compensation algorithm was added to spectral feature extraction, estimating and compensating for water absorption based on water depth and the distance between the spectrometer and the rock surface. A water acoustic characteristic compensation algorithm was added to acoustic feature extraction, estimating and compensating for sound velocity based on hydrostatic pressure and temperature.

[0089] In terms of data communication, this embodiment adds two methods: underwater acoustic communication and fiber optic communication. Underwater acoustic communication is used for short-range communication with surface vessels or underwater vehicles, with a communication distance of up to 2 kilometers and a communication rate of approximately 2400 bits per second. Fiber optic communication is used for large-volume data transmission. The fiber optic cable is integrated with the umbilical cable, enabling the transmission of real-time video and spectral data at a transmission rate of up to 1000 megabits per second. The system also supports local storage mode; the edge computing module has a built-in 2TB solid-state drive, capable of storing all data collected during 48 hours of continuous operation, which can then be processed offline after the equipment is recovered.

[0090] The technical advantages of this embodiment are as follows: Addressing the specific needs of the underwater marine environment, reliable lithology identification is achieved within a depth range of 30 to 3000 meters through technologies such as pressure-resistant shells, anti-corrosion coatings, and specialized sensors; acquisition parameters and algorithms have been optimized and adjusted to account for underwater light attenuation and changes in acoustic characteristics, resulting in an identification accuracy rate essentially on par with that of the terrestrial environment; the system can be used in fields such as seabed geological surveys, marine oil and gas exploration, and marine engineering geological surveys, with broad application prospects.

[0091] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0092] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated units described above can be implemented in hardware or as software functional units. The above are merely embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made based on the description and drawings of this application, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

[0093] The specific embodiments of the invention have been described in detail above, but they are only examples, and this application is not limited to the specific embodiments described above. For those skilled in the art, any equivalent modifications or substitutions to the invention are also within the scope of this application. Therefore, all equivalent changes, modifications, and improvements made without departing from the spirit and principles of this application should be covered within the scope of this application.

Claims

1. A lithology determination method based on audiovisual fusion recognition, comprising the following steps: S1. Multimodal data synchronous acquisition is achieved through image acquisition, spectral acquisition, voiceprint acquisition, and unified time-series synchronization; S2, Multi-source data fusion, performs spatial alignment, feature extraction, feature-level fusion and decision-level fusion to achieve deep fusion of image, spectral and voiceprint features; S3, Artificial Intelligence Recognition: Rock classification is performed through a multimodal Transformer network, and real-time and efficient recognition is achieved by combining a semi-supervised training strategy and edge quantization deployment.

2. The lithology determination method based on audiovisual fusion recognition according to claim 1, characterized in that, S1 specifically includes the following steps: S11. Image acquisition: An industrial camera is set up directly above the rock surface to capture at least three frames of images before and after the moment of hammering, in order to obtain information on rock surface texture, chipping, and debris distribution. S12. Spectral acquisition: A visible-near-infrared spectrometer is set obliquely on the rock surface to simultaneously acquire the rock surface reflectance spectrum at the moment of hammering, covering the preset waveband. S13. Acoustic print acquisition: A wideband microphone and a three-axis accelerometer are set on the rock surface to simultaneously record the sound pressure waveform and acceleration waveform at the moment of hammering. S14. Timing synchronization: Based on the trigger pulse output by the piezoelectric sensor built into the hammer, image, spectrum, and acoustic data are stamped with a unified timestamp to achieve millisecond-level synchronization.

3. The lithology determination method based on audiovisual fusion recognition according to claim 2, characterized in that, In step S11, the industrial camera uses a global shutter CMOS image sensor with a pixel count of not less than a preset value. It works with a fixed-focus lens at a preset working distance, and the corresponding field of view covers the rock surface to be measured area. The pixel resolution meets the requirements for identifying fine textures on the rock surface. The camera is equipped with a ring LED light source, which consists of a high color rendering index LED array. It has independent dimming in multiple quadrants and uses preset frequency PWM modulation to avoid flickering with image exposure. A polarizing optical element is added in front of the light source to reduce specular reflection interference. In step S12, the visible-near-infrared spectrometer adopts a MEMS scanning grating structure, has a preset resolution and a preset signal-to-noise ratio, a field-of-view collimating lens is installed at the front end of the fiber optic probe, the working distance corresponds to the measurement spot diameter and is coaxially arranged with the image center area, and a laser pointer is placed next to the probe for on-site alignment.

4. The lithology determination method based on audiovisual fusion recognition according to claim 2, characterized in that, In S13, the wideband microphone is a free-field capacitor microphone with preset sensitivity and preset dynamic range. The triaxial accelerometer is of the IEPE type, with preset sensitivity and preset bandwidth. It is acquired by a synchronous acquisition card at a sampling rate of not less than the preset sampling rate to ensure that data is not lost in the event of sudden triggering. In step S14, the trigger pulse is converted into a voltage signal by a charge amplifier and then sent to the acquisition card. It is then distributed to each acquisition module via TTL. The total delay of the trigger chain is less than a preset microsecond value, which is less than the image exposure time and the spectral integration time. The data acquisition card uses a temperature-compensated crystal oscillator as its clock source. The frequency stability meets the preset specifications. It synchronizes with the reference clock source once during the first power-on each day, and the accumulated drift meets the real-time requirements.

5. The lithology determination method based on audiovisual fusion recognition according to claim 1, characterized in that, S2 specifically includes the following steps: S21. Spatial alignment: Establish the mapping relationship between image pixel coordinates and spectral field of view spatial coordinates based on checkerboard joint calibration. S22. Feature extraction: The image is processed by a deep convolutional neural network to extract the first-dimensional deep features, the spectrum is processed by a one-dimensional convolutional neural network to extract the second-dimensional features, and the voiceprint is processed by time-frequency transformation combined with a two-dimensional convolutional neural network to extract the third-dimensional features. S23, Feature-level fusion, which concatenates three-modal features in the channel dimension to obtain multimodal fused features; S24. Decision-level fusion: The soft probabilities output by the three modes are summed by dynamic weights to obtain the final lithology probability distribution.

6. The lithology determination method based on audiovisual fusion recognition according to claim 5, characterized in that, In S22, the image feature extraction adopts a deep residual network structure with a preset output dimension to extract visual features such as rock surface texture, color distribution, and collapse morphology. Spectral feature extraction employs a one-dimensional convolutional neural network to extract spectral features such as the position and depth of mineral characteristic absorption peaks. Voiceprint feature extraction employs filter-based time-frequency transformation to extract acoustic features, followed by extraction of voiceprint time-frequency spectrum features via a two-dimensional convolutional neural network. In S23, feature-level fusion concatenates the first-dimensional feature, the second-dimensional feature, and the third-dimensional feature in the channel dimension to form a fused feature vector. The fused feature dimension is the sum of the feature dimensions of each modality, providing rich cross-modal information for the subsequent multimodal Transformer network. In S24, the decision-level fusion adopts a dynamic weighting mechanism. The weights are dynamically adjusted according to the quality assessment results of each modality feature. When the quality of a certain modality feature deteriorates due to environmental interference, its weight is automatically reduced, thereby achieving robust multimodal decision fusion.

7. The lithology determination method based on audiovisual fusion recognition according to claim 1, characterized in that, S3 specifically includes the following steps: S31. Construct a multimodal Transformer network, with fused features as input and lithology category and confidence level as output; S32. Joint training with cross-entropy loss and center loss is adopted, and unlabeled field samples are expanded through a semi-supervised strategy. S33. After quantization, the model is deployed on the edge computing unit, and the single recognition latency meets the real-time requirements. S34. When the confidence level is lower than the preset threshold, manual review is triggered, and the review results are updated with the model parameters through incremental learning to achieve closed-loop optimization.

8. The lithology determination method based on audiovisual fusion recognition according to claim 7, characterized in that, In S31, the multimodal Transformer network adopts a multi-layer encoder structure, each layer containing a multi-head self-attention mechanism and a feedforward network. The number of self-attention heads meets the requirements of multimodal feature interaction, and the hidden layer dimension meets the feature representation capability. The network introduces modality location encoding and modality type embedding to represent the spatial positional relationship and modality identity information of each modality feature, respectively, thereby enhancing the network's ability to model multimodal temporal relationships; In S32, the joint training strategy simultaneously optimizes classification loss and metric loss. The classification loss ensures that the network learns the correct lithology category mapping, while the metric loss ensures that features of the same type of lithology are compactly distributed in the embedding space and features of different types are far apart. The semi-supervised strategy utilizes a large number of unlabeled field samples for pseudo-label training, and ensures the quality of pseudo-labels through a confidence screening mechanism, thereby enabling the model to adapt to complex field environments.

9. The lithology determination method based on audiovisual fusion recognition according to claim 7, characterized in that, In S33, the model quantization adopts a preset precision quantization method and is deployed on an embedded AI computing platform. The platform has GPU resources that meet the computing power requirements and the power consumption is controlled within a preset range. In S34, the incremental learning method, while retaining the original knowledge, incorporates the results of manual verification into the model through weight importance assessment and regularization constraints, enabling the model to continuously learn new lithological characteristics and ensuring the model's long-term adaptive capability under complex geological conditions.

10. A lithology determination device based on audiovisual fusion recognition, used to execute a lithology determination method based on audiovisual fusion recognition as described in any one of claims 1-9, the device comprising: The force hammer module has a built-in piezoelectric force sensor that outputs trigger pulses and real-time force value signals. The image module includes an industrial camera, a ring light source, and a trigger interface; The spectral module includes a miniature spectrometer, a fiber optic collimator, and a trigger interface; The voiceprint module includes a wideband microphone, a three-axis accelerometer, and a synchronous acquisition card; The edge computing module has a built-in quantized multimodal Transformer model; The display and interaction module shows the recognition results and explanatory diagrams in real time on the touch screen; The power supply and housing are integrated into a portable case structure.

Images