Fusion method and device suitable for acoustic image long-distance identification system

Abstract

The present application relates to the field of acoustic image detection, and particularly relates to the optimization design in the data feature fusion process.The present application is realized by the following technical scheme: a fusion method suitable for an acoustic image long-distance identification system, S01, hardware arrangement and data acquisition; a gait acquisition device for acquiring gait information and a distance measuring device for obtaining the distance d between the acoustic image instrument and the to-be-measured equipment are arranged on the acoustic image instrument; S03, dynamic adjustment of weight; S05, acquisition of identification features.The purpose of the present application is to provide a fusion method and equipment suitable for an acoustic image long-distance identification system, to fuse the data features in three dimensions of video, audio and gait, and to generate a weight value based on the distance between the acoustic image instrument and the measured object, thereby significantly improving the detection accuracy and robustness of the identification system.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of sound image detection, in particular to the optimization design in the data feature fusion process. BACKGROUND

[0002] Industrial equipment, such as large manufacturing equipment, molding equipment, and other commercial equipment, is widely used in factories, research institutions, and other places, and is an important asset of its affiliated unit. During the use of these industrial equipment, the health level needs to be monitored regularly to avoid the equipment from stopping running due to faults or causing safety hazards.

[0003] Among many health detection methods, compared with video monitoring, on-site monitoring by administrators, and other methods, the sound image detection technology has become one of the important means for industrial equipment fault detection due to its controllable hardware investment cost, labor saving, and wide monitoring range. The implementation of sound image detection uses a sound image instrument, such as the sound image instrument structure disclosed in the Chinese patent document with application number 202320049695.7, which has a display screen, a microphone, and a camera in the main part. The microphone and the camera capture audio signals and video signals, respectively.

[0004] The detection device extracts the audio signal and processes it into an audio feature F audio; extracts the video signal and processes it into a video feature F visual. Then, the two features need to be fused to obtain a fused feature F fused, where α and β are the weight values of the two sets of feature signals, respectively.

[0005] However, this technical solution has two technical defects: on the one hand, α and β are often fixed weights, and different scenes, environments, and conditions are complex and changeable. Although the fixed weight processing method simplifies the calculation, the detection accuracy decreases. On the other hand, this technical solution only considers the audio feature and the video feature, and has a certain applicability when the sound image instrument is stationary. However, the operator and the sound image instrument are not stationary and are often moving. At this time, only using the video feature and the audio feature has a negative impact on the accuracy of the data. SUMMARY

[0006] The purpose of the present application is to provide a fusion method and device suitable for a sound image long-distance recognition system, which fuses data features in three dimensions of video, audio, and gait, and generates weight values based on the distance between the sound image instrument and the measured object, significantly improving the detection accuracy and robustness of the recognition system.

[0007] The application is realized by the following technical scheme: a fusion method suitable for a sound image long-distance identification system, characterized by comprising the following steps: S01, hardware arrangement and data collection; setting a gait acquisition device for acquiring gait information and a ranging device for obtaining the distance d between the sound image instrument and the equipment to be measured on the sound image instrument, and the sound image instrument collects audio features X audio, visual features X visual and gait features X gait; S03, dynamic adjustment of weight; calculating the distance attenuation factor ; , e is a natural constant, d0 is a self-defined distance turning threshold, and s is a self-defined smoothing factor; calculating the distance enhancement factor of the gait model ; , tanh is a hyperbolic tangent function, d1 is a self-defined gait activation threshold, s1 is a self-defined smoothing factor, and the distance adaptive weight vector w(d) is calculated , softmax is a normalized exponential function, is the preset initial base weight coefficient of audio, video and gait; S05, identification feature acquisition; based on the distance adaptive weight vector w(d) and the audio features X audio, the visual features X visual and the gait features X gait collected by the sound image instrument, the identification features .

[0008] As a preferred embodiment of the application, after S05, S07, attention concentration fusion step, is further included; in this step, the identification features are input into a multi-head attention model to obtain fusion identification features .

[0009] As a preferred embodiment of the application, the S07 comprises the following sub-steps: S071, projection matrix generation; , wherein Q', K', V' are query vector, key vector and value vector respectively. i is the number of the number of multi-head attention heads, is the projection matrix of the i-th head; S073, single-head attention calculation; ; ; is the result of the i-th attention head, is the weighted value vector, softmax is a normalized standard function, is a query-key similarity matrix, is a self-defined scaling factor; S075, multi-head fusion; , Q, K, V are query, key and value respectively, MultiHead(Q,K,V) represents a multi-head attention output tensor, h is the number of attention heads, and W ois an output linear projection matrix, and Concat is a tensor concatenation operation.

[0010] As a preferred embodiment of the present application, before S01, a training step of the multi-head attention model is further included, in which original test data is acquired by using a sonographic measurement, and the original test data is manually labeled to obtain a labeled data set, and the original test data is subjected to processes S01-S07 to obtain a fusion mapping function; the difference between the labeled data set and the fusion mapping function is calculated based on an overall loss function to complete the training of the multi-head attention model.

[0011] As a preferred embodiment of the present application, in S01, the audio encoder converts the original waveform into a frequency domain representation based on a Mel spectrogram to obtain audio features X audio; the visual encoder captures image spatial structure based on a convolutional neural network to obtain visual features X visual; and the gait encoder processes an acceleration sequence based on a time-recurrent network to obtain gait features X gait.

[0012] As a preferred embodiment of the present application, the distance measuring device is a laser range finder or an ultrasonic sensor.

[0013] An electronic device includes a processor and a memory; the processor is connected to the memory;

[0014] The memory is configured to store executable program codes; the processor is configured to run a program corresponding to the executable program codes by reading the executable program codes stored in the memory, so as to execute the method described above.

[0015] In summary, the present application has the following advantages:

[0016] 1. The fusion of audio signals and video signals is not fixed weight, but dynamic adjustment weight based on distance calculation, the audio weight is larger when the distance is short, and the video weight is larger when the distance is far, which is beneficial to the accuracy of the fused signal, can adapt to different industrial test environments, and improves the robustness of the recognition system.

[0017] 2. Compared with the traditional "binary fusion", the present technical solution increases the consideration and calculation of "ternary fusion" of gait, which is more beneficial to the signal input of the sonographic motion state through the analysis of acceleration data.

[0018] 3. The attention mechanism fusion is increased, the multi-head attention model is introduced, the expression richness of the fusion is enhanced, the feature continuity from the weight calculation to the optimization stage is ensured, and the single mechanism missing key cross-modal interaction is avoided.

[0019] 4. The multi-head attention model is pre-trained before formal use, the gap between prediction and real features is minimized, overfitting is controlled, high generalization fusion output is ensured, and stable performance in real-time application is supported. BRIEF DESCRIPTION OF DRAWINGS

[0020] Figure 1 is a flowchart of embodiment 1 of the present application.

[0021] Figure 2 is a flowchart of the sub-step S03 in embodiment 2. DETAILED DESCRIPTION

[0022] The present application will be further described in conjunction with the accompanying drawings of the embodiments in the present specification.

[0023] The technical solutions in the embodiments of the present specification will be clearly and completely described in conjunction with the accompanying drawings of the embodiments in the present specification.

[0024] The terms “first”, “second”, “third” and the like in the specification and claims of the present specification and the above-mentioned accompanying drawings are used to distinguish different objects, and are not used to describe a specific order. In addition, the terms “include” and “have” and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device including a series of steps or units is not limited to the listed steps or units, but can optionally include steps or units not listed or can optionally include other steps or units inherent to the process, method, product or device.

[0025] The following description provides examples and does not limit the scope, applicability or examples set forth in the claims. Changes can be made to the functions and arrangements of described elements without departing from the scope of the present specification. Various examples can appropriately omit, replace or add various processes or components. For example, the described methods can be executed in different order from the described order, and various steps can be added, omitted or combined. In addition, features described with respect to some examples can be combined into other examples.

[0026] Embodiment 1, as Figure 1As shown, the fusion method suitable for the long-distance identification system of the sound image is first entered into S01, the hardware arrangement and data collection step. The hardware arrangement is increased with the gait acquisition device on the traditional sound image instrument. The gait sensing device can select the mature acceleration sensor in the prior art. It often contains an IMU, that is, an inertial measurement unit, which can obtain the acceleration data of the sound image instrument. When it is static, the acceleration data is zero. When the operator holds the sound image instrument to move and collect signals, the sound image instrument obtains three original signals, which are the video original signal obtained through the camera on the sound image instrument, the audio original signal obtained through the microphone on the sound image instrument, and the gait original signal obtained through the acceleration sensor on the camera.

[0027] Subsequently, the feature extraction module extracts features from the three original signals, removing useless information in the signals. This step is the same as the prior art, for example, the feature extraction module contains various encoders, the audio encoder converts the original waveform into a frequency domain representation based on the Mel spectrogram, the visual encoder captures the image spatial structure based on the convolutional neural network, and the gait encoder processes the acceleration sequence based on the time sequence recurrent network. This step ensures that the modal data is converted into a unified high-dimensional vector form, providing a standardized input basis for subsequent weight adjustment and attention fusion.

[0028] This step ends, and the audio feature X audio, the visual feature X visual and the gait feature X gait are obtained.

[0029] Subsequently, S03 is entered, which is also a key step, the weight dynamic adjustment step.

[0030] The distance factor is introduced to bridge the physical environment and data fusion, and the weight function is defined. The distance value d refers to the straight-line distance between the sound image instrument and the measured object, which can be obtained by the ranging device installed on the sound image instrument, such as laser ranging or ultrasonic sensor.

[0031] The formula is:

[0032] ,

[0033] Wherein, w(d) represents the distance adaptive weight vector, which is a comprehensive, three-dimensional vector. It corresponds to the normalized weights of the audio, visual and gait modalities respectively;

[0034] The softmax function is a normalized exponential function, which is used to convert vector elements into probability distribution, ensuring that the weight sum is 1; is the base weight coefficient of audio, video and gait, which is used for initial scaling. The initial value is usually set to 1 to achieve an equal weight starting point.

[0035] and Then the following sub-formula is obtained: .

[0036] wherein, represents the distance decay factor of the audio modality, the output range [0, 1], simulating the nonlinear process of signal decay. e is the natural constant; d is the distance value, which is obtained by the ranging device as described above; d0 is the distance turning threshold, which is self-defined, and is generally preset according to the specific application scenario, such as d0 = 5m in an indoor environment and d0 = 10m in a long-distance outdoor recognition. s is a self-defined smoothing factor, which is usually in the range of 5-10, and is fine-tuned in training through sensitivity analysis and gradient descent to avoid weight mutation.

[0037] The function design is actually a sigmoid function, which smoothes the influence of distance on audio reliability. When d is close to or less than d0, tends to 1, maintaining a high weight of audio. When d is much greater than d0, it tends to 0, attenuating the contribution of audio.

[0038] .

[0039] wherein, is the distance enhancement factor of the gait modality, the output range [-1, 1], which can be mapped to [0, 1] through an activation function later. tanh is the hyperbolic tangent function, which provides a S-shaped nonlinear transformation; d is the distance value, and d1 is the gait activation threshold, which is self-defined and set according to the effective distance range of gait data, such as setting d1 = 20m, indicating that long-distance gait is more reliable; s1 is a self-defined smoothing factor, usually 10-20, which is optimized through hyperparameter search in model training.

[0040] The function design is a tanh function, which gradually enhances the weight of the long-distance gait modality. When d is greater than d1, tends to 1, i.e. gait dominates. When d is less than d1, tends to -1, which can be mapped to a low weight, thereby compensating for the attenuation of other modalities and promoting the environmental robustness of the overall method.

[0041] This step realizes the smooth transition from near-distance audio dominance to long-distance vision / gait compensation through the adaptive weight mechanism.

[0042] This step is the key step of the present application, and is also the difference from the prior art. On the one hand, it avoids using the same weight to process the video features and the audio features in different scenes, but dynamically adjusts the weight based on the distance value d. When the distance value is low, the audio signal attenuation is small, and the weight is larger; when the distance value is far, the audio signal attenuation is large, and the weight of the video features and the gait features is larger. On the other hand, compared with the binary consideration of "video + audio" in the prior art, the present scheme is a ternary consideration of "video + audio + gait", thereby improving the data accuracy of the sound image instrument in the non-stationary conditions such as motion and shaking.

[0043] S05, identifying feature acquisition step.

[0044] The idea of this step is the same as that of the prior art, and it is an integration step. The identifying feature is the feature value corresponding to the dimension multiplied by the weight value corresponding to it, and is obtained by adding them together.

[0045] The formula is:

[0046] ,

[0047] Among them, is the identifying feature, which is a multi-modal vector, and so on are three weight values calculated in the S03 step; and and so on are the output of the feature value through the encoder, which has been obtained in the S01 step above.

[0048] Embodiment 2, the difference from Embodiment 1 is that an S07, attention concentrated fusion step is added.

[0049] In this embodiment, a multi-head attention model is introduced, and the above is taken as the input data. Through the multi-head attention model, the output data is the fusion identifying feature . Compared with the previous identifying feature, the fusion identifying feature further cross-cues depth association, can capture the deep interaction between modalities, thereby enhancing the expression richness of fusion, ensuring the feature continuity from weight calculation to optimization stage, and avoiding missing key cross-modal interactions by a single mechanism.

[0050] The selection and use method of the multi-head attention model can adopt the multi-head model and method in the prior art. In this embodiment, it is specifically as shown in Figure 2 .

[0051] S071, projection matrix generation;

[0052] ,

[0053] Q', K', V' are query vector, key vector, value vector respectively. i is the number of the head of multi-head attention, is the projection matrix of the i-th head, used for subspace mapping. While has been obtained in step S05.

[0054] S073, single-head attention calculation;

[0055] ,

[0056] ,

[0057] wherein, is the result of the i-th attention head.

[0058] is the weighted value vector, and softmax is the attention probability normalization standard function, is the query-key similarity matrix; is the scaling factor to prevent gradient explosion; d k is the custom key vector dimension, usually 64, preset by model architecture design. The role of the attention function is to adaptively weight important features, realize the inter-modal soft attention mechanism, and ensure the coherence and robustness of the fusion stage.

[0059] S075, multi-head fusion;

[0060] ,

[0061] Q, K, V are query, key, and value respectively. MultiHead(Q, K, V) represents the multi-head attention output tensor, integrating multiple attention views. h is the number of attention heads, W o is the output linear projection matrix for dimension matching. Concat is the tensor concatenation operation. The role of the entire S07 is to calculate the head-level attention, realize fine-grained modal attention, promote the deep integration of features in the process, and improve the adaptability to high-dimensional space.

[0062] At this point, the entire multi-head attention model has been built, and the user will input the above as input data, and the output data of the multi-head attention model is the fusion recognition feature .

[0063] Embodiment 3, on the basis of embodiment 2, adds a training step for the multi-head attention model.

[0064] The implementation time of this step is before S01, which is a pre-training step.

[0065] The operator first operates the acoustic image instrument to obtain a set of test data, for example, the test data is 1 minute long, 10 frames per second, and a total of 600 frames of original test data are obtained.

[0066] The original test data is subjected to all the steps S01-S07 described above, and the fusion mapping function .

[0067] The optimization method is:

[0068] wherein J is the overall loss function, is the fusion mapping function, f i is the ground truth target feature, from the labeled data set that has been labeled. Lambda is the regularization weight hyperparameter, and is a self-defined hyperparameter, ranging from 0.001 to 0.1, which is optimized by cross-validation. W is all trainable weight matrices in the entire mapping function Phi.

[0069] The entire function uses the Adam optimizer to iteratively minimize J, ensuring feature space density and generalization ability. That is, by adjusting the audio weight, visual weight and gait weight in the multi-head attention model, the value of J is minimized. When J is minimized, it is considered that the training of the entire multi-head attention model is completed, and the model is trained to be optimal.

[0070] The purpose of this step is to minimize the difference between the prediction and the true feature, while controlling overfitting to ensure that the entire method produces a highly generalized fusion output, supporting stable performance in real-time applications.

[0071] The application also discloses an electronic device which can include at least one processor, at least one network interface, memory and at least one communication bus.

[0072] The communication bus can be used to realize the connection and communication of the above-mentioned various components.

[0073] The user interface can include a key, and the optional user interface can further include a standard wired interface, a wireless interface.

[0074] The network interface can include, but is not limited to, a Bluetooth module, an NFC module, a Wi-Fi module, etc.

[0075] The processor can include one or more processing cores. The processor connects various parts within the entire electronic device by various interfaces and lines, executes various functions of the electronic device and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory, and calling data stored in the memory. Optionally, the processor can be implemented in at least one hardware form of a DSP, an FPGA, and a PLA. The processor can integrate one or a combination of a CPU, a GPU, and a modem. Among them, the CPU mainly processes operating systems, user interfaces, and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; and the modem is used for processing wireless communication. It can be understood that the above-mentioned modem can also not be integrated into the processor, but can be realized by a separate chip.

[0076] The memory can include a RAM and can also include a ROM. Optionally, the memory includes a non-transitory computer readable medium. The memory can be used to store instructions, programs, codes, code sets or instruction sets. The memory can include a program storage area and a data storage area, wherein the program storage area can store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playing function, an image playing function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area can store data involved in the above-mentioned various method embodiments, etc. The memory can also be at least one storage device located away from the above-mentioned processor. As a computer storage medium, the memory can include an operating system, a network communication module, a user interface module, and a broadcast upgrade application. The processor can be used to call the application stored in the memory and execute the steps of the above-mentioned high versatility design method of transition metal chromophore.

[0077] The embodiments of the present specification also provide a computer readable storage medium, which stores instructions, when running on a computer or a processor, causes the computer or the processor to execute the steps of one or more of the above-mentioned broadcast upgrade method embodiments. When the above-mentioned components of the electronic device are realized in the form of software function units and sold or used as independent products, they can be stored in the computer readable storage medium.

[0078] In the above embodiments, all or part of the methods can be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the methods can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the specification are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in or transmitted by a computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center through a wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) manner. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server, data center, etc. that includes one or more available media sets. The available medium can be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a digital versatile disc (DVD)), or a semiconductor medium (such as a solid state disk (SSD)), etc.

[0079] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by a computer program instructing related hardware, which can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the above-mentioned embodiments. The storage medium includes ROM, RAM, magnetic or optical disc, and other program code storage media. In the case of no conflict, the technical features in the embodiments and the embodiments can be combined arbitrarily.

[0080] The above-described embodiments are merely described as the preferred embodiments of the specification, and do not limit the scope of the specification. Without departing from the design spirit of the specification, various modifications and improvements of the technical solutions of the specification made by a person of ordinary skill in the art should fall within the protection scope determined by the claims of the specification.

Claims

1. A fusion method suitable for a long distance acoustic image recognition system, characterized in that, Comprising the following steps: S01, hardware arrangement and data collection; setting gait acquisition device for acquiring gait information and ranging device for obtaining distance d between sonolator and to-be-tested equipment on sonolator, and sonolator collects audio feature X audio, visual feature X visual and gait feature X gait; S03, dynamic adjustment of weight; calculating distance attenuation factor ; , e is a natural constant, d0 is a self-defined distance turning threshold, and s is a self-defined smoothing factor; calculating distance enhancement factor of gait model ; , tanh is hyperbolic tangent function, d1 is a self-defined gait activation threshold, s1 is a self-defined smoothing factor, and distance adaptive weight vector w(d) is calculated , softmax is a normalized exponential function, is a preset initial base weight coefficient of audio, video and gait; S05, recognition feature acquisition; based on the distance adaptive weight vector w(d) and the audio feature X audio, the visual feature X visual and the gait feature X gait collected by the sonolator, the recognition feature is obtained.

2. The fusion method suitable for the acoustic image long-distance recognition system according to claim 1, characterized in that: After S05, S07 is further included, a step of focused fusion; in this step, the recognition features are input to a multi-head attention model to obtain fused recognition features .

3. The fusion method suitable for the acoustic image long-distance recognition system according to claim 2, characterized in that: The S07 comprises the following sub-steps: S071, projection matrix generation; , wherein Q', K', V' are query vector, key vector, value vector respectively; i is the number of the number of attention heads, is the projection matrix of the i-th head; S073, single-head attention calculation; ; ; is the result of the i-th attention head, is the weighted value vector, and softmax is a normalization standard function, is the query-key similarity matrix, is a user-defined scaling factor; S075, multi-head fusion; , Q, K, V are query, key, value respectively, MultiHead(Q, K, V) represents a multi-head attention output tensor, h is the number of attention heads, W o is an output linear projection matrix, and Concat is a tensor concatenation operation.

4. The fusion method suitable for the acoustic image long-distance recognition system according to claim 2, characterized in that: Before S01, a training step of the multi-head attention model is further included, in which original test data is acquired using an audiometer measurement, and the original test data is manually labeled to obtain a labeled data set, and the original test data is subjected to processes S01-S07 to obtain a fusion mapping function; based on an overall loss function, the difference J between the labeled data set and the fusion mapping function is calculated to be minimum, and the training of the multi-head attention model is completed.

5. The fusion method suitable for the acoustic image long-distance recognition system according to claim 4, characterized in that: The difference J is minimized using an Adam optimizer iteration.

6. The fusion method suitable for the acoustic image long-distance identification system according to claim 1, characterized in that: In S01, the audio encoder converts the original waveform into a frequency domain representation based on a Mel spectrogram to obtain audio features X audio; the visual encoder captures image spatial structure based on a convolutional neural network to obtain visual features X visual; The gait encoder processes the acceleration sequence based on a time series recurrent network to obtain gait features X gait.

7. The fusion method suitable for the acoustic image long distance recognition system according to claim 1, characterized in that: The distance measuring device is a laser range finder or an ultrasonic sensor. 8.An electronic device, comprising a processor and a memory; the processor is connected with the memory; The memory is used for storing executable program codes; the processor runs a program corresponding to the executable program codes by reading the executable program codes stored in the memory, so as to execute the method according to any one of claims 1-7.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the method according to any one of claims 1-7. The computer program is executed by the processor to implement the method according to any one of claims 1-7.

Images