Multi-modal data acquisition system and method thereof

Abstract

The invention discloses a multi-modal data acquisition system and a method thereof, and relates to the technical field of sensor data acquisition, and the method comprises the following steps: multi-source sensor basic data acquisition: an infrared camera acquires temperature distribution of a target surface or tissue; the millimeter-wave radar scans the internal structure of the organization; the optical fiber tactile sensor senses the acting force intensity in the contact process in real time; denoising processing is carried out on the collected multi-source data, and then the current environment state is judged through an environment state judgment method according to the collected infrared data, radar data and tactile data, and the current environment state comprises the environment risk level and the environment complexity. And based on the judged environment state, adaptively adjusting an acquisition strategy. According to the invention, the environmental risk level and complexity can be accurately identified in real time, data acquisition parameters are automatically adjusted, and the adaptive ability of data acquisition is improved; by dynamically adjusting the fusion weight of the infrared, radar and touch signals, it is guaranteed that acquisition results are more targeted and effective in different environments.

Description

Technical Field

[0001] The present invention relates to the technical field of sensor data acquisition, and in particular to a multimodal data acquisition system and method thereof. Background Art

[0002] In existing technologies, multimodal data acquisition is widely used in systems such as medical robots, industrial inspection robots, and hazardous environment inspection robots. By integrating multiple sensors, such as infrared, millimeter wave, and tactile sensors, multi-angle and multi-level perception of environmental conditions is enhanced. However, existing data acquisition methods often suffer from the following problems.

[0003] Lack of intelligent perception-feedback adaptive mechanism: Traditional solutions mostly use fixed data collection frequencies or collection strategies, which cannot be flexibly adjusted according to real-time environmental changes (such as complexity and risk level). This leads to low collection efficiency in low-risk environments and potential safety hazards in high-risk environments.

[0004] In response to the above technical problems, the present invention proposes a multimodal data acquisition method that integrates environmental state recognition + dynamic adaptive strategy + real-time feedback control. Summary of the Invention

[0005] The purpose of this section is to summarize some aspects of the embodiments of the present invention and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section and the abstract and title of this application to avoid obscuring the purpose of this section, the abstract and the title of the invention, and such simplifications or omissions should not be used to limit the scope of the present invention.

[0006] In view of the above problems in the prior art, the present invention is proposed.

[0007] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0008] Step 1: Multi-source sensor basic data collection, including: infrared camera to collect the temperature distribution of the target surface or tissue; millimeter wave radar to scan the internal structure of the tissue; fiber optic tactile sensor to sense the force intensity during contact in real time;

[0009] Step 2: De-noise the collected multi-source data, and then determine the current environmental status based on the collected infrared, radar and tactile data through the environmental status determination method, including: environmental risk level and environmental complexity.

[0010] Step 3: Dynamically and adaptively adjust the collection strategy based on the determined environmental status.

[0011] As a preferred embodiment of the multimodal data acquisition system and method described in the present invention, the basic data acquisition method of the multi-source sensors is as follows: within the time window [0, τ], continuous data acquisition is performed on the infrared, radar, and tactile sensors respectively to establish a time series signal, including:

[0012] Infrared signal I(t(: represents the temperature distribution data at time t;

[0013] Radar signal R(t): represents the echo intensity of the millimeter-wave radar at time t;

[0014] Tactile signal T(t): represents the contact force at time t;

[0015] Mathematical transformations are applied to the three time series signals to extract key features for risk perception, including: edge detection features f1(t) of infrared signals; dynamic change features f2(t) of radar signals; and nonlinear mapping features f3(t) of tactile signals.

[0016] As a preferred embodiment of the multimodal data acquisition system and method described in the present invention, the environmental state determination method includes the following steps:

[0017] S201: Perform weighted fusion on the three extracted characteristic signals to obtain the original intensity of environmental risk. The calculation formula is:

[0018] N(t)=α1·f1(t)+α2·f2(t)+α3·f1(t);

[0019] Among them, α1, α2, and α3 reflect the signal weights of infrared, radar, and tactile to environmental risks, respectively, and N(t) represents the original risk intensity signal;

[0020] S202: Based on the infrared signal I(t) and the radar signal R(t), an infrared-radar collaborative perception factor g1(t) is constructed. Based on the radar signal R(t) and the tactile signal T(t), a tactile-radar coupling normalization factor g2(t) is constructed. Then, the total complexity factor D(t) = λ1·g1(t) + λ2·g2(t) is calculated, where λ1 and λ2 are weight factors for collaborative perception and coupling normalization, respectively.

[0021] S203: Perform time domain integration based on N(t), perform square root normalization based on D(t), and comprehensively calculate to obtain the environmental health status index S; the calculation formula is:

[0022]

[0023] Here, γ represents a tuning parameter greater than 0 to prevent the denominator from approaching 0.

[0024] As a preferred solution of the multimodal data acquisition system and method described in the present invention, wherein: the item mapping is performed through the environmental health status index S to obtain the final risk level;

[0025] Among them, S>the first threshold indicates high risk; the second threshold ≤ S≤the first threshold indicates medium risk; S<the second threshold indicates low risk;

[0026] The S value is directly used as the complexity score. The higher the S is, the more complex the environment is.

[0027] As a preferred solution of the multimodal data acquisition system and method described in the present invention, wherein: the acquisition strategy adjustment method is to adjust the frequency F i ;

[0028]

[0029] Among them, F i represents the dynamic sampling frequency of the three types of sensors, i∈{I,R,T}; F i is the basic sampling frequency of the sensor, δ1, δ2>0; they are the sensitivity factors of environmental risk and environmental complexity to the sampling frequency, S1 and S2 are the quantitative values of environmental risk and environmental complexity based on the environmental health status index S.

[0030] As a preferred embodiment of the multimodal data acquisition system and method described in the present invention, the real-time collected and dynamically adjusted data are integrated to further detect and determine whether a safety threshold is exceeded. If so, a feedback mechanism is triggered to automatically adjust the operating behavior based on the current tactile, infrared, and radar integrated data. The specific method is as follows:

[0031] First, the signal weight α and the weight factor λ of cooperative sensing and coupling normalization are readjusted according to the real-time results of S1 and S2; the adjustment method is:

[0032] μ1,μ2>0: represent the dynamic adjustment factors of environmental risk and complexity on the weighting coefficient respectively;

[0033] Then, based on the adjusted parameters, the risk environment health status index S' for the next time window Δt is calculated and predicted in real time;

[0034] Finally, a dynamic safety threshold is set. If S' is greater than the threshold, the feedback mechanism is triggered.

[0035] As a preferred solution of the multimodal data acquisition system and method described in the present invention, the feedback mechanism is: by respectively calculating the risk contribution η of f1(t), f2(t), and f3(t) iTo determine the source of risk;

[0036] If η1 is the largest, the robot contact strength is automatically lowered to maintain stable contact and avoid damaging the thermal anomaly area;

[0037] If η2 is the largest, reduce the movement speed and increase the signal buffer time to prevent radar reflection interference;

[0038] If v3 is maximum, reduce the contact force or back off to avoid further damage to the high-force area.

[0039] An acquisition system applied to the above-mentioned multimodal data acquisition method includes:

[0040] The multi-source sensor acquisition module is responsible for the synchronous acquisition of multi-modal data; the data preprocessing and denoising module performs denoising, standardization, filtering and other processing on the original multi-source sensor data to provide cleaned data for subsequent recognition;

[0041] The environmental status recognition module determines the current environmental status based on the cleaned infrared, radar and tactile data and the environmental status determination method;

[0042] Adaptive acquisition strategy scheduling module, dynamically adjusts sensor acquisition strategy based on environmental status recognition results;

[0043] As well as the safety threshold detection and feedback control module, it compares the fused multimodal data with the safety threshold. If it exceeds the limit, it immediately activates the emergency feedback mechanism to dynamically adjust or interrupt the operation.

[0044] The present invention also discloses a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above-mentioned multimodal data acquisition system and method when executing the computer program.

[0045] The present invention also discloses a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the multimodal data acquisition system and method are implemented.

[0046] Beneficial effects of the present invention:

[0047] 1. The present invention can accurately identify the environmental risk level and complexity in real time, automatically adjust data collection parameters, and improve the adaptability of data collection. By dynamically adjusting the fusion weights of infrared, radar, and tactile signals, it ensures that the collection results in different environments are more targeted and effective.

[0048] 2. The present invention is real-time through the feedback mechanism: it can realize real-time control of operation behavior during acquisition, reduce the risk of target damage caused by environmental anomalies or acquisition errors, and improve system safety and stability. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] To more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. Those skilled in the art can also derive other drawings based on these drawings without inventive effort. Among them:

[0050] Figure 1 This is a schematic diagram of the overall structure of a multimodal data acquisition system and method proposed by the present invention. DETAILED DESCRIPTION

[0051] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0052] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0053] Secondly, the term "one embodiment" or "embodiment" herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in various places throughout this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive of other embodiments.

[0054] Example 1

[0055] Reference Figure 1 , as an embodiment of the present invention, provides a multimodal data acquisition method, the method comprising the following steps:

[0056] Step 1: Basic data collection by multi-source sensors, including: infrared camera to collect temperature distribution of target surface or tissue; millimeter-wave radar to scan internal structure of tissue; fiber optic tactile sensor to perceive force intensity during contact in real time.

[0057] Specifically, the basic data collection method of multi-source sensors is as follows: within the time window [0, τ], continuous data is collected from infrared, radar, and tactile sensors respectively, and a time series signal is established as the subsequent basic data input, including:

[0058] Infrared signal I(t): represents the temperature distribution data at time t;

[0059] Radar signal R(t): represents the echo intensity of the millimeter-wave radar at time t;

[0060] Tactile signal T(t): represents the contact force at time t.

[0061] Mathematical transformations are applied to the three time series signals to extract key features for risk perception, including: edge detection features f1(t) of infrared signals; dynamic change features f2(t) of radar signals; and nonlinear mapping features f3(t) of tactile signals.

[0062] It is further explained that the edge detection feature f1(t) of the infrared signal means the temperature mutation area of the Laplacian operator detection heat map, which usually represents the tissue edge or risk area;

[0063] in Represents the Laplacian operator.

[0064] The dynamic change characteristic f2(t) of the radar signal is the time rate of change of the radar echo intensity, which reflects the dynamic characteristics of the tissue surface or deep structure;

[0065]

[0066] The nonlinear mapping feature f3(t) of the tactile signal means that the tactile signal is mapped through an exponential mapping.

[0067] f3(t)=e -β·T(t) ; β represents the sensitivity factor of controlling tactile signals.

[0068] Step 2: De-noise the collected multi-source data, and then determine the current environmental status based on the collected infrared, radar and tactile data through the environmental status determination method, including: environmental risk level and environmental complexity.

[0069] Specifically, the environmental status determination method includes the following steps:

[0070] S201: Perform weighted fusion on the three extracted characteristic signals to obtain the original intensity of environmental risk. The calculation formula is:

[0071] N(t)=α1·f1(t)+α2·f2(t)+α3·f3(t);

[0072] Among them, α1, α2, and α3 reflect the signal weights of infrared, radar, and tactile to environmental risks, respectively, and N(t) represents the original risk intensity signal;

[0073] S202: Based on the infrared signal I(t) and the radar signal R(t), an infrared-radar collaborative perception factor g1(t) is constructed. g1(t) = log(1 + I(t) · R(t)). This factor indicates a potential high-risk area when infrared anomalies (high temperatures) are synergistically enhanced with the radar signal.

[0074] Based on the radar signal R(t) and the tactile signal T(t), the tactile-radar coupling normalization factor g2(t) is constructed;

[0075]

[0076] This means that high tactile pressure is suppressed under high radar reflection, preventing excessive amplification of force signals under hard tissue.

[0077] Then the total complexity factor is calculated: D(t) = λ1·g1(t) + λ2·g2(t), where λ1 and λ2 are the weight factors of collaborative perception and coupling normalization, respectively.

[0078] S203: Perform time domain integration based on N(t), perform square root normalization based on D(t), and comprehensively calculate to obtain the environmental health status index S; the calculation formula is:

[0079]

[0080] Here, γ represents a tuning parameter greater than 0 to prevent the denominator from approaching 0.

[0081] Mapping is performed through the environmental health status index S to obtain the final risk level;

[0082] Among them, S>the first threshold indicates high risk; the second threshold ≤ S ≤ the first threshold indicates medium risk; S<the second threshold indicates low risk.

[0083] Step 3: Based on the determined environmental state, dynamically and adaptively adjust the acquisition strategy. The acquisition strategy adjustment method is to adjust the frequency F i ;

[0084]

[0085] Among them, F i represents the dynamic sampling frequency of the three types of sensors, i∈{I,R,T}; F i is the basic sampling frequency of the sensor, δ1, δ2>0; are the sensitivity factors of environmental risk and environmental complexity to the sampling frequency, respectively, S1 and S2 correspond to the quantitative values of environmental risk and environmental complexity, respectively. According to different levels of risk, S1 is assigned different floating coefficients, such as using nonlinear weights;

[0086] That is, the quantized value of S1 can be as follows:

[0087] <![CDATA[S1 level]]> Low risk Medium risk High risk <![CDATA[Quantization value of S1]]> 0.2 0.6 1.0

[0088] The S value is directly used as the complexity quantification value score S2. The higher S is, the more complex the environment is.

[0089] Example 2

[0090] The difference between Example 2 and Example 1 is that the real-time collected and dynamically adjusted data are integrated to further detect and determine whether a safety threshold is exceeded. If so, a feedback mechanism is triggered to automatically adjust the operating behavior based on the current tactile, infrared, and radar integrated data. The specific method is as follows:

[0091] First, the signal weight α and the weight factor λ of cooperative sensing and coupling normalization are readjusted according to the real-time results of S1 and S2; the adjustment method is:

[0092] μ1,μ2>0: dynamic adjustment factors of environmental risk and complexity on weighting coefficients;

[0093] Then, based on the adjusted parameters, the risk environment health status index S' for the next time window Δt is calculated and predicted in real time;

[0094] Finally, a dynamic safety threshold is set. If S' is greater than the threshold, the feedback mechanism is triggered.

[0095] The feedback mechanism is: by calculating the risk contribution η of f1(t), f2(t), and f3(t) respectively i To determine the source of risk;

[0096] If η1 is the largest, it means that the "temperature risk factor" is dominant, indicating that the robot (e.g., medical rehabilitation robot) is in an environment with abnormally high temperature or severe temperature fluctuations (e.g., hot spots or temperature abnormality areas are detected in the infrared data). In this case, the contact strength of the medical robot is automatically reduced to maintain stable contact and avoid damaging the thermal abnormality area.

[0097] If η2 is the largest, it means that abnormal reflection signals appear in the environmental features detected by the millimeter-wave radar. In this case, the movement speed is reduced and the signal buffer time is increased to prevent radar reflection interference.

[0098] If η3 is the largest, it means that the “tactile force risk factor” is dominant, which means that the robot’s tactile sense detects that the contact force is too large or the instantaneous impact is strong (for example, when contacting a hard object or a pressure-sensitive area), so the contact force is reduced or retreated to avoid further damage to the high-force area.

[0099] In summary, this invention can accurately identify environmental risk levels and complexity in real time, automatically adjust data collection parameters, and enhance the adaptive capabilities of data collection. By dynamically adjusting the fusion weights of infrared, radar, and tactile signals, it ensures that data collection results are more targeted and effective in different environments. Through real-time feedback mechanisms, operational behavior can be controlled in real time during data collection, reducing the risk of target damage caused by environmental anomalies or data collection errors, and improving system security and stability.

[0100] This embodiment also discloses a multimodal data acquisition system, which includes:

[0101] The multi-source sensor acquisition module is responsible for the synchronous acquisition of multi-modal data; the data preprocessing and denoising module performs denoising, standardization, filtering and other processing on the original multi-source sensor data to provide cleaned data for subsequent identification; the environmental state recognition module judges the current environmental state based on the cleaned infrared, radar and tactile data with the help of the environmental state determination method; the adaptive acquisition strategy scheduling module dynamically adjusts the sensor acquisition strategy according to the environmental state identification results; and the safety threshold detection and feedback control module compares the fused multi-modal data with the safety threshold. If the limit is exceeded, the emergency feedback mechanism is immediately activated to dynamically adjust or interrupt the operation.

[0102] This embodiment also provides a computer device suitable for a multimodal data acquisition system and method thereof, comprising: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute computer-executable instructions to implement a multimodal data acquisition system and method thereof as proposed in the above embodiment.

[0103] The computer device may be a terminal, comprising a processor, a memory, a communication interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device comprises a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner may be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. The display screen of the computer device may be a liquid crystal display or an electronic ink display screen, and the input device of the computer device may be a touch layer covering the display screen, or a button, trackball or touchpad provided on the housing of the computer device, or an external keyboard, touchpad or mouse.

[0104] This embodiment also provides a storage medium having a computer program stored thereon, which, when executed by a processor, implements a multimodal data acquisition system and method thereof as proposed in the above embodiment; the storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0105] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A multimodal data acquisition method, characterized in that: The method comprises the following steps: Step 1: Multi-source sensor basic data collection, including: infrared camera to collect the temperature distribution of the target surface or tissue; millimeter wave radar to scan the internal structure of the tissue; fiber optic tactile sensor to sense the force intensity during contact in real time; Step 2: De-noise the collected multi-source data, and then use the environmental state determination method to determine the current environmental state, including environmental risk level and environmental complexity, based on the collected infrared, radar, and tactile data. Step 3: Dynamically and adaptively adjust the collection strategy based on the determined environmental status.

2. The multimodal data acquisition method according to claim 1, wherein: The basic data collection method of multi-source sensors is: within the time window [0,τ], continuously collect data from infrared, radar, and tactile sensors respectively to establish a time series signal, including: Infrared signal I(t(: represents the temperature distribution data at time t; Radar signal R(t): represents the echo intensity of the millimeter-wave radar at time t; Tactile signal T(t): represents the contact force at time t; Mathematical transformations are applied to the three time series signals to extract key features for risk perception, including: edge detection features f1(t) of infrared signals; dynamic change features f2(t) of radar signals; and nonlinear mapping features f3(t) of tactile signals.

3. The multimodal data acquisition method according to claim 2, wherein: The environmental status determination method comprises the following steps: S201: Perform weighted fusion on the three extracted characteristic signals to obtain the original intensity of environmental risk. The calculation formula is: N(t)=α1·f1(t)+α2·f2(t)+α3·f1(t); Among them, α1, α2, and α3 reflect the signal weights of infrared, radar, and tactile to environmental risks, respectively, and N(t) represents the original risk intensity signal; S202: Based on the infrared signal I(t) and the radar signal R(t), an infrared-radar collaborative perception factor g1(t) is constructed. Based on the radar signal R(t) and the tactile signal T(t), a tactile-radar coupling normalization factor g2(t) is constructed. Then, the total complexity factor D(t) = λ1·g1(t) + λ2·g2(t) is calculated, where λ1 and λ2 are weight factors for collaborative perception and coupling normalization, respectively. S203: Perform time domain integration based on N(t), perform square root normalization based on D(t), and comprehensively calculate to obtain the environmental health status index S; the calculation formula is: Here, γ represents a tuning parameter greater than 0 to prevent the denominator from approaching 0.

4. The multimodal data acquisition method according to claim 3, wherein: The said picking up and mapping is performed through the environmental health status index S to obtain the final risk level; Among them, S>the first threshold indicates high risk; the second threshold ≤ S≤the first threshold indicates medium risk; S<the second threshold indicates low risk; The S value is directly used as the complexity score. The higher the S is, the more complex the environment is.

5. The multimodal data acquisition method according to claim 4, characterized in that: The acquisition strategy is adjusted by adjusting the frequency F i ; Among them, F i represents the dynamic sampling frequency of the three types of sensors, i∈{I,R,T}; F i is the basic sampling frequency of the sensor, δ1, δ2>0; are the sensitivity factors of environmental risk and environmental complexity to the sampling frequency, respectively; S1 and S2 are the quantitative values of environmental risk and environmental complexity corresponding to the environmental health status index S, respectively.

6. The multimodal data acquisition method according to claim 5, characterized in that: The real-time collected and dynamically adjusted data are integrated to further detect and determine whether a safety threshold is exceeded. If so, a feedback mechanism is triggered to automatically adjust the operating behavior based on the current comprehensive data of tactile, infrared, and radar. The specific method is as follows: First, the signal weight α and the weight factor λ of cooperative sensing and coupling normalization are readjusted according to the real-time results of S1 and S2; the adjustment method is: μ1,μ2>0: represent the dynamic adjustment factors of environmental risk and complexity on the weighting coefficient respectively; Then, based on the adjusted parameters, the risk environment health status index S' for the next time window Δt is calculated and predicted in real time; Finally, a dynamic safety threshold is set. If S' is greater than the threshold, the feedback mechanism is triggered.

7. The multimodal data acquisition method according to claim 6, characterized in that: The feedback mechanism is: by calculating the risk contribution η of f1(t), f2(t), and f3(t) respectively i To determine the source of risk; If η1 is the largest, the robot contact strength is automatically lowered to maintain stable contact and avoid damaging the thermal anomaly area; If η2 is the largest, reduce the movement speed and increase the signal buffer time to prevent radar reflection interference; If η3 is maximum, reduce the contact force or back off to avoid further damage to the high-force area.

8. A multimodal data acquisition system, applied to the multimodal data acquisition method according to any one of claims 1 to 7, characterized in that: The system includes: The multi-source sensor acquisition module is responsible for the synchronous acquisition of multi-modal data; the data preprocessing and denoising module performs denoising, standardization, filtering and other processing on the original multi-source sensor data to provide cleaned data for subsequent recognition; The environmental status recognition module determines the current environmental status based on the cleaned infrared, radar and tactile data and the environmental status determination method; Adaptive acquisition strategy scheduling module, dynamically adjusts sensor acquisition strategy based on environmental status recognition results; As well as the safety threshold detection and feedback control module, it compares the fused multimodal data with the safety threshold. If it exceeds the limit, it immediately activates the emergency feedback mechanism to dynamically adjust or interrupt the operation.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the multimodal data acquisition method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the multimodal data acquisition method according to any one of claims 1 to 7 are implemented.

Images