Method and system for controlling visual fusion of small brain AI of embodied agent

Abstract

The application belongs to the technical field of intelligent control, and discloses a big-small brain AI vision fusion control method and system of embodied intelligent agent; the method comprises the following steps: collecting multi-modal data of the embodied intelligent agent and analyzing the multi-modal data, extracting dominant frequencies of each mode, obtaining a dangerous frequency set through analysis, performing harmonic decomposition on each frequency, and constructing a complete frequency profile; taking the dangerous frequency set and the complete frequency profile as constraints, optimizing vision acquisition parameters of the intelligent agent, and simultaneously generating a cognitive flow parameter set and a reflex flow parameter set; constructing a big-small brain double path according to the cognitive flow parameter set and the reflex flow parameter set, and analyzing to obtain a threat score and an abnormal area; analyzing the activity intensity of the abnormal area, when the activity intensity is greater than an intensity threshold, constructing a reflex threat map according to the threat score and the abnormal area, analyzing the reflex threat map, and calculating a real-time obstacle avoidance direction and an obstacle avoidance speed; the application significantly improves the perception reliability and obstacle avoidance safety of the embodied intelligent agent in a high-risk scene.

Description

Technical Field

[0001] This invention relates to the field of intelligent control technology, and more specifically, to a method and system for controlling the cerebral and cerebellar visual fusion of an embodied intelligent agent. Background Technology

[0002] In core application scenarios of embodied intelligent agents, such as industrial inspection, marine engineering equipment operation, and autonomous movement in high-risk environments, dangerous rotating machinery such as motors, gears, and propellers are commonly present. The high-speed periodic motion of this type of machinery is not only a potential source of collision hazards, but it can also have adverse effects when its characteristic frequency is in sync with the acquisition parameters autonomously adjusted by the intelligent agent's visual system. Current embodied intelligent agents widely employ cerebellar-cerebrum visual fusion control systems, which achieve complementary perception through high-precision semantic cognition of the cerebrum pathway and rapid motion reflexes of the cerebellum pathway, can trigger visual resonance in complex dynamic environments when the frequency corresponding to the system's autonomously optimized frame rate or shutter speed is close to the fundamental frequency and its Nth harmonic frequency of the dangerous rotating machinery. This resonance causes the high-speed rotating machinery to appear stationary or moving slowly in the acquired images. Essentially, the synchronization effect between the sampling frequency and the object's motion frequency causes the posture of the rotating object to highly overlap in each frame, thus masking its true high-speed motion. Since both the cerebellum and cerebellum pathways rely on visual data for decision-making, this illusion of stillness can deceive both perceptual pathways simultaneously: the cerebellum pathway's target detection and 3D velocity estimation may mistakenly classify high-speed rotating objects as static or low-dynamic targets, resulting in incorrect semantic classifications and threat levels; the cerebellum pathway's intensive optical flow calculations and abnormal motion detection may also fail to capture the true characteristics of high-speed motion, causing reflexive obstacle avoidance commands to fail. Ultimately, this leads to misjudgments by the entire fusion control system, resulting in the loss of effective early warning and obstacle avoidance capabilities for dangerous rotating machinery, which can easily cause safety accidents such as equipment collisions and mechanical jamming.

[0003] In view of this, the present invention proposes a brain-brain fusion control method and system for embodied intelligent agents to solve the above problems. Summary of the Invention

[0004] To overcome the aforementioned deficiencies of the prior art and to achieve the above objectives, the present invention provides the following technical solution: a method for controlling the cerebellum and cerebrum-based AI visual fusion of embodied intelligent agents, comprising:

[0005] Collect and analyze multimodal data of embodied intelligent agents, extract the dominant frequencies of each mode, perform parallel screening of the dominant frequencies of each mode to obtain a set of dangerous frequencies, perform harmonic decomposition on each frequency in the set of dangerous frequencies, and construct a complete frequency profile.

[0006] Constrained by a dangerous frequency set and a complete frequency profile, the visual acquisition parameters of the embodied intelligent agent are optimized, and a set of cognitive flow parameters controlling the brain and a set of reflex flow parameters controlling the cerebellum are generated.

[0007] Based on the set of cognitive flow parameters controlling the brain and the set of reflex flow parameters controlling the cerebellum, the brain and cerebellum are respectively controlled to collect corresponding images. The corresponding images collected by the brain and cerebellum are analyzed to obtain threat scores and abnormal regions.

[0008] Analyze the activity intensity in abnormal areas. When the activity intensity exceeds the intensity threshold, construct a reflected threat map based on the threat score and the abnormal area. Analyze the threat centroid and global threat intensity of the reflected threat map. Calculate the real-time obstacle avoidance direction and speed based on the threat centroid and global threat intensity.

[0009] Furthermore, methods for obtaining a set of cognitive flow parameters that control the brain include:

[0010] Based on the dangerous frequencies marked as harmonics in the complete frequency profile, and combined with dynamic safety thresholds, a safe no-go zone for controlling the frame rate and shutter speed of the brain is set; wherein, the dynamic safety threshold is adjusted according to the dangerous frequencies;

[0011] Calculate the product of the speed coefficient and the relative speed, where the relative speed is the relative speed between the embodied intelligent agent and the hazard source. Calculate the sum of the product of the basic threshold and the speed to obtain the dynamically adjusted safety threshold.

[0012] By removing the safe no-go zones of frame rate and shutter speed from the pre-calibrated feasible region of the visual parameters controlled by the camera, a safe and feasible parameter space is obtained. Within this safe and feasible parameter space, [the system] generates [the appropriate parameters]. A set of visual parameters is used to acquire images and calculate the overall image quality score of each set of visual parameters. Based on the overall image quality score, optimized visual parameters are selected and used as the cognitive flow parameter set. This represents the number of visual parameter combinations; visual parameters include frame rate, shutter speed, aperture, and gain.

[0013] Furthermore, methods for selecting optimized visual parameters include:

[0014] Images are acquired using each set of visual parameter combinations, and the image quality loss value of the images is calculated; the initial sample set is obtained by concatenating the visual parameter combinations and the corresponding image quality loss values.

[0015] Generate a combination of basic visual parameters within a safe and feasible parameter space;

[0016] The image quality comprehensive score is calculated based on the combination of basic visual parameters. Within the safe and feasible parameter space, V candidate visual parameters are generated and combined with the basic visual parameter combination as the center to form a candidate visual parameter combination set. The image quality comprehensive score of each candidate visual parameter combination is calculated. The visual parameter corresponding to the candidate visual parameter combination with the highest image quality comprehensive score is selected as the cognitive flow parameter set; V is the number of candidate visual parameters.

[0017] Furthermore, methods for calculating the overall image quality score include:

[0018] The Sobel operator is used to detect edges in the image to obtain the gradients in the X and Y directions. The gradient magnitudes are calculated, the average gradient magnitude of the image is statistically analyzed, and normalization is performed to obtain the sharpness.

[0019] Calculate the gray level difference and gray level sum of the maximum and minimum gray levels of the image, respectively. Calculate the ratio of the gray level difference to the gray level sum to obtain the actual contrast ratio and perform normalization processing to obtain the contrast ratio.

[0020] In the background region of the image, the variance of the pixel gray value is extracted and calculated by threshold segmentation. Then, it is divided by the mean gray value of the background region to obtain the image noise intensity and normalization processing to obtain the noise level.

[0021] Calculate the average grayscale value of the image, calculate the absolute value of the difference between the average grayscale value and the ideal grayscale value, and calculate the ratio of the absolute value to the ideal grayscale value to obtain a first level value; calculate the sum of the number of overexposed pixels and the number of underexposed pixels, and then calculate the exposure ratio of the sum of the pixel counts to the total number of pixels in the image to obtain a second level value; then calculate the sum of the first level value and the second level value to obtain the exposure level; wherein, the number of overexposed pixels is the number of pixels with a grayscale value not lower than the overexposure threshold, and the number of underexposed pixels is the number of pixels with a grayscale value not greater than the underexposure threshold;

[0022] A comprehensive image quality score is obtained by weighting sharpness, contrast, noise level, and exposure level.

[0023] Furthermore, methods for obtaining threat scores include:

[0024] A cognitive flow parameter set was used to control the acquisition of standard images of the brain.

[0025] The standard image is used as input to the pre-trained object detection model to obtain the initial detection result and the corresponding detection confidence. The initial detection result includes the object bounding box and the object's category label. The initial detection results with a detection confidence of not less than the confidence detection threshold are retained to obtain the object detection result.

[0026] By combining camera intrinsics and visual odometry, the position of each object in the world coordinate system is calculated;

[0027] Calculate the distance between the embodied intelligent agent's own world coordinates and the object's position to obtain distance features;

[0028] Calculate the distance ratio between the distance feature and the preset safe distance of the object, calculate the distance difference between the value 1 and the distance ratio, and select the smaller value between the distance difference and the value 1 as the distance threat factor;

[0029] Calculate the ratio of the object's current speed to the preset maximum safe speed, and select the smaller of the speed ratio and the value 1 as the speed threat factor;

[0030] Based on the object category, obtain the preset category threat value as the category threat factor;

[0031] The threat score is obtained by weighting the distance threat factor, speed threat factor, and category threat factor.

[0032] Furthermore, methods for analyzing the activity intensity of anomalous regions include:

[0033] The average optical flow amplitude of each anomalous region is calculated, and the ratio of the average optical flow amplitude of each anomalous region to the maximum average optical flow amplitude of all anomalous regions is calculated to obtain the activity intensity of the anomalous region.

[0034] Furthermore, methods for constructing reflection threat maps include:

[0035] Create a grid map with the same size as the embodied agent's current viewpoint image, with an initial value of 0;

[0036] For each detected object, calculate the cumulative threat value within the object's L×L pixel area based on the object's location and threat score;

[0037] Calculate the ratio of each cumulative threat value in the grid map to the maximum cumulative threat value in the grid map to obtain a normalized reflection threat map.

[0038] Furthermore, methods for obtaining the set of danger frequencies include:

[0039] Parallel visual dominant frequencies and inertial dominant frequencies are used to obtain the frequency set to be evaluated;

[0040] Calculate the visual signal-to-noise ratio based on the complex spectrum corresponding to the dominant visual frequency;

[0041] Calculate the root mean square of the dominant inertial frequency in the inertial spectrum to obtain the amplitude of the synthesized signal;

[0042] Calculate the composite vibration amplitude of the triaxial acceleration data, perform a fast Fourier transform on the composite vibration amplitude to obtain the composite spectrum, and calculate the inertial signal-to-noise ratio of the composite spectrum.

[0043] The overall signal-to-noise ratio is obtained by weighting the visual signal-to-noise ratio and the inertial signal-to-noise ratio.

[0044] The total sampling time is divided into Nf consecutive sub-windows, each with an equal duration;

[0045] For each sub-window, a consistency rule check is performed; if the consistency rule is satisfied, the corresponding sub-window's check result is 1, otherwise the check result is 0; the ratio of the total number of check results of 1 in Nf sub-windows to Nf is calculated to obtain the time consistency.

[0046] If neither the visual dominant frequency nor the inertial dominant frequency is 0, then the cross-modal consistency is the first modal value; if either the visual dominant frequency or the inertial dominant frequency is 0, then the cross-modal consistency is the second modal value.

[0047] The confidence level of each frequency to be evaluated is obtained by weighting the overall signal-to-noise ratio, time consistency, and cross-modal consistency. Only frequencies with confidence levels exceeding the confidence threshold are retained to form a set of dangerous frequencies.

[0048] Furthermore, methods for constructing a complete frequency profile include:

[0049] Traverse the set of dangerous frequencies and filter them based on deduplication rules to obtain the filtering frequencies;

[0050] Sort the selection frequencies in ascending order to obtain an ordered set;

[0051] Arrange each danger frequency in ascending order. As a potential fundamental frequency, it detects whether a corresponding dangerous frequency exists in an ordered set. If it meets the harmonic filtering rules, then mark it. The frequency is marked as the fundamental frequency, and the marking is... If the frequency is not specified, mark it as a harmonic; otherwise, mark the frequency as 0.

[0052] Iterate through each dangerous frequency in the ordered set, mark the frequency label and confidence level of all dangerous frequencies in the ordered set, and for dangerous frequencies marked as fundamental frequency or harmonic, mark the harmonics corresponding to the fundamental frequency and the fundamental frequency corresponding to the harmonics to obtain a complete frequency profile.

[0053] A brain-brain fusion control system for an embodied intelligent agent, comprising the following methods for implementing brain-brain fusion control of the embodied intelligent agent:

[0054] Hazard Analysis Module: Collects and analyzes multimodal data of the embodied intelligent agent, extracts the dominant frequencies of each mode, performs parallel screening of the dominant frequencies of each mode to obtain a set of hazardous frequencies, performs harmonic decomposition on each frequency in the set of hazardous frequencies, and constructs a complete frequency profile;

[0055] Parameter Analysis Module: Based on the dangerous frequency set and the complete frequency profile, optimize the visual acquisition parameters of the embodied intelligent agent, and generate the cognitive flow parameter set controlling the brain and the reflex flow parameter set controlling the cerebellum;

[0056] Path analysis module: Based on the set of cognitive flow parameters controlling the brain and the set of reflex flow parameters controlling the cerebellum, the brain and cerebellum are respectively controlled to collect corresponding images, and the corresponding images collected by the brain and cerebellum are analyzed to obtain threat scores and abnormal regions.

[0057] Fusion control module: Analyzes the activity intensity in abnormal areas. When the activity intensity is greater than the intensity threshold, it constructs a reflected threat map based on the threat score and the abnormal area, analyzes the threat centroid and global threat intensity of the reflected threat map, and calculates the real-time obstacle avoidance direction and obstacle avoidance speed based on the threat centroid and global threat intensity.

[0058] The technical effects and advantages of the cerebellar-brain-visual fusion control method and system for the embodied intelligent agent of this invention are as follows:

[0059] This invention extracts the dominant frequency of dangerous rotating machinery through dual-modal co-sensing of vision and IMU and constructs a complete fundamental frequency and harmonic frequency profile. Based on this, it accurately delineates frame rate / shutter speed safety zones and dynamically optimizes the visual parameters of cognitive flow and reflection flow. This avoids the illusion of stillness of high-speed rotating objects caused by resonance from the source, ensuring that the cerebral-brain visual fusion control system obtains real and distortion-free image input. Subsequently, through precise target detection, 3D positioning, and multi-dimensional threat factor weighted calculation, it achieves accurate quantification of the threat score of dangerous objects. Combined with dual-modal abnormal area detection of reflection flow optical flow and IMU vibration, reflection threat map construction, and obstacle avoidance decision-making, it forms a full-process protection, effectively preventing the cerebral-brain system from being deceived at the same time. This significantly improves the perception reliability and obstacle avoidance safety of the embodied intelligent agent for dangerous rotating machinery in high-risk scenarios such as industrial inspection, and reduces the risk of safety accidents such as equipment collisions and personnel casualties. Attached Figure Description

[0060] Figure 1 This is a schematic diagram of the process of the cerebellar-brain-visual fusion control method for the embodied intelligent agent of the present invention;

[0061] Figure 2 This is a schematic diagram of the method for obtaining a set of cognitive flow parameters for controlling the brain according to the present invention;

[0062] Figure 3 This is a schematic diagram of the method for obtaining threat scores according to the present invention;

[0063] Figure 4 This is a schematic diagram of the cerebellar-brain-visual fusion control system structure of the embodied intelligent agent of the present invention. Detailed Implementation

[0064] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0065] Example 1:

[0066] Please see Figure 1 As shown, this embodiment provides a method for controlling the cerebral and cerebellar AI visual fusion of an embodied intelligent agent, including:

[0067] Multimodal data of the embodied intelligent agent is collected and analyzed. The dominant frequencies of each modality are extracted, and the dominant frequencies of each modality are screened in parallel to obtain a set of dangerous frequencies. Harmonic decomposition is performed on each frequency in the set of dangerous frequencies to construct a complete frequency profile. The multimodal data includes visual data streams and triaxial acceleration data.

[0068] Methods for extracting the dominant frequencies of each mode include:

[0069] Triaxial acceleration data is acquired by IMU, and a fast Fourier transform is performed on the triaxial acceleration data to obtain the inertial spectrum. The triaxial spectral energy is synthesized, and the frequency components within the target frequency band with spectral energy greater than the energy threshold are extracted as the dominant inertial frequency. If none exists, the dominant inertial frequency is marked as 0. The target frequency band is preferably 0-500Hz. The energy threshold is set according to the arithmetic mean of the energy values ​​of all frequency points within the current target frequency band, preferably 3-5 times the arithmetic mean of the energy values ​​of all frequency points within the target frequency band. Using IMU to directly capture the physical frequency of mechanical vibration is completely unaffected by the visual resonance stillness illusion. It is a core redundant perception method to avoid the simultaneous deception of the cerebellum and cerebellum systems, realize cross-modal cross-verification of dangerous frequencies, and ensure the reliable extraction of dangerous features.

[0070] Visual data streams for a preset time period are acquired using the camera's highest frame rate to delineate the region of interest. The highest frame rate is not equal to the dominant inertial frequency. The preset time period is set according to the actual situation. By setting the highest frame rate to be different from the dominant inertial frequency, the probability of resonance between the visual sampling frequency and the frequency of dangerous rotating machinery is reduced from the source. At the same time, the high frame rate ensures temporal resolution, providing sufficient grayscale change details for subsequent frequency domain analysis and reducing the risk of loss of motion features caused by resonance.

[0071] Methods for defining regions of interest include:

[0072] The grayscale difference between two consecutive frames of visual data streams is calculated, and the cumulative difference value is obtained by summing the grayscale differences within the calculation period. The cumulative difference value is then normalized to obtain the inter-frame difference cumulative map.

[0073] The visual data stream of two consecutive frames is divided into blocks of size G×G, where G is preferably 8-16. For each block, the pixel grayscale within the window is calculated based on E×E, where E is preferably 3-5. The linear motion model is then solved using the least squares method. ,in, For pixels in the visual data stream from Exercise The resulting grayscale value; For visual data stream The pixel grayscale value at pixel coordinates; For pixels in Displacement in the direction of motion; For pixels in Displacement in the direction of motion; for pixel position Gray-scale gradient in direction; for pixel position Gray-level gradient in direction; for each pixel Calculate pixels in Directional displacement and The root mean square of the motion displacement in the direction is used as the optical flow amplitude; the cumulative optical flow amplitude is obtained by summing the optical flow amplitudes within the calculation period; the cumulative optical flow amplitude is normalized to obtain a dense optical flow amplitude accumulation map; wherein, the calculation period is preferably 5-10 minutes.

[0074] The initial motion saliency map is obtained by adding the inter-frame difference cumulative map and the dense optical flow amplitude cumulative map.

[0075] Regions larger than an adaptive threshold in the initial motion saliency map are selected as candidate regions, and those smaller than the area threshold are removed. This yields a set of regions of interest. The area threshold is set based on the minimum effective target size of the image. The adaptive threshold is obtained based on historical data statistics, calculated by adding a factor of g (1.0-1.5) to the mean grayscale value and the standard deviation of the initial motion saliency map. The ratio of the total area of ​​the candidate regions to the total area of ​​the visual data stream is adjusted; if the ratio is greater than 30%, g is increased; if the ratio is less than 5%, g is decreased. Combining inter-frame grayscale changes and optical flow amplitude as dual visual motion features, potential danger zones are accurately located, reducing background interference.

[0076] The average grayscale value time series of the region of interest is obtained. The average grayscale value time series of the region of interest is detected by sliding a window with a step size s. A fast Fourier transform is performed to obtain a complex spectrum. The frequency index of the complex spectrum is combined with the frequency resolution of the sliding window to convert it into the actual frequency. The periodic features of grayscale changes are directly mined through frequency domain analysis, which can overcome the time domain limitations of visual stillness and extract the true frequency of rotating machinery from the grayscale period.

[0077] Reserve target frequency band The actual frequency within the range is used to obtain the filtered amplitude and corresponding actual frequency; and These are the maximum and minimum frequency values ​​for the target frequency band, respectively; the design is based on industry experience values, such as the target frequency band for the gearbox being 50Hz-200Hz.

[0078] For each sliding window, frequency components with spectral amplitudes greater than the previous and next frequency components, and whose spectral amplitudes are greater than the spectral threshold, are selected as the initial dominant frequency within the sliding window. If no such frequency exists, the initial dominant frequency is marked as 0 and used as the visual dominant frequency. The spectral threshold is set based on the arithmetic mean of the spectral amplitudes of all frequency points within the current target frequency band, preferably 3-5 times the arithmetic mean of the spectral amplitudes of all frequency points within the target frequency band.

[0079] By using frequency domain analysis of time series, we can overcome the time domain limitations of the visual stillness illusion and directly extract the true characteristic frequencies of rotating machinery from the gray-scale change cycle, thus avoiding being deceived by the appearance of overlapping poses between frames.

[0080] Methods for obtaining a set of dangerous frequencies include:

[0081] Parallel connection of visual and inertial dominant frequencies yields a set of frequencies to be evaluated. Integrating the dual-modal frequencies to be evaluated provides a basis for multi-dimensional verification of dangerous frequencies. Furthermore, the redundant input of the inertial modality reduces the risk of missed detection of dangerous frequencies due to visual failure caused by resonant static artifacts.

[0082] Visual signal-to-noise ratio (SNR) is calculated based on the complex spectrum corresponding to the dominant visual frequency: The peak amplitude of the dominant visual frequency in the corresponding complex spectrum is extracted; in the complex spectrum, the median of all amplitudes within the target frequency band is calculated, and all candidate peak points greater than k times the median are found to form an initial candidate set. All candidate peak points and their adjacent frequency points (B points each) are removed from the complex spectrum to form a guard band. In the remaining spectrum data after removing the guard band, the average of all frequency amplitudes is calculated to obtain the noise amplitude. The ratio of the peak amplitude to the noise amplitude is calculated to obtain the original visual SNR; the original visual SNR is normalized to obtain the final visual SNR; where k is a multiple of the median of all amplitudes, and B is the number of frequency points. This quantifies the signal reliability of the dominant visual frequency and filters out false detection frequencies caused by visual noise. However, there are logical limitations: when visual resonance causes a static illusion, the peak of the visual spectrum may be masked, and this step may not be able to effectively calculate a reliable visual SNR, or even misjudge signals of truly dangerous frequencies as noise.

[0083] Calculate the root mean square of the dominant inertial frequency in the inertial spectrum to obtain the amplitude of the synthesized signal.

[0084] The composite vibration amplitude of triaxial acceleration data can be calculated by calculating the root mean square of the triaxial acceleration data. A fast Fourier transform is then performed on the composite vibration amplitude to obtain the composite spectrum. The calculation steps for the inertial signal-to-noise ratio of the composite spectrum are the same as those for calculating the visual signal-to-noise ratio, and will not be repeated here. The signal reliability quantification method, which is independent of vision and is not affected by the visual resonance stillness artifact, can stably extract the vibration frequency signal quality of dangerous rotating machinery, and is a key support for addressing core technical issues.

[0085] The overall signal-to-noise ratio (SNR) is obtained by weighting the visual SNR and the inertial SNR. The preferred weighting weight for the visual SNR is 0.3, and the preferred weighting weight for the inertial SNR is 0.7. Since visual data is susceptible to resonance interference and inertial data is more reliable, inertial data is given a higher weight. The specific weighting values ​​can be adjusted according to the actual situation. By fusing the signal quality characteristics of the two modes, the risk of misjudgment caused by resonance failure of a single visual mode is reduced.

[0086] The total sampling time is divided into Nf consecutive sub-windows, each with an equal duration; consistency rule checks are performed on each sub-window, including:

[0087] Condition 1: Obtain the visual spectrum by performing a Fast Fourier Transform on the visual grayscale time series, and detect the visual spectrum. If the amplitude exceeds the visual detection threshold, then the condition is met; otherwise, it is not met. The visual detection threshold is obtained by collecting N1 sets of visual grayscale time series of non-dangerous scenes, performing fast Fourier transform processing to obtain the corresponding visual spectrum, extracting the maximum amplitude of the visual spectrum, and calculating h1 times the maximum amplitude. h1 is preferably 3-5 to ensure that the signal is more than 3 times the noise. Wherein, N1 is the number of visual grayscale time series sets of non-dangerous scenes. The dominant frequency in vision; This represents the frequency resolution of the sub-window.

[0088] Condition 2: Obtain the inertial spectrum by performing a Fast Fourier Transform on the triaxial acceleration data, and detect the inertial spectrum. If the amplitude exceeds the inertial detection threshold, then the condition is met; otherwise, it is not. The inertial detection threshold is obtained by collecting N2 sets of triaxial acceleration data from non-hazardous scenarios, performing fast Fourier transform processing to obtain the corresponding inertial spectrum, extracting the maximum amplitude of the inertial spectrum, and calculating h2 times the maximum amplitude. h2 is preferably 4-6, as the noise of the triaxial acceleration data is lower, which can relatively improve the coefficient. Here, N2 is the number of sets of triaxial acceleration data from non-hazardous scenarios. The frequency is dominated by inertia.

[0089] If a child window satisfies condition 1 or condition 2, then it satisfies the consistency rule, and the detection result of the child window that satisfies the consistency rule is marked as 1; otherwise, the detection result is 0.

[0090] The ratio of the total number of detections of 1 in Nf sub-windows to Nf is used to obtain temporal consistency. This verifies the stability of the frequency in the time dimension and avoids false detections due to instantaneous noise. At the same time, when visual resonance fails to meet condition 1, condition 2 of the inertial mode can still be effectively triggered, ensuring the effectiveness of temporal consistency detection and preventing dangerous frequencies from being excluded due to visual failure.

[0091] If neither the visual dominant frequency nor the inertial dominant frequency is 0, the cross-modal consistency is the first modal value. If either the visual dominant frequency or the inertial dominant frequency is 0, the cross-modal consistency is the second modal value. Preferably, the first modal value is 1, and the second modal value is preferably 0.5. Enhancing the reliability of dangerous frequencies through bimodal cross-validation is an innovative design to address technical issues. When the visual resonance value is 0 and only the inertial dominant frequency is detected to be non-zero, the candidate is still retained with a consistency value of 0.5, which can prevent dangerous frequencies from being missed due to visual failure.

[0092] The confidence level of each frequency to be evaluated is obtained by weighting the overall signal-to-noise ratio, temporal consistency, and cross-modal consistency. Only frequencies with confidence levels exceeding the confidence threshold are retained, forming a set of dangerous frequencies. The confidence threshold is designed based on empirical values, preferably 0.7-0.8. The weighting coefficients can be designed empirically, with the overall signal-to-noise ratio being the most direct reliability indicator, and a weighting weight preferably of 0.5. Temporal consistency reflects the persistence of the frequency, and a weighting weight preferably of 0.3. Cross-modal consistency verification provides redundant confirmation, and a weighting weight preferably of 0.2. By comprehensively considering bimodal signal quality, temporal stability, and cross-modal consistency, frequency reliability is quantified, and only high-confidence frequencies are retained. In visual resonance scenarios, the high signal-to-noise ratio, temporal consistency, and cross-modal consistency, which rely on inertia, can still include dangerous frequencies in the set, preventing the cerebrospinal and extracorporeal brain systems from being deceived by visual artifacts.

[0093] Methods for constructing a complete frequency profile include:

[0094] Traverse the set of dangerous frequencies and filter them based on deduplication rules to obtain the filtering frequencies;

[0095] The deduplication rules include: for adjacent repetitive frequencies with a frequency difference less than half of the harmonic frequency matching threshold, retain the frequency with higher confidence; if the confidence levels are equal, retain the frequency with the larger frequency amplitude. The harmonic frequency matching threshold is set based on empirical values. Eliminating redundant repetitive frequencies with minimal frequency differences and prioritizing the retention of reliable frequencies with higher confidence or larger amplitudes avoids interference from redundant data in subsequent harmonic analysis. In scenarios where visual resonance fails, this approach highlights high-confidence, high-risk frequencies dominated by inertial modes, laying a reliable foundation for subsequent frequency profile construction.

[0096] Sort the selected frequencies in ascending order to obtain an ordered set; organize the discrete dangerous frequencies into an ordered sequence, which can provide a clear retrieval logic for subsequent matching of fundamental frequency and harmonics, and avoid missed harmonic detection due to disordered frequencies.

[0097] Arrange each danger frequency in ascending order. As a potential fundamental frequency, it detects whether a corresponding dangerous frequency exists in an ordered set. If it meets the harmonic filtering rules, then mark it. The frequency is marked as the fundamental frequency, and the marking is... If the frequency is not specified, mark it as a harmonic; otherwise, mark the frequency as 0.

[0098] Harmonic filtering rules include: and The ratio is an integer or within the range of integers; wherein, the range of integers is preferably within ±1% of the integer; by using harmonic screening rules to restore the intrinsic correlation between the fundamental frequency and harmonics of dangerous rotating machinery, even if the visual signal fails due to resonance, the high confidence frequency based on the inertial mode retention can still complete the matching of the fundamental frequency and harmonics, avoiding the omission of key harmonics due to the failure of a single mode.

[0099] The process iterates through each dangerous frequency in the ordered set, marking the frequency label and confidence level for each dangerous frequency. For dangerous frequencies labeled as fundamental or harmonic, the harmonics corresponding to the fundamental frequency and the fundamental frequency corresponding to the harmonics are also labeled, thus obtaining a complete frequency profile. A structured fundamental and harmonic frequency system is constructed, enabling subsequent safety exclusion zone delineation to comprehensively cover the fundamental frequency and its various harmonics, avoiding the resonance risk caused by only avoiding the fundamental frequency while ignoring harmonics. The resulting complete frequency profile provides a comprehensive dangerous frequency reference for the cerebellar-macrocerebral visual fusion control system, reducing the probability of simultaneous deception of both pathways from the source.

[0100] Constrained by a dangerous frequency set and a complete frequency profile, the visual acquisition parameters of the embodied intelligent agent are optimized, and a set of cognitive flow parameters controlling the brain and a set of reflex flow parameters controlling the cerebellum are generated.

[0101] Reference Figure 2 Methods for obtaining a set of cognitive flow parameters that control the brain include:

[0102] Obtain the dangerous frequencies marked as harmonics in the complete frequency profile, and combine this with dynamic safety threshold settings to control the frame rate and shutter speed of the brain, such as a frame rate safety zone. ,in, For frame rate; This is a dangerous frequency; Harmonic order; The maximum value of the harmonic order is preferably 3-5; The preferred safety threshold is the maximum value of 10% of the dangerous frequency and 5Hz; such as the shutter safety no-go zone. ,in, For shutter speed; by fully covering the fundamental frequency and its 3rd to 5th harmonics of dangerous frequencies, a full-dimensional safety zone for acquisition parameters is constructed. From a physical perspective, the resonance between frame rate and shutter speed and the fundamental frequency and harmonics of dangerous rotating machinery is actively avoided, eliminating the generation of static visual illusions from the source. This is the core protective measure to ensure the authenticity of dual-path perception of the cerebellum and cerebrum.

[0103] The process involves adjusting dynamic safety thresholds based on hazard frequency; using a calculation window to obtain the world coordinate system position sequence of the object within the window; fitting a curve of the world coordinate system position sequence versus time based on linear interpolation; calculating the ratio of the difference between the current world coordinate system position and the previous world coordinate system position to the change in time to obtain instantaneous velocity; weighting the instantaneous velocity and the velocity of the previous moment; and adjusting the weighting coefficient based on the average linear interpolation fitting error. The smaller the average linear interpolation fitting error, the more accurate the positioning and the higher the reliability of the instantaneous velocity. When the average linear interpolation fitting error is less than 5%, the weighting weight of the instantaneous velocity is preferably 0.7-0.8; when the average linear interpolation fitting error is between 5% and 15%, the weighting weight of the instantaneous velocity is preferably 0.5-0.6; and when the average linear interpolation fitting error is greater than 15%, the weighting weight of the instantaneous velocity is preferably 0.2-0.4, thus obtaining the velocity of the hazard source at the current moment.

[0104] By calculating window fitting and velocity smoothing, the true velocity of the hazard is accurately quantified, providing reliable data support for subsequent relative velocity calculation and dynamic adjustment of the dynamic safety threshold. Even if the visual illusion of stillness occurs due to resonance, the accuracy of velocity calculation can still be guaranteed by combining IMU data, avoiding the failure of dynamic safety threshold adjustment caused by velocity distortion. The difference between the velocity of the embodied agent and the velocity of the hazard is calculated to obtain the velocity vector difference. The magnitude of the velocity vector difference is calculated to obtain the relative velocity between the embodied agent and the hazard. The relative intensity of their motion is quantified, providing a core input for the dynamic safety threshold. The greater the relative velocity, the more severe the consequences of a collision caused by resonance, requiring stronger protection through a wider safety exclusion zone.

[0105] The calculation involves multiplying the velocity coefficient by the relative velocity, where the relative velocity is the relative velocity between the agent and the hazard, obtained by calculating the magnitude of the vector difference between the agent's velocity vector and the hazard's velocity vector. The sum of the base threshold and the velocity product yields a dynamically adjusted safety threshold. The base threshold is adjusted based on actual conditions, preferably the larger of 0.1 times the hazard frequency and 5Hz. The velocity coefficient, representing the threshold increment per meter per second of relative velocity, compensates for the influence of the relative velocity between the agent and the hazard on the frequency, and is preferably between 0.5 and 1.0. This approach overcomes the limitations of fixed safety thresholds by dynamically adapting the restricted area range based on the relative velocity between the agent and the hazard. Higher relative velocities indicate a greater and more imminent collision risk. Expanding the safety threshold further reduces the resonance probability, avoiding visual illusions caused by insufficient protection in high-risk scenarios due to fixed thresholds, and making the restricted area more closely reflect real-time risk conditions.

[0106] By removing the safe no-go zones of frame rate and shutter speed from the pre-calibrated feasible domain of visual parameters controlled by the camera in the brain, a safe and feasible parameter space is obtained. Through strict filtering of the resonance no-go zones of frame rate and shutter speed, it is ensured that all candidate visual parameters are far from dangerous frequencies and their harmonics, physically eliminating visual stillness artifacts caused by resonance, and providing distortion-free image input to the cerebellum and cerebellum systems. Within this safe and feasible parameter space, [the following parameters are generated]. A set of visual parameter combinations is used; images are acquired using each set of visual parameter combinations, and a comprehensive image quality score is calculated. Optimized visual parameters are then selected based on this comprehensive image quality score and used as the cognitive flow parameter set. The visual parameters include frame rate, shutter speed, aperture, and gain. The number of combinations of visual parameters is determined to meet the real-time requirements of embodied intelligent agents. The optimal value is 3-5, which can be determined based on actual conditions. Within a safe and feasible parameter space, a small number of representative samples can be quickly obtained to provide initial data support for subsequent parameter optimization. All samples avoid resonance risks, ensuring that the initial data does not carry quality defects related to static artifacts.

[0107] Methods for obtaining optimized visual parameters through screening include:

[0108] Images are acquired using each set of visual parameter combinations, and the image quality loss value of the images is calculated. An initial sample set is obtained by concatenating the visual parameter combinations and their corresponding image quality loss values. For example, the initial sample set includes: frame rate, shutter speed, aperture, gain, and image quality loss value.

[0109] Generate a combination of basic visual parameters within a safe and feasible parameter space; determine the initial starting point for parameter optimization, ensuring that the basic visual parameters are far from the resonance forbidden zone to avoid touching the risk zone during the optimization process, while providing anchor points for subsequent local searches to ensure that the optimization direction does not deviate from the dual goals of anti-resonance and high quality.

[0110] The image quality comprehensive score is calculated based on the combination of basic visual parameters. Within a safe and feasible parameter space, V candidate visual parameters are generated and combined at each step, forming a candidate visual parameter combination set. The image quality comprehensive score of each candidate visual parameter combination is calculated, and the visual parameter corresponding to the candidate visual parameter combination with the highest image quality comprehensive score is selected as the cognitive flow parameter set. V is the number of candidate visual parameters. To meet the real-time requirements of the embodied intelligent agent, V is preferably 3-5 and can be set according to the actual situation. Through local iterative search within the safe and feasible parameter space, the image quality is maximized while avoiding the risk of resonance. This ensures that the cerebellum and cerebellum systems are not deceived by resonance, and obtains high-quality images that meet the requirements of high-precision semantic cognition and rapid reflex.

[0111] Methods for calculating the overall image quality score include:

[0112] The Sobel operator is used to detect edges in the image to obtain the gradients in the X and Y directions. The gradient magnitudes are calculated, the average gradient magnitude of the image is statistically analyzed and normalized to obtain the sharpness. The image sharpness is directly quantified. The illusion of a stationary object caused by high-speed rotation due to resonance is essentially motion blur. Sharpness can accurately reflect the degree of this blur, providing core quality feedback for parameter optimization and avoiding the brain system from misjudging the motion state of the object due to the blurry image.

[0113] The gray level difference and gray level sum of the maximum and minimum gray levels of the image are calculated separately. The ratio of the gray level difference to the gray level sum is calculated to obtain the actual contrast and then normalized to obtain the contrast. The ability to distinguish between the target and the background is quantified. The resonant stillness artifact may cause the rotating object to visually merge with the background. The decrease in contrast will further aggravate the loss of features. It can ensure that the optimized image retains enough target features and provides clear distinction for the detection and segmentation of the cerebellum and cerebrum systems.

[0114] In the background region of the image, the variance of the pixel grayscale value is extracted and calculated by threshold segmentation, and then divided by the mean grayscale value of the background region to obtain the image noise intensity and normalize it to obtain the noise level; filter the interference of camera noise on image quality and avoid noise from obscuring the true features of the target; however, it should be noted that this step does not directly deal with the stillness artifact caused by resonance. Its core function is to ensure the image signal-to-noise ratio and eliminate noise interference for the evaluation of key dimensions such as sharpness and contrast.

[0115] The process involves calculating the average grayscale value of the image, the absolute value of the difference between the average grayscale value and the ideal grayscale value, and the ratio of the absolute value to the ideal grayscale value to obtain a first level value. The ideal grayscale value is set based on empirical data, such as 128 for an 8-bit image. Next, the sum of overexposed and underexposed pixels is calculated, and the exposure ratio of this sum to the total number of pixels in the image is calculated to obtain a second level value. Finally, the sum of the first and second level values ​​is calculated to obtain the exposure level. Overexposed pixels are those with a grayscale value not lower than an overexposure threshold, which is set based on empirical values ​​and is preferably 240-260. Underexposed pixels are those with a grayscale value not greater than an underexposure threshold, which is set based on empirical values ​​and is preferably 15-20. This process quantifies the image exposure uniformity to avoid loss of target details due to overexposure and underexposure. Specifically, under the resonant stillness artifact, abnormal exposure further weakens target recognition. This step ensures that the image retains complete target details, providing effective feature input for the brain and cerebellum systems.

[0116] A comprehensive image quality score is obtained by weighting sharpness, contrast, noise level, and exposure level. The weighting coefficients are designed based on empirical values, with a preferred weighting of 0.4 for sharpness, 0.3 for contrast, 0.15 for noise level, and 0.15 for exposure level. These weightings can be adjusted according to actual conditions. This comprehensive approach covers the core image quality dimensions, directly combating motion blur and feature fusion caused by resonance through sharpness and contrast, while ensuring overall image reliability through noise and exposure levels, providing a comprehensive quality basis for visual parameter optimization.

[0117] Methods for obtaining the set of reflex flow parameters controlling the cerebellum include:

[0118] The maximum value between the minimum safe frame rate and the highest frame rate supported by the camera is selected as the frame rate of the reflection stream. The minimum safe frame rate is the calibration parameter corresponding to the camera, which is generally 30fps. By selecting the highest frame rate supported by the camera, the sampling density of the reflection stream is maximized, reducing the probability of frame rate resonance with the frequency of dangerous rotating machinery from the source. At the same time, the high frame rate ensures the ability to capture fast motion and avoids the cerebellar pathway being deceived by the illusion of stillness due to insufficient sampling.

[0119] The reciprocal of twice the frame rate of the reflected flow is used as the shutter speed of the reflected flow. This avoids the loss of motion features caused by motion blur and keeps the shutter frequency away from the common frequencies of most dangerous rotating machinery in industry, further reducing the risk of resonance and preventing stillness artifacts caused by the coupling of the shutter with dangerous frequencies.

[0120] The aperture is set to a preset safe aperture value, which is preferably designed based on empirical values ​​and is set to 1 / 5.6 times the focal length of cameras commonly used in industrial scenarios. This ensures sufficient light intake for the image while avoiding an excessively large aperture that results in a shallow depth of field. By ensuring a stable amount of light intake through the preset safe aperture value, overexposure and underexposure of the image due to aperture abnormalities are avoided. This ensures that the optical flow calculation of reflected flow and abnormal motion detection can be carried out based on a clear image, indirectly reducing the risk of misjudgment caused by the superposition of stillness artifacts and exposure abnormalities.

[0121] The gain is set to a preset safe gain value, which is designed based on empirical values ​​and is preferably 1x. A low gain of 1x can minimize camera noise, ensure image purity, avoid noise interference with the motion perception accuracy of the reflected flow, ensure that the cerebellar pathway can accurately capture the real motion characteristics of dangerous rotating machinery, and not be masked or misjudged by noise, thus indirectly counteracting the influence of resonant static artifacts.

[0122] Based on the cognitive flow parameter set and the reflex flow parameter set, a dual pathway of the cerebellum and cerebrum is constructed, and the threat score and abnormal regions are obtained through analysis.

[0123] Reference Figure 3Methods for obtaining a threat score include:

[0124] The cognitive flow parameter set is used to control the brain to collect standard images. The cognitive flow parameter set has avoided resonance risks by passing through the security restricted area. The collected standard images do not have the illusion of a stationary object rotating at high speed. It provides real and distortion-free image input for subsequent threat assessment. It is the core premise to ensure the reliability of threat score calculation and avoids the misjudgment of threats caused by resonance deception from the source.

[0125] Standard images are used as input to a pre-trained object detection model to obtain initial detection results and their corresponding detection confidence levels. The initial detection results include object bounding boxes and object category labels. Initial detection results with a confidence level not lower than a detection confidence threshold are retained to obtain the object detection results. The detection confidence threshold is designed based on empirical values, preferably between 0.7 and 0.8. The preferred object detection model is the YOLOv8 model. Based on standard images without resonance artifacts, object detection can accurately identify object categories and bounding boxes. False detections are eliminated by filtering results with a confidence level not lower than the detection confidence threshold. This avoids missing detections or insufficient detection confidence for dangerous objects such as rotating equipment due to static artifacts, ensuring that the core objects of threat assessment are not missing.

[0126] Training methods for object detection models include:

[0127] Pre-collect B1 sets of target prediction data, which include standard images, initial detection results, and detection confidence corresponding to the initial detection results. The initial detection results include object bounding boxes and object category labels.

[0128] The standard image is used as the input to the object detection model, and the corresponding initial detection result and the detection confidence corresponding to the initial detection result are used as the output of the object detection model. The goal is to minimize the error between the output initial detection result and the detection confidence corresponding to the initial detection result and the actual initial detection result and the detection confidence corresponding to the initial detection result. The network parameters of the object detection model are iteratively optimized to obtain the network parameters that minimize the error between the output initial detection result and the detection confidence corresponding to the initial detection result and the actual initial detection result and the detection confidence corresponding to the initial detection result. The object detection model constructed with the corresponding network parameters is used as the trained object detection model.

[0129] By combining camera intrinsics and visual odometry, the position of each object in the world coordinate system is calculated;

[0130] Methods for calculating the position of each object in the world coordinate system include:

[0131] The region corresponding to the bounding box of the object in the target detection result is used as the input of the semantic segmentation model to obtain the binary mask of each object; the semantic segmentation model is preferably a CNN network; the binary mask can accurately extract the foreground region of the object and eliminate background interference, while the image without resonance artifacts ensures the accuracy of the segmentation boundary, avoids segmentation failure due to visual fusion of the object and the background, and provides reliable regional constraints for subsequent depth estimation and position calculation.

[0132] Training methods for semantic segmentation models include:

[0133] Pre-collect B2 sets of semantic segmentation data, which include the regions corresponding to the bounding boxes of objects in the target detection results, as well as the binary masks of each object.

[0134] The regions corresponding to the bounding boxes of objects in the object detection results are used as input to the semantic segmentation model, and the binary masks of each corresponding object are used as outputs of the semantic segmentation model. With the goal of minimizing the error between the output binary masks of each corresponding object and the actual binary masks of each corresponding object, the network parameters of the semantic segmentation model are iteratively optimized to obtain the network parameters that minimize the error between the output binary masks of each corresponding object and the actual binary masks of each corresponding object. The semantic segmentation model constructed with the corresponding network parameters is used as the trained semantic segmentation model.

[0135] The foreground pixel grayscale values ​​of the object are extracted based on the binary mask to obtain the foreground image. The foreground image is then used as the input of a pre-trained monocular depth model to obtain the depth value of each foreground pixel.

[0136] Training methods for monocular depth models include:

[0137] B3 sets of deep training data were collected in advance. The deep training data included the foreground image and the corresponding depth value of each foreground pixel.

[0138] The foreground image from the object detection results is used as the input to the monocular depth model, and the depth value of each corresponding foreground pixel is used as the output of the monocular depth model. With the goal of minimizing the error between the output depth value of each corresponding foreground pixel and the actual depth value of each corresponding foreground pixel, the network parameters of the monocular depth model are iteratively optimized to obtain the network parameters that minimize the error between the output depth value of each corresponding foreground pixel and the actual depth value of each corresponding foreground pixel. The monocular depth model constructed with the corresponding network parameters is used as the trained monocular depth model.

[0139] Based on the coordinates of the top left and bottom right corners of the object's bounding box in the target detection results, the average coordinates are calculated to obtain the center position of the object's bounding box, and the depth value of the center position is extracted as the object's center depth. Depth estimation relies on a clear object foreground region. Standard images without resonance blur allow for more accurate calculation of the depth value of each foreground pixel. The statistical results of the object's center depth can truly reflect the distance between the object and the embodied intelligent agent, avoiding depth estimation bias caused by static artifacts, and providing reliable data support for distance threat factors.

[0140] By combining camera intrinsic parameters, the center of the bounding box is converted into 3D coordinates in the camera coordinate system based on the depth of the object center. For example, the coordinate difference between the center of the bounding box and the principal point of the camera is calculated, then the product of the coordinate difference and the depth of the object center is calculated, and finally the ratio of the product to the focal length is calculated to obtain the 3D coordinates in the camera coordinate system.

[0141] Based on the embodied agent's own motion, 3D coordinates in the camera coordinate system are converted into the object's position in the world coordinate system. This is achieved by calculating the vector product of the rotation matrix and the 3D coordinate column vectors in the camera coordinate system, and then adding this product to the translation vector to obtain the object's position in the world coordinate system. The rotation matrix and translation vector are obtained by pre-calibrating the transformation relationship between the camera and world coordinate systems. Based on precise center depth and bounding box center, combined with calibration parameters, the coordinate transformation is completed, enabling the acquisition of the object's true position in the world coordinate system. This avoids misjudgments of position due to resonant static artifacts and lays a spatial foundation for calculating distance and velocity threat factors.

[0142] Calculate the distance between the embodied intelligent agent's own world coordinates and the object's position to obtain distance features;

[0143] The distance feature is calculated as the ratio of the distance feature to the preset safe distance of the object. The difference between the value 1 and the distance ratio is calculated, and the smaller of the two values, the distance difference and the value 1, is selected as the distance threat factor. The preset safe distance is set based on empirical values, preferably 30cm-50cm. The distance feature calculated based on the real-world position of the object can accurately quantify the collision risk. The distance threat factor implements the logic that the closer the distance, the higher the threat. The absence of resonance illusion ensures the authenticity of the distance and avoids underestimation of the threat factor due to misjudgment of the distance, effectively combating the risk misjudgment caused by resonance deception.

[0144] Calculate the ratio of the object's current velocity to the preset maximum safe velocity, and select the smaller of the velocity ratio and the value 1 as the velocity threat factor; the preset maximum safe velocity is set based on empirical values, preferably 2-3 m / s; the velocity threat factor can reflect the degree of danger of the object's movement.

[0145] Based on the object category, a preset category threat value is obtained as a category threat factor. The preset category threat value is obtained directly from pre-set data, such as static objects = 0.1, rotating equipment = 0.7, and moving objects = 0.9. The preset high category threat value of dangerous objects such as rotating equipment directly targets dangerous rotating machinery in core technical issues. Even if there is a risk of resonance, this factor can give such objects a high threat weight in advance, avoiding a decrease in vigilance against core hazards due to the illusion of stillness. It is a targeted design to deal with core technical issues.

[0146] Threat scores are obtained by weighting distance threat factors, speed threat factors, and category threat factors; a higher threat score indicates a higher threat. The weighting coefficients are designed based on empirical values, with the preferred weights for distance threat factors (0.4), speed threat factors (0.3), and category threat factors (0.3), which can be adjusted according to actual circumstances. By comprehensively considering threat characteristics across distance, speed, and category, and ensuring each characteristic is based on accurate data free of resonance artifacts, the weighted calculation provides a comprehensive and objective assessment of the threat level of an object. The high weighting of the category threat factor on rotating equipment strengthens focus on core hazards, effectively avoiding threat underestimation caused by resonance deception in the brain.

[0147] Methods for obtaining abnormal regions include:

[0148] A reference image is acquired using a reflection flow parameter set, and the optical flow amplitude of each pixel in the reference image is calculated. The calculation method is the same as that used in visual data streams, and will not be elaborated further. The reflection flow parameter set avoids resonance risks through a high frame rate and shutter-based collaborative design. The acquired reference image is free of artifacts of stationary high-speed rotating objects, providing real motion feature input for anomaly detection. This prevents the reflection flow from being deceived by resonance at the source, ensuring the accuracy of subsequent motion perception. Based on the reference image without resonance artifacts, block optical flow calculation can accurately capture the real motion displacement of objects, and the optical flow amplitude directly reflects the intensity of motion. This avoids the underestimation of the motion displacement of dangerous objects such as rotating equipment due to stationary artifacts, ensuring the reliability of anomaly pixel marking.

[0149] If the synthesized vibration amplitude is greater than the vibration threshold, it is determined to be an abnormal vibration. The trigger time is recorded, and the vibration direction is converted to the camera coordinate system through external parameters. The vibration threshold is set based on empirical values. For example, the vibration threshold for a common rotating motor is generally 0.1-0.15 times the acceleration.

[0150] Calculate the root mean square (RMS) values ​​of the triaxial acceleration data for the X, Y, and Z axes to obtain the energy value corresponding to each axis. Sum the three axis energy values ​​to obtain the total energy. Calculate the proportion of each axis's energy value to the total energy. Identify the axis with a single proportion exceeding a threshold as the dominant direction for that single axis. Obtain the unit vector of the dominant direction as the dominant vibration direction vector. If no single axis has a proportion exceeding the threshold, it is determined to be a composite direction. Use the three axis energies as three directional components to construct a composite direction vector. Calculate the magnitude of the composite direction vector, i.e., calculate the RMS of the three axis energies. Then calculate the ratio of the composite direction vector to its magnitude to obtain the dominant vibration direction vector. The preferred proportion threshold is 50% to highlight the dominant direction.

[0151] The dominant vibration direction vector is multiplied by the extrinsic rotation matrix to obtain the vibration direction vector in the camera coordinate system. The extrinsic rotation matrix is ​​obtained by pre-calibrating the transformation relationship between the IMU coordinate system and the camera coordinate system. The field of view corresponding to the vibration direction is determined according to the preset region division rules, and the field of view corresponding to the vibration direction is regarded as an abnormal pixel.

[0152] The preset region division rules include: when the positive X-axis dominates, the column range of the field of view corresponding to the vibration direction is the right 1 / 3 of the entire reference image, and the entire row range; when the negative X-axis dominates, the column range of the field of view corresponding to the vibration direction is the left 1 / 3 of the entire reference image, and the entire row range; when the positive Y-axis dominates, the row range of the field of view corresponding to the vibration direction is the bottom 1 / 3 of the entire reference image, and the entire column range; when the negative Y-axis dominates, the row range of the field of view corresponding to the vibration direction is the top 1 / 3 of the entire reference image, and the entire column range; when the positive Z-axis dominates or is combined with the negative Z-axis, the field of view corresponding to the vibration direction is the entire reference image; when the negative Z-axis dominates, there is no field of view corresponding to the vibration direction.

[0153] If the synthesized vibration amplitude is not greater than the vibration threshold, then optical flow detection is used: pixels with optical flow amplitude greater than the optical flow threshold are marked as abnormal pixels, and the rest are background pixels; whereby the optical flow threshold is set based on empirical values, preferably 8-12; visual optical flow is supplemented by IMU vibration detection to form a dual-modal anomaly perception. Even if visual detection is missed due to resonance in extreme cases, the IMU can still lock the area of ​​dangerous rotating machinery through vibration characteristics, avoiding the failure to detect abnormal pixels caused by being deceived by a single visual modality. This is a redundant design to address core technical issues.

[0154] Based on connected component analysis, adjacent anomalous pixels are merged to form anomalous regions. Using real anomalous pixels, connected component analysis can accurately aggregate the movement areas of the same hazardous object, avoiding fragmentation or missed detection of hazardous areas due to stationary illusions. This provides complete hazardous area input for subsequent activity intensity analysis and reflection threat map construction.

[0155] The activity intensity in abnormal areas is analyzed. When the activity intensity exceeds an intensity threshold, a reflected threat map is constructed based on the threat score and the abnormal area. The threat centroid and global threat intensity of the reflected threat map are analyzed, and the real-time obstacle avoidance direction and speed are calculated based on the threat centroid and global threat intensity. The intensity threshold is set based on empirical values, preferably 30%.

[0156] Methods for analyzing the activity intensity of anomalous regions include:

[0157] The average optical flow amplitude of each anomalous region is calculated separately. The ratio of the average optical flow amplitude of each anomalous region to the maximum average optical flow amplitude of all anomalous regions is then used to obtain the activity intensity of the anomalous region. Activity intensity directly quantifies the intensity of motion in the anomalous region. The optical flow amplitude without resonance artifacts ensures the authenticity of the activity intensity and avoids the misjudgment of low activity intensity of rotating equipment due to static artifacts. This provides an accurate basis for motion risk assessment and obstacle avoidance decisions.

[0158] Methods for constructing reflection threat maps include:

[0159] Create a grid map with the same size as the embodied agent's current viewpoint image, with an initial value of 0;

[0160] For each detected object, calculate the cumulative threat value within an L×L pixel area of ​​the object, based on the object's location and threat score; if multiple threat sources exist, calculate the threat value of each individual threat first. ,in, For the first Threat scores for each threat region; For the first The center position of each threat region in the image coordinate system; It is an exponential function with the natural constant as its base. For diffusion radius parameters, ; The base diffusion radius is preferably 5%-10% of the image width; and This is the adjustment coefficient; The normalized value is the area of ​​the threat region; then the threat values ​​of each threat source are linearly superimposed and normalized to obtain the cumulative threat value;

[0161] Calculate the ratio of each cumulative threat value in the grid map to the maximum cumulative threat value in the grid map to obtain the normalized reflection threat map, as follows:

[0162] Traverse all abnormal regions;

[0163] If an abnormal region overlaps with a semantic threat projection region, the abnormality is determined to be the real-time motion behavior of the known threat object. In this case, the product of the enhancement amplitude coefficient and the intensity of the abnormal activity is superimposed on the accumulated threat value as a new accumulated threat value. The enhancement amplitude coefficient is obtained based on historical experience statistics.

[0164] If an anomalous region does not overlap with any semantic threat projection region, it is considered an unknown threat; a base threat value is superimposed on its region location as an accumulated threat value, the base threat value being the product of the threat coefficient and the intensity of the anomalous activity; whereby the threat coefficient is obtained based on historical experience statistics;

[0165] The ratio of the accumulated threat value in the grid map to the maximum accumulated threat value in the grid map is calculated to obtain a normalized reflection threat map. The known threat score from the cognitive stream is fused with the real-time abnormal activity intensity from the reflection stream. Overlapping areas are enhanced with increased threats, and unknown anomalies are used to supplement basic threats. This approach avoids underestimating the threat of known dangerous rotating equipment due to the illusion of stillness, while also ensuring no unknown motion anomaly threats are overlooked, achieving full-scene threat coverage.

[0166] Methods for calculating real-time obstacle avoidance direction and speed based on the threat center of gravity and global threat intensity include:

[0167] When the overall threat value of the reflected threat map is not 0, the weighted center of gravity of the reflected threat map is calculated as the threat center of gravity, with the weighting weight being the threat value of each pixel; otherwise, the center of the field of view is used as the threat center of gravity. The weighted center of gravity can accurately locate the core threat area, while the reflected threat map without resonance artifacts ensures the authenticity of the threat distribution, avoids the threat center of gravity of dangerous rotating equipment shifting due to static artifacts, and ensures that the obstacle avoidance direction always points away from the real danger, thus avoiding obstacle avoidance failure.

[0168] Calculate the threat offset between the coordinates of the threat centroid and the coordinates of the field of view center; convert the threat offset into an angular offset, and calculate the obstacle avoidance direction angle based on the angular offset;

[0169] The maximum value in the reflected threat map is used as the global threat intensity. The obstacle avoidance speed is obtained by multiplying the global threat intensity by the maximum permissible speed. The maximum permissible speed is set according to the maximum movement speed of the embodied intelligent agent's hardware, preferably 0.8 times the maximum movement speed of the hardware. The maximum movement speed of the hardware can be extracted from the device manual. The global threat intensity is based on the actual threat distribution and can accurately reflect the overall level of danger. The obstacle avoidance speed is positively correlated with the threat intensity, avoiding the underestimation of the global threat intensity of dangerous rotating devices due to the illusion of stillness. This ensures that the higher the danger, the faster the obstacle avoidance speed, improving the timeliness and effectiveness of obstacle avoidance.

[0170] Example 2:

[0171] Please see Figure 4As shown, this embodiment provides a brain-brain fusion control system for an embodied intelligent agent, including:

[0172] Hazard Analysis Module: Collects and analyzes multimodal data of the embodied intelligent agent, extracts the dominant frequencies of each mode, performs parallel screening of the dominant frequencies of each mode to obtain a set of hazardous frequencies, performs harmonic decomposition on each frequency in the set of hazardous frequencies, and constructs a complete frequency profile;

[0173] Parameter Analysis Module: Based on the dangerous frequency set and the complete frequency profile, optimize the visual acquisition parameters of the embodied intelligent agent, and generate the cognitive flow parameter set controlling the brain and the reflex flow parameter set controlling the cerebellum;

[0174] Path analysis module: Based on the set of cognitive flow parameters controlling the brain and the set of reflex flow parameters controlling the cerebellum, the brain and cerebellum are respectively controlled to collect corresponding images, and the corresponding images collected by the brain and cerebellum are analyzed to obtain threat scores and abnormal regions.

[0175] Fusion control module: Analyzes the activity intensity in abnormal areas. When the activity intensity is greater than the intensity threshold, it constructs a reflected threat map based on the threat score and the abnormal area, analyzes the threat centroid and global threat intensity of the reflected threat map, and calculates the real-time obstacle avoidance direction and obstacle avoidance speed based on the threat centroid and global threat intensity.

[0176] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

[0177] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for controlling the cerebellum and cerebrum-based AI visual fusion in an embodied intelligent agent, characterized in that, include: Collect and analyze multimodal data of embodied intelligent agents, extract the dominant frequencies of each mode, perform parallel screening of the dominant frequencies of each mode to obtain a set of dangerous frequencies, perform harmonic decomposition on each frequency in the set of dangerous frequencies, and construct a complete frequency profile. Methods for obtaining a set of dangerous frequencies include: Parallel visual dominant frequencies and inertial dominant frequencies are used to obtain the frequency set to be evaluated; Calculate the overall signal-to-noise ratio, time consistency, and cross-modal consistency for each frequency to be evaluated; The confidence level of each frequency to be evaluated is obtained by weighting the overall signal-to-noise ratio, time consistency and cross-modal consistency. Only frequencies with confidence levels exceeding the confidence level threshold are retained to form a set of dangerous frequencies. Constrained by a set of dangerous frequencies and a complete frequency profile, the visual acquisition parameters of the embodied agent are optimized, and a set of cognitive flow parameters controlling the brain and a set of reflex flow parameters controlling the cerebellum are generated. Methods for constructing the complete frequency profile include: Traverse the set of dangerous frequencies and filter them based on deduplication rules to obtain the filtering frequencies; Sort the selection frequencies in ascending order to obtain an ordered set; Arrange each danger frequency in ascending order. As a potential fundamental frequency, it detects whether a corresponding dangerous frequency exists in an ordered set. If it meets the harmonic filtering rules, then mark it. The frequency is marked as the fundamental frequency, and the marking is... If the frequency is not specified, mark it as a harmonic; otherwise, mark the frequency as 0. Iterate through each dangerous frequency in the ordered set, mark the frequency label and confidence level of all dangerous frequencies in the ordered set, and for dangerous frequencies marked as fundamental frequency or harmonic, mark the harmonics corresponding to the fundamental frequency and the fundamental frequency corresponding to the harmonic to obtain a complete frequency profile. Based on the set of cognitive flow parameters controlling the brain and the set of reflex flow parameters controlling the cerebellum, the brain and cerebellum are respectively controlled to collect corresponding images. The corresponding images collected by the brain and cerebellum are analyzed to obtain threat scores and abnormal regions. Analyze the activity intensity in abnormal areas. When the activity intensity exceeds the intensity threshold, construct a reflected threat map based on the threat score and the abnormal area. Analyze the threat centroid and global threat intensity of the reflected threat map. Calculate the real-time obstacle avoidance direction and speed based on the threat centroid and global threat intensity.

2. The cerebellar-brain-based AI visual fusion control method for embodied intelligent agents according to claim 1, characterized in that, Methods for obtaining a set of cognitive flow parameters that control the brain include: Based on the dangerous frequencies marked as harmonics in the complete frequency profile, and combined with dynamic safety thresholds, a safe no-go zone for controlling the frame rate and shutter speed of the brain is set; wherein, the dynamic safety threshold is adjusted according to the dangerous frequencies; Calculate the product of the speed coefficient and the relative speed, where the relative speed is the relative speed between the embodied intelligent agent and the hazard source. Calculate the sum of the product of the basic threshold and the speed to obtain the dynamically adjusted safety threshold. By removing the safe no-go zones of frame rate and shutter speed from the pre-calibrated feasible region of the visual parameters controlled by the camera, a safe and feasible parameter space is obtained. Within this safe and feasible parameter space, [the system] generates [the appropriate parameters]. A set of visual parameters is used to acquire images and calculate the overall image quality score of each set of visual parameters. Based on the overall image quality score, optimized visual parameters are selected and used as the cognitive flow parameter set. This represents the number of visual parameter combinations; visual parameters include frame rate, shutter speed, aperture, and gain.

3. The cerebellar-cerebral AI visual fusion control method for embodied intelligent agents according to claim 2, characterized in that, Methods for obtaining optimized visual parameters through filtering include: Images are acquired using each set of visual parameter combinations, and the image quality loss value of the images is calculated. An initial sample set is obtained by concatenating the visual parameter combinations and the corresponding image quality loss values. Generate a combination of basic visual parameters within a safe and feasible parameter space; The image quality comprehensive score is calculated based on the combination of basic visual parameters. Within the safe and feasible parameter space, V candidate visual parameters are generated and combined with the basic visual parameter combination as the center to form a candidate visual parameter combination set. The image quality comprehensive score of each candidate visual parameter combination is calculated. The visual parameter corresponding to the candidate visual parameter combination with the highest image quality comprehensive score is selected as the cognitive flow parameter set; V is the number of candidate visual parameters.

4. The cerebellar-cerebral AI visual fusion control method for embodied intelligent agents according to claim 3, characterized in that, Methods for calculating the overall image quality score include: The Sobel operator is used to detect edges in the image to obtain the gradients in the X and Y directions. The gradient magnitudes are calculated, the average gradient magnitude of the image is statistically analyzed, and normalization is performed to obtain the sharpness. Calculate the gray level difference and gray level sum of the maximum and minimum gray levels of the image, respectively. Calculate the ratio of the gray level difference to the gray level sum to obtain the actual contrast ratio and perform normalization processing to obtain the contrast ratio. In the background region of the image, the variance of the pixel gray value is extracted and calculated by threshold segmentation. Then, it is divided by the mean gray value of the background region to obtain the image noise intensity and normalization processing to obtain the noise level. Calculate the average grayscale value of the image, calculate the absolute value of the difference between the average grayscale value and the ideal grayscale value, and calculate the ratio of the absolute value to the ideal grayscale value to obtain a first level value; calculate the sum of the number of overexposed pixels and the number of underexposed pixels, and then calculate the exposure ratio of the sum of the pixel counts to the total number of pixels in the image to obtain a second level value; then calculate the sum of the first level value and the second level value to obtain the exposure level; wherein, the number of overexposed pixels is the number of pixels with a grayscale value not lower than the overexposure threshold, and the number of underexposed pixels is the number of pixels with a grayscale value not greater than the underexposure threshold; A comprehensive image quality score is obtained by weighting sharpness, contrast, noise level, and exposure level.

5. The cerebellar-brain-based AI visual fusion control method for embodied intelligent agents according to claim 1, characterized in that, Methods for obtaining a threat score include: A cognitive flow parameter set was used to control the acquisition of standard images of the brain. The standard image is used as input to the pre-trained object detection model to obtain the initial detection result and the corresponding detection confidence. The initial detection result includes the object bounding box and the object's category label. The initial detection results with a detection confidence of not less than the confidence detection threshold are retained to obtain the object detection result. By combining camera intrinsics and visual odometry, the position of each object in the world coordinate system is calculated; Calculate the distance between the embodied intelligent agent's own world coordinates and the object's position to obtain distance features; Calculate the distance ratio between the distance feature and the preset safe distance of the object, calculate the distance difference between the value 1 and the distance ratio, and select the smaller value between the distance difference and the value 1 as the distance threat factor; Calculate the ratio of the object's current speed to the preset maximum safe speed, and select the smaller of the speed ratio and the value 1 as the speed threat factor; Based on the object category, obtain the preset category threat value as the category threat factor; The threat score is obtained by weighting the distance threat factor, speed threat factor, and category threat factor.

6. The cerebellar-cerebral AI visual fusion control method for embodied intelligent agents according to claim 1, characterized in that, Methods for analyzing the activity intensity of anomalous regions include: The average optical flow amplitude of each anomalous region is calculated, and the ratio of the average optical flow amplitude of each anomalous region to the maximum average optical flow amplitude of all anomalous regions is calculated to obtain the activity intensity of the anomalous region.

7. The cerebellar-cerebral AI visual fusion control method for embodied intelligent agents according to claim 1, characterized in that, Methods for constructing reflection threat maps include: Create a grid map with the same size as the embodied agent's current viewpoint image, with an initial value of 0; For each detected object, calculate the cumulative threat value within the object's L×L pixel area based on the object's location and threat score; Calculate the ratio of each cumulative threat value in the grid map to the maximum cumulative threat value in the grid map to obtain a normalized reflection threat map.

8. The cerebellar-cerebral AI visual fusion control method for embodied intelligent agents according to claim 1, characterized in that, Methods for obtaining the overall signal-to-noise ratio, time consistency, and cross-modal consistency for each frequency to be evaluated include: Calculate the visual signal-to-noise ratio based on the complex spectrum corresponding to the dominant visual frequency; Calculate the root mean square of the dominant inertial frequency in the inertial spectrum to obtain the amplitude of the synthesized signal; Calculate the composite vibration amplitude of the triaxial acceleration data, perform a fast Fourier transform on the composite vibration amplitude to obtain the composite spectrum, and calculate the inertial signal-to-noise ratio of the composite spectrum. The overall signal-to-noise ratio is obtained by weighting the visual signal-to-noise ratio and the inertial signal-to-noise ratio. The total sampling time is divided into Nf consecutive sub-windows, each with an equal duration; For each sub-window, a consistency rule check is performed; if the consistency rule is satisfied, the corresponding sub-window's check result is 1, otherwise the check result is 0; the ratio of the total number of check results of 1 in Nf sub-windows to Nf is calculated to obtain the time consistency. If neither the visual dominant frequency nor the inertial dominant frequency is 0, then the cross-modal consistency is the first modal value. If either the visual dominant frequency or the inertial dominant frequency is 0, then the cross-modal consistency is the second modal value.

9. A brain-brain fusion control system for an embodied intelligent agent, implementing the brain-brain fusion control method for an embodied intelligent agent as described in any one of claims 1-8, characterized in that, include: Hazard Analysis Module: Collects and analyzes multimodal data of the embodied intelligent agent, extracts the dominant frequencies of each mode, performs parallel screening of the dominant frequencies of each mode to obtain a set of hazardous frequencies, performs harmonic decomposition on each frequency in the set of hazardous frequencies, and constructs a complete frequency profile; Methods for obtaining a set of dangerous frequencies include: Parallel visual dominant frequencies and inertial dominant frequencies are used to obtain the frequency set to be evaluated; Calculate the overall signal-to-noise ratio, time consistency, and cross-modal consistency for each frequency to be evaluated; The confidence level of each frequency to be evaluated is obtained by weighting the overall signal-to-noise ratio, time consistency and cross-modal consistency. Only frequencies with confidence levels exceeding the confidence level threshold are retained to form a set of dangerous frequencies. The parameter analysis module optimizes the visual acquisition parameters of the embodied agent, constrained by a set of dangerous frequencies and a complete frequency profile, and generates a set of cognitive flow parameters controlling the brain and a set of reflex flow parameters controlling the cerebellum. The methods for constructing the complete frequency profile include: Traverse the set of dangerous frequencies and filter them based on deduplication rules to obtain the filtering frequencies; Sort the selection frequencies in ascending order to obtain an ordered set; Arrange each danger frequency in ascending order. As a potential fundamental frequency, it detects whether a corresponding dangerous frequency exists in an ordered set. If it meets the harmonic filtering rules, then mark it. The frequency is marked as the fundamental frequency, and the marking is... If the frequency is not specified, mark it as a harmonic; otherwise, mark the frequency as 0. Iterate through each dangerous frequency in the ordered set, mark the frequency label and confidence level of all dangerous frequencies in the ordered set, and for dangerous frequencies marked as fundamental frequency or harmonic, mark the harmonics corresponding to the fundamental frequency and the fundamental frequency corresponding to the harmonic to obtain a complete frequency profile. Path analysis module: Based on the set of cognitive flow parameters controlling the brain and the set of reflex flow parameters controlling the cerebellum, the brain and cerebellum are respectively controlled to collect corresponding images, and the corresponding images collected by the brain and cerebellum are analyzed to obtain threat scores and abnormal regions. Fusion control module: Analyzes the activity intensity in abnormal areas. When the activity intensity is greater than the intensity threshold, it constructs a reflected threat map based on the threat score and the abnormal area, analyzes the threat centroid and global threat intensity of the reflected threat map, and calculates the real-time obstacle avoidance direction and obstacle avoidance speed based on the threat centroid and global threat intensity.

Images