Dynamic holder control system of MEMS (micro-electromechanical system) optical and inertial fused skynet system

Abstract

The invention discloses a dynamic holder control system of a skynet system with MEMS optical and inertial fusion, and relates to the field of multi-sensor data fusion and multi-modal system design. According to the dynamic holder control system for the skynet system based on MEMS optics and inertia fusion, related data of ship swing amplitude and related data of ship vibration are collected; an inertial device is used for compensating the defects of optical equipment in a high-dynamic scene, and visual attitude determination positioning is used for compensating the defect of long-time drift of the inertial device, so that the overall stability and accuracy of the system are improved; and finally optimizing the dynamic holder system through an optimization module according to a threshold comparison result of the jitter evaluation coefficient of the holder equipment. According to the invention, the jitter evaluation coefficient of the cradle head equipment is compared with the jitter evaluation threshold value of the cradle head equipment, so that the dynamic cradle head system is finally optimized through the optimization module according to the threshold value comparison result of the jitter evaluation coefficient of the cradle head equipment, and the problem of poor monitoring quality of the cradle head equipment in the prior art is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of electrical data processing, and in particular to a Skynet system dynamic pan-tilt control system integrating MEMS optics and inertia. Background Art

[0002] The investment in R&D and application promotion of marine intelligent vehicles have become the focus of promoting the development of marine resource exploration, environmental protection, disaster prevention and response, and other fields. Furthermore, the application of marine intelligent vehicles is not limited to this. They not only enhance the autonomy and flexibility in marine exploration, but also reduce the dependence on human resources and potential risks.

[0003] The existing Skynet system dynamic gimbal control system that integrates MEMS optics and inertia is implemented through an airborne inertial stabilized gimbal system, including a two-degree-of-freedom airborne gimbal mechanism, a micro-attitude reference system designed using MEMS sensors, and a Kalman filter algorithm that fuses various sensor information to obtain optimal attitude information. In addition, an adaptive fuzzy control algorithm is used to design the control system of the airborne gimbal to achieve the stability of the gimbal mechanism and the feasibility of the sensor system and control system.

[0004] For example, the patent application with publication number CN118733901A discloses a pan-tilt control path search method based on big data, including: data collection, data preprocessing, establishment of a pan-tilt path planning system, adjustment of pan-tilt path planning system parameters and pan-tilt control path search. The present invention belongs to the field of path search technology, and specifically refers to a pan-tilt control path search method based on big data. This scheme generates an effective path planning strategy by initializing multiple commentator networks and an actor network; designs a nonlinear piecewise function, adjusts the exploration rate based on the time step, introduces experience playback and designs loss calculation; to enhance the certainty and stability of path planning; the initialized candidate solution position can cover a wider search space, taking into account the total system loss value and the distance calculation fitness from the optimal solution, and guiding the search process closer to the global optimal solution.

[0005] For example, the control method, device, storage medium and electronic device of the gimbal announced in the invention patent with announcement number: CN113724324B include: obtaining the first coordinate of the target gimbal, wherein the first coordinate is the first coordinate of the first position where the target gimbal is currently located; in the case of detecting a target event, determining the second coordinate of the second position to which the target gimbal needs to rotate, wherein the target event is an event for triggering the monitoring device carried by the target gimbal to perform a motion detection operation, and the second position is the end position of a rotation completed when the monitoring device performs the motion detection operation; determining the third coordinate based on the first coordinate and the second coordinate, wherein the distance between the first coordinate and the third coordinate is less than the distance between the first coordinate and the second coordinate; controlling the rotation of the target gimbal to rotate the target gimbal to the third coordinate.

[0006] However, in the process of implementing the technical solution of the invention in the embodiments of the present application, the present application found that the above technology has at least the following technical problems:

[0007] In the prior art, when a ship is sailing, the influence of waves, sea breeze and tides causes the ship to sway and vibrate, which in turn affects the pan / tilt device installed on the ship, causing the captured images or data to shake and distort, resulting in a decrease in the monitoring quality of the pan / tilt device. Summary of the invention

[0008] The embodiment of the present application solves the problem of reduced monitoring quality of pan-tilt devices in the prior art by providing a Skynet system dynamic pan-tilt control system that integrates MEMS optics and inertia, thereby achieving the effect of improving the monitoring quality of pan-tilt devices.

[0009] The embodiment of the present application provides a Skynet system dynamic pan-tilt control system that integrates MEMS optics and inertia, including: a data collection and preprocessing module, a data analysis module, a threshold comparison module and an optimization module: a data collection and preprocessing module: collects ship sway amplitude-related data and ship vibration-related data, preprocesses the ship sway amplitude-related data and the ship vibration-related data to obtain preprocessed ship sway amplitude-related data and preprocessed ship vibration-related data, and obtains an impact assessment threshold of the sway degree on the pan-tilt device and a jitter assessment threshold of the pan-tilt device from a database; a data analysis module: analyzes the preprocessed ship sway amplitude-related data to obtain an impact assessment coefficient of the sway degree on the pan-tilt device, analyzes the preprocessed ship vibration-related data to obtain an impact assessment coefficient of the vibration on the pan-tilt device, and analyzes the preprocessed ship vibration-related data to obtain an impact assessment coefficient of the vibration on the pan-tilt device. The impact assessment coefficient of the sway degree on the gimbal device and the impact assessment coefficient of the vibration on the gimbal device are comprehensively evaluated to obtain the jitter assessment coefficient of the gimbal device; the threshold comparison module: the impact assessment coefficient of the sway degree on the gimbal device is compared with the impact assessment threshold of the sway degree on the gimbal device to obtain the threshold comparison result of the impact assessment coefficient of the sway degree on the gimbal device, the jitter assessment coefficient of the gimbal device is compared with the jitter assessment threshold of the gimbal device to obtain the threshold comparison result of the jitter assessment coefficient of the gimbal device; the optimization module: according to the threshold comparison result of the impact assessment coefficient of the sway degree on the gimbal device, the dynamic gimbal system is preliminarily optimized through the optimization module, and according to the threshold comparison result of the jitter assessment coefficient of the gimbal device, the dynamic gimbal system is finally optimized through the optimization module.

[0010] Furthermore, the specific process of collecting ship sway amplitude related data and ship vibration related data is as follows: ship sway amplitude related data include roll angle, pitch angle, vertical roll angle, roll displacement, pitch displacement and vertical roll displacement; ship vibration related data include vibration velocity, vibration period, natural frequency and dynamic amplification factor.

[0011] Furthermore, the specific process of preprocessing the ship roll amplitude related data and the ship vibration related data is: checking the timestamps of the sensors on the ship, identifying and eliminating outliers, filling in missing values, using a low-pass filter to remove high-frequency noise, retaining low-frequency ship roll amplitude related data and ship vibration related data, and normalizing the ship roll amplitude related data and ship vibration related data.

[0012] Furthermore, the specific process of analyzing the pre-processed data related to the ship's sway amplitude is as follows: the ship's roll angle, pitch angle, vertical roll angle, roll displacement, pitch displacement and vertical roll displacement are transmitted from the data acquisition and preprocessing module to the data analysis module, and the ship's roll angle, pitch angle, vertical roll angle, roll displacement, pitch displacement and vertical roll displacement are summed and processed to obtain the impact evaluation coefficient of the sway degree on the gimbal equipment.

[0013] Furthermore, the specific process of analyzing the pre-processed data related to the ship's swing amplitude is as follows: the vibration speed, vibration period, natural frequency and dynamic amplification factor of the ship are transmitted from the data acquisition and preprocessing module to the data analysis module, and the vibration speed, vibration period, natural frequency and dynamic amplification factor of the ship are summed and processed to obtain the impact evaluation coefficient of vibration on the gimbal equipment.

[0014] Furthermore, the specific comparison process of comparing the impact assessment coefficient of the sway degree on the gimbal device with the impact assessment threshold of the sway degree on the gimbal device is as follows: compare the impact assessment coefficient of the sway degree on the gimbal device with the impact assessment threshold of the sway degree on the gimbal device; if the impact assessment coefficient of the sway degree on the gimbal device is greater than or equal to the impact assessment threshold of the sway degree on the gimbal device, perform preliminary optimization of the dynamic gimbal system through the optimization module according to the threshold comparison result of the impact assessment coefficient of the sway degree on the gimbal device; if the impact assessment coefficient of the sway degree on the gimbal device is less than the impact assessment threshold of the sway degree on the gimbal device, mark the gimbal device as not requiring optimization.

[0015] Furthermore, the specific comparison process of comparing the jitter evaluation coefficient of the gimbal device with the jitter evaluation threshold of the gimbal device is: comparing the jitter evaluation coefficient of the gimbal device with the jitter evaluation threshold of the gimbal device; if the jitter evaluation coefficient of the gimbal device is greater than or equal to the jitter evaluation threshold of the gimbal device, the dynamic gimbal system is finally optimized through the optimization module according to the threshold comparison result of the jitter evaluation coefficient of the gimbal device; if the jitter evaluation coefficient of the gimbal device is less than the jitter evaluation threshold of the gimbal device, the gimbal device is marked as not requiring optimization.

[0016] Furthermore, according to the threshold comparison result of the evaluation coefficient of the impact of the sway degree on the gimbal device, the specific optimization process of the dynamic gimbal system is preliminarily optimized through the optimization module: the preliminary optimization includes optical image stabilization processing of the gimbal device and adjustment and calibration of the sensor; optical image stabilization processing of the gimbal device: actively adjusting the optical components to achieve adaptive adjustment of the optical path, thereby compensating for the jitter caused by the high-frequency vibration of the camera; adjusting and calibrating the sensor: adjusting the exposure time of the sensor, color correction, and calibrating the bad pixels and color distortion on the sensor.

[0017] Furthermore, the specific optimization process of finally optimizing the dynamic gimbal system through the optimization module according to the threshold comparison result of the jitter evaluation coefficient of the gimbal device is as follows: the final optimization refers to the introduction of an inertia compensation algorithm in the electronic image stabilization. The inertia compensation algorithm calculates the speed and position changes of the object and predicts its future motion trajectory, thereby compensating for the instability caused by the motion.

[0018] Furthermore, the specific method for obtaining the jitter evaluation coefficient of the PTZ device is: JEC = ln[sec- 1 (SD*σ1+VOG*σ2+1)]; where JEC is the jitter evaluation coefficient of the gimbal device, SD is the impact evaluation coefficient of the sway degree on the gimbal device, VOG is the impact evaluation coefficient of the vibration on the gimbal device, σ1 is the weight factor of the impact evaluation coefficient of the sway degree on the gimbal device, and σ2 is the weight factor of the impact evaluation coefficient of the vibration on the gimbal device.

[0019] One or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages:

[0020] 1. By comparing the jitter assessment coefficient of the pan-tilt device with the jitter assessment threshold of the pan-tilt device, the threshold comparison result of the jitter assessment coefficient of the pan-tilt device is obtained, and then the dynamic pan-tilt system is finally optimized through the optimization module according to the threshold comparison result of the jitter assessment coefficient of the pan-tilt device, thereby achieving the effect of improving the monitoring quality of the pan-tilt device, and effectively solving the problem of reduced monitoring quality of the pan-tilt device in the prior art.

[0021] 2. By comprehensively evaluating the impact evaluation coefficients of the sway degree on the gimbal device and the vibration on the gimbal device, the jitter evaluation coefficient of the gimbal device is obtained, thereby achieving the effect of improving the response speed of the gimbal device, and effectively solving the problem of insufficient response speed of the gimbal device in the prior art.

[0022] 3. By preprocessing the ship swing amplitude related data and the ship vibration related data, the preprocessed ship swing amplitude related data and the preprocessed ship vibration related data are obtained, thereby achieving the effect of reducing the calculation complexity and effectively solving the problem of high calculation complexity in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 A schematic diagram of the structure of a dynamic pan-tilt control system of a Skynet system with MEMS optical and inertial fusion provided in an embodiment of the present application;

[0024] Figure 2This is an image of the evaluation coefficient of the impact of vibration on the gimbal device in the Skynet system dynamic gimbal control system with MEMS optical and inertial fusion provided in an embodiment of the present application. DETAILED DESCRIPTION

[0025] The embodiment of the present application solves the problem of reduced monitoring quality of pan-tilt devices in the prior art by providing a Skynet system dynamic pan-tilt control system that integrates MEMS optics and inertia. By comparing the jitter assessment coefficient of the pan-tilt device with the jitter assessment threshold of the pan-tilt device, a threshold comparison result of the jitter assessment coefficient of the pan-tilt device is obtained. The dynamic pan-tilt system is finally optimized through an optimization module based on the threshold comparison result of the jitter assessment coefficient of the pan-tilt device, thereby achieving the effect of improving the monitoring quality of the pan-tilt device.

[0026] The technical solution in the embodiment of the present application is to solve the problem of decreased monitoring quality of the above-mentioned PTZ equipment, and the overall idea is as follows:

[0027] The Skynet system dynamic pan-tilt control system that integrates MEMS optics and inertia collects and pre-processes ship sway and vibration data through sensors, analyzes the impact assessment coefficient, compares it with the assessment threshold in the database, and performs preliminary and final optimization of the dynamic pan-tilt system through the optimization module to reduce the impact of ship motion on the pan-tilt stability, thereby improving the monitoring quality of the pan-tilt equipment.

[0028] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0029] like Figure 1As shown, it is a structural schematic diagram of the Skynet system dynamic pan-tilt control system of the MEMS optical and inertial fusion provided in the embodiment of the present application. The Skynet system dynamic pan-tilt control system of the MEMS optical and inertial fusion provided in the embodiment of the present application includes: a data collection and preprocessing module, a data analysis module, a threshold comparison module and an optimization module: Data collection and preprocessing module: collects ship sway amplitude related data and ship vibration related data, preprocesses the ship sway amplitude related data and the ship vibration related data to obtain the preprocessed ship sway amplitude related data and the preprocessed ship vibration related data, and obtains the impact evaluation threshold of the sway degree on the pan-tilt device and the jitter evaluation threshold of the pan-tilt device from the database; Data analysis module: analyzes the preprocessed ship sway amplitude related data to obtain the impact evaluation coefficient of the sway degree on the pan-tilt device, and performs the preprocessed ship vibration evaluation on the preprocessed ship vibration. The dynamic related data are analyzed to obtain the impact assessment coefficient of vibration on the gimbal device, and the impact assessment coefficient of the sway degree on the gimbal device and the impact assessment coefficient of vibration on the gimbal device are comprehensively evaluated to obtain the jitter assessment coefficient of the gimbal device; threshold comparison module: the impact assessment coefficient of the sway degree on the gimbal device is compared with the impact assessment threshold of the sway degree on the gimbal device to obtain the threshold comparison result of the impact assessment coefficient of the sway degree on the gimbal device, the jitter assessment coefficient of the gimbal device is compared with the jitter assessment threshold of the gimbal device to obtain the threshold comparison result of the jitter assessment coefficient of the gimbal device; optimization module: according to the threshold comparison result of the impact assessment coefficient of the sway degree on the gimbal device, the dynamic gimbal system is preliminarily optimized through the optimization module, and according to the threshold comparison result of the jitter assessment coefficient of the gimbal device, the dynamic gimbal system is finally optimized through the optimization module.

[0030] Furthermore, the specific process of collecting ship sway amplitude related data and ship vibration related data is as follows: ship sway amplitude related data include roll angle, pitch angle, vertical roll angle, roll displacement, pitch displacement and vertical roll displacement; ship vibration related data include vibration velocity, vibration period, natural frequency and dynamic amplification factor.

[0031] In this embodiment, the roll angle refers to the angle at which the gimbal is offset from the horizontal axis when affected by environmental factors such as waves, and can be obtained through a gyroscope, a sensor, and an accelerometer in a micro-electromechanical system (MEMS).

[0032] The pitch angle refers to the angle at which the gimbal deviates from the vertical longitudinal axis when affected by environmental factors such as waves, and can be obtained through the gyroscope, sensor and accelerometer in the micro-electromechanical system.

[0033] The pitch angle refers to the angle at which the gimbal deviates from the vertical axis when affected by waves and other environmental factors, which can be obtained through the gyroscopes, sensors and accelerometers in the micro-electromechanical system.

[0034] Roll displacement refers to the linear displacement of the gimbal in the roll direction when affected by waves and other environmental factors, which can be obtained through the gyroscope, sensor and accelerometer in the micro-electromechanical system.

[0035] Pitch displacement refers to the linear displacement of the gimbal in the pitch direction when affected by waves and other environmental factors, which can be obtained through the gyroscope, sensor and accelerometer in the micro-electromechanical system.

[0036] The pitch displacement refers to the linear displacement of the gimbal in the pitch direction when affected by waves and other environmental factors, which can be obtained through the gyroscope, sensor and accelerometer in the micro-electromechanical system.

[0037] Accelerometers include liquid-floating pendulum accelerometers, gyro-integrated accelerometers and flexible pendulum accelerometers, and gyroscopes refer to MEMS gyroscopes.

[0038] Furthermore, the specific process of preprocessing the ship roll amplitude related data and the ship vibration related data is: checking the timestamps of the sensors on the ship, identifying and eliminating outliers, filling in missing values, using a low-pass filter to remove high-frequency noise, retaining low-frequency ship roll amplitude related data and ship vibration related data, and normalizing the ship roll amplitude related data and ship vibration related data.

[0039] In this embodiment, for example, there is a series of ship sensor data as shown in Table 1.

[0040] Table 1 Ship sensor data example table

[0041] Timestamp (seconds) Roll angle (degrees) Pitch angle (degrees) Heave angle (degrees) 0 1.2 0.5 -0.3 1 1.3 0.6 -0.4 2 NaN 0.7 -0.4 3 1.5 0.8 -0.5 4 1.6 0.9 -0.6 5 10.0 1.0 -0.7 6 1.7 NaN -0.8 7 1.8 1.1 -0.9

[0042] In the table, NaN indicates that the value is missing and needs to be filled. The roll angle of the 5th timestamp is 10.0 degrees, which is too different from the roll angles of other timestamps. It is judged as an abnormal value and needs to be removed.

[0043] Check the timestamps to ensure that they are sequentially increasing and there are no missing timestamps.

[0044] Outliers may be caused by sensor failure or external interference. The specific steps to remove outliers using Python are:

[0045]

[0046]

[0047] For missing data, linear interpolation or front and back value filling can be used to fill it. The specific steps are as follows:

[0048]

[0049] A simple moving average filter is used as a low-pass filter in Python. The specific process is:

[0050]

[0051]

[0052] The data in filtered_data is now considered to be low-frequency.

[0053] The specific method of using the minimum-maximum standardization method in Python to normalize data to the [0,1] interval is:

[0054]

[0055] Furthermore, the specific process of analyzing the pre-processed data related to the ship's sway amplitude is as follows: the ship's roll angle, pitch angle, vertical roll angle, roll displacement, pitch displacement and vertical roll displacement are transmitted from the data acquisition and preprocessing module to the data analysis module, and the ship's roll angle, pitch angle, vertical roll angle, roll displacement, pitch displacement and vertical roll displacement are summed and processed to obtain the impact evaluation coefficient of the sway degree on the gimbal equipment.

[0056] In this embodiment, the specific method for obtaining the evaluation coefficient of the impact of the swing degree on the pan / tilt device is:

[0057] SD=sec -1 {ln[(RA+PA+YA)*δ1]+(RD+PD+YD)*δ2};

[0058] Wherein, SD is the impact assessment coefficient of the sway degree on the gimbal device, RA is the average roll angle within the preset time period, PA is the average pitch angle within the preset time period, YA is the average vertical roll angle within the preset time period, RD is the average roll displacement within the preset time period, PD is the average pitch displacement within the preset time period, YD is the average vertical roll displacement within the preset time period, δ1 is the weight factor of the angle caused by the offset of roll, pitch and vertical roll, and δ2 is the weight factor of the displacement caused by the offset of roll, pitch and vertical roll.

[0059] The average roll angle within the preset time period refers to the average value of the sum of all roll angles within the preset time period.

[0060] The average pitch angle within the preset time period refers to the average value of the sum of all pitch angles within the preset time period.

[0061] The average pitch angle within the preset time period refers to the average value of the sum of all pitch angles within the preset time period.

[0062] The average roll displacement within the preset time period refers to the average value of the sum of all roll displacements within the preset time period.

[0063] The average pitch displacement within the preset time period refers to the average value of the sum of all pitch angles within the preset time period.

[0064] The average vertical displacement within the preset time period refers to the average value of the sum of all vertical displacements within the preset time period.

[0065] The weight factors of the angles of offset caused by roll, pitch and vertical roll and the weight factors of the displacements of offset caused by roll, pitch and vertical roll can be obtained through the real-time dynamic database of the ship's sway degree, indicating the proportion of the angles of offset caused by roll, pitch and vertical roll and the displacements of offset caused by roll, pitch and vertical roll in the evaluation coefficient of the impact of the sway degree on the pan / tilt equipment. The weight factors of the angles of offset caused by roll, pitch and vertical roll and the weight factors of the displacements of offset caused by roll, pitch and vertical roll can also be obtained through a mapping relationship, such as through the sway degree in historical data The relationship between the impact assessment coefficient of the gimbal device and the angle of offset caused by roll, pitch and yaw and the displacement of the offset caused by roll, pitch and yaw is established by respectively establishing a mapping set of the angle of offset caused by roll, pitch and yaw and the displacement of the offset caused by roll, pitch and yaw and their corresponding weights, and the weight factor of the angle of offset caused by roll, pitch and yaw and the weight factor of the displacement of the offset caused by roll, pitch and yaw corresponding to the mapping set are obtained by inputting the real-time angle of offset caused by roll, pitch and yaw and the displacement of the offset caused by roll, pitch and yaw.

[0066] Changes in the roll angle, pitch angle and vertical roll angle directly lead to corresponding changes in the roll displacement, pitch displacement and vertical roll displacement. Changes in the vertical roll angle and vertical roll displacement will simultaneously affect the stability of the roll and pitch roll, thereby affecting the roll angle, pitch angle, roll displacement and pitch displacement. When the ship rolls, the control system may need to adjust the compensation of the pitch and vertical roll to keep the gimbal stable.

[0067] Furthermore, the specific process of analyzing the pre-processed data related to the ship's swing amplitude is as follows: the vibration speed, vibration period, natural frequency and dynamic amplification factor of the ship are transmitted from the data acquisition and preprocessing module to the data analysis module, and the vibration speed, vibration period, natural frequency and dynamic amplification factor of the ship are summed and processed to obtain the impact evaluation coefficient of vibration on the gimbal equipment.

[0068] In this embodiment, the specific method for obtaining the evaluation coefficient of the impact of vibration on the pan / tilt device is:

[0069]

[0070] Wherein, VOG is the impact assessment coefficient of vibration on the PTZ equipment, VE is the average vibration velocity within the preset time period, VD is the average vibration period within the preset time period, NF is the average natural frequency within the preset time period, DMF is the average dynamic amplification factor within the preset time period, ε1 is the weight factor of the vibration velocity, ε2 is the weight factor of the vibration period, ε3 is the weight factor of the natural frequency, and ε4 is the weight factor of the dynamic amplification factor.

[0071] The average vibration velocity within a preset time period refers to the average value of the vibration velocity of the ship during navigation within the set time period. The vibration can be measured by the MEMS acceleration sensor installed on the ship. The acceleration signal is integrated to obtain the velocity signal. Within the preset time period, all velocity data are averaged to obtain the average vibration velocity.

[0072] The average vibration period within the preset time period refers to the average value of the ship's vibration period (i.e. the time required to complete a complete vibration cycle) within the set time period. The main frequency of the vibration can be determined by performing spectral analysis (such as FFT) on the vibration signal, and then calculating its corresponding period. All periodic data are averaged within the preset time period.

[0073] The average natural frequency within the preset time period refers to the average value of the frequency of free vibration of the ship without other external excitations before leaving the shore within the set time period, which can be obtained through the database.

[0074] The average power amplification factor within a preset time period refers to the average value of the ratio of the response amplitude to the excitation amplitude of the ship at a specific excitation frequency within the set time period, which can be directly obtained through the database.

[0075] In this embodiment, examples of evaluation coefficient data of the impact of vibration on the pan-tilt device are shown in the following table.

[0076] Table 2 Example of evaluation coefficient data for vibration impact on PTZ equipment

[0077]

[0078] When the weight factor of vibration velocity, the weight factor of vibration period, the weight factor of natural frequency and the weight factor of dynamic amplification factor are 0.2, 0.3, 0.1 and 0.4 respectively, Figure 2 As shown, it is an image of the evaluation coefficient of the impact of vibration on the pan-tilt device in the dynamic pan-tilt control system of the Skynet system with MEMS optical and inertial fusion provided in the embodiment of the present application, through Figure 2It can be seen from the data in Table 2 that when the average vibration period within the preset time period, the average natural frequency within the preset time period and the average dynamic amplification factor within the preset time period remain unchanged, the greater the average vibration speed within the preset time period, the greater the evaluation coefficient of the impact of vibration on the gimbal equipment.

[0079] The weight factor of vibration velocity, the weight factor of vibration period, the weight factor of natural frequency and the weight factor of power amplification coefficient can be obtained through the network database, which represents the proportion of vibration velocity, vibration period, natural frequency and power amplification coefficient in the vibration impact assessment coefficient on the pan-tilt device. The weight factor of vibration velocity, the weight factor of vibration period, the weight factor of natural frequency and the weight factor of power amplification coefficient can also be obtained through mapping relationships. For example, through the relationship between vibration velocity, vibration period, natural frequency and power amplification coefficient and the vibration impact assessment coefficient on the pan-tilt device in the historical database, a mapping set of vibration velocity, vibration period, natural frequency and power amplification coefficient and their corresponding weights is established respectively. By inputting the real-time vibration velocity, vibration period, natural frequency and power amplification coefficient, the corresponding vibration velocity weight factor, vibration period weight factor, natural frequency weight factor and power amplification coefficient weight factor in the mapping set are obtained.

[0080] There is a mutual influence relationship between the vibration velocity and the vibration period. The greater the vibration velocity, the smaller the vibration period. The vibration velocity and the vibration period describe the motion state of the vibration. The natural frequency is the frequency at which the sensor measures the free vibration of the ship before the structure leaves the shore. When the external excitation frequency is close to the natural frequency of the structure, the vibration period and velocity will increase significantly. The dynamic amplification factor describes the degree of amplification of the structural response (displacement or stress) relative to the static response at a specific frequency. When the excitation frequency is close to the natural frequency, the dynamic amplification factor will increase.

[0081] Furthermore, the specific comparison process of comparing the impact assessment coefficient of the sway degree on the gimbal device with the impact assessment threshold of the sway degree on the gimbal device is as follows: compare the impact assessment coefficient of the sway degree on the gimbal device with the impact assessment threshold of the sway degree on the gimbal device; if the impact assessment coefficient of the sway degree on the gimbal device is greater than or equal to the impact assessment threshold of the sway degree on the gimbal device, perform preliminary optimization of the dynamic gimbal system through the optimization module according to the threshold comparison result of the impact assessment coefficient of the sway degree on the gimbal device; if the impact assessment coefficient of the sway degree on the gimbal device is less than the impact assessment threshold of the sway degree on the gimbal device, mark the gimbal device as not requiring optimization.

[0082] In this embodiment, for example, the impact assessment threshold of the sway degree on the pan-tilt device is set to 2. By analyzing the ship's roll angle, pitch angle, vertical roll angle, roll displacement, pitch displacement and vertical roll displacement, the impact assessment coefficient of the sway degree on the pan-tilt device is obtained to be 1.5. At this time, the impact assessment coefficient of the sway degree on the pan-tilt device is less than the impact assessment threshold of the sway degree on the pan-tilt device, which means that the sway degree is not sufficient to affect the pan-tilt device. If the ship's roll angle, pitch angle, vertical roll angle, roll displacement, pitch displacement and vertical roll displacement are analyzed, the impact assessment coefficient of the sway degree on the pan-tilt device is obtained to be 2.5. At this time, the impact assessment coefficient of the sway degree on the pan-tilt device is greater than the impact assessment threshold of the sway degree on the pan-tilt device, which means that the dynamic pan-tilt system needs to be preliminarily optimized through the optimization module.

[0083] Furthermore, the specific comparison process of comparing the jitter evaluation coefficient of the gimbal device with the jitter evaluation threshold of the gimbal device is: comparing the jitter evaluation coefficient of the gimbal device with the jitter evaluation threshold of the gimbal device; if the jitter evaluation coefficient of the gimbal device is greater than or equal to the jitter evaluation threshold of the gimbal device, the dynamic gimbal system is finally optimized through the optimization module according to the threshold comparison result of the jitter evaluation coefficient of the gimbal device; if the jitter evaluation coefficient of the gimbal device is less than the jitter evaluation threshold of the gimbal device, the gimbal device is marked as not requiring optimization.

[0084] In this embodiment, for example, the jitter evaluation coefficient of the gimbal device is set to 1.5. By analyzing the impact evaluation coefficient of the ship's swaying degree on the gimbal device and the impact evaluation coefficient of the vibration on the gimbal device, the jitter evaluation coefficient of the gimbal device is obtained to be 1.3. At this time, the jitter evaluation coefficient of the gimbal device is less than the jitter evaluation threshold of the gimbal device, which means that the jitter of the gimbal device due to waves and other environmental factors is not enough to affect the gimbal device. If the impact evaluation coefficient of the ship's swaying degree on the gimbal device and the impact evaluation coefficient of the vibration on the gimbal device are analyzed, the jitter evaluation coefficient of the gimbal device is obtained to be 2.5. At this time, the jitter evaluation coefficient of the gimbal device is greater than the jitter evaluation threshold of the gimbal device, which means that the dynamic gimbal system needs to be finally optimized through the optimization module.

[0085] Furthermore, according to the threshold comparison result of the evaluation coefficient of the impact of the sway degree on the gimbal device, the specific optimization process of the dynamic gimbal system is preliminarily optimized through the optimization module: the preliminary optimization includes optical image stabilization processing of the gimbal device and adjustment and calibration of the sensor; optical image stabilization processing of the gimbal device: actively adjusting the optical components to achieve adaptive adjustment of the optical path, thereby compensating for the jitter caused by the high-frequency vibration of the camera; adjusting and calibrating the sensor: adjusting the exposure time of the sensor, color correction, and calibrating the bad pixels and color distortion on the sensor.

[0086] The jitter signal of the gimbal device is collected by the gyroscope and transmitted to the processor. In the processor, the time-frequency characteristics of the jitter signal are analyzed by short-time Fourier transform to obtain the jitter amount of the gimbal device. According to the jitter amount obtained by analysis, the position of the optical path in the lens is adjusted on the X and Y axes to reduce the impact of the jitter on the gimbal device.

[0087] Adjusting and calibrating the sensor includes increasing or decreasing the exposure time to adapt to changes in ambient light, adjusting the gain and offset of the RGB channels, detecting bad pixels through the sparse single-term mixed total change algorithm, compensating for them during image processing, and adjusting the RGB values ​​using a color correction matrix.

[0088] For example, the specific process of actively adjusting optical components to achieve adaptive adjustment of the optical path is as follows: the MEMS gyroscope on the gimbal device collects angular velocity data of the X-axis and Y-axis in real time and transmits it to the processor. After the processor receives the jitter signal, it uses short-time Fourier transform (STFT) to analyze the signal to obtain the jitter amount of the gimbal device, obtains the mapping set between the jitter amount of the gimbal device and the adjustment amount of the optical path in the lens through the database, inputs the jitter amount of the gimbal device, and outputs the adjustment amount of the optical path in the lens. Then, the processor directs the optical path of the lens in the gimbal device to move to offset the impact of the jitter.

[0089] The specific process of adjusting and calibrating the sensor is as follows: according to the ambient light intensity, obtain the exposure time adjustment value corresponding to the light intensity from the database, and adjust the exposure time. For example, on a sunny day, the light intensity is 1000 lux, and the exposure time may need to be shortened by 0.4ms to avoid overexposure. At dusk or on a cloudy day, the light intensity is 200lx, and the exposure time may need to be increased by 0.2ms to obtain sufficient brightness. Take a picture at night or under light-shielded conditions, use Adobe Photoshop, GIMP, and MATLAB to determine the gain and bias of the RGB channels, and apply the obtained gain and bias of the RGB channels to the color settings of the camera. Take a picture of all white or all black, identify dead pixels, use Dead PixelTest to record the location of the dead pixels, and correct them during image processing. Correct color distortion through the color correction matrix, such as a color channel that is too saturated or insensitive.

[0090] In this embodiment, it is assumed that the detected high-frequency vibration data is as follows:

[0091] Roll Angular Velocity: 0.5 rad / s

[0092] Pitch Angular Velocity: 0.3rad / s

[0093] Yaw Angular Velocity: 0.2rad / s

[0094] Based on the detected vibration data, actively adjust the optical components to compensate for jitter. The specific steps are as follows:

[0095] def adjust_opt ics(rol l_angular_veloci ty,pi tch_angular_veloci ty,yaw_angular_veloci ty):

[0096] #Assumption function used to adjust optical components

[0097] print(f"Adjust ing opt ics for rol l:{rol l_angular_veloci ty},pitch:{pi tch_angular_veloci ty},yaw:{yaw_angular_veloci ty}")

[0098] #The actual operation may involve precision motor control to move the floating lens or optical stabilizer adjust_optics(0.5,0.3,0.2)

[0099] #Call function for optical image stabilization

[0100] Adjust the sensor exposure time according to the ambient light conditions. Assuming that the current ambient light is dim, the exposure time needs to be increased. The specific steps are as follows:

[0101] def set_exposure_t ime(exposure_t ime_ms):

[0102] #Assume function to set exposure time

[0103] print(f"Sett ing exposure t ime to{exposure_t ime_ms}ms")

[0104] #Actual operation may involve modifying sensor registers

[0105] set_exposure_t ime(30) #increase to 30 milliseconds

[0106] Specific steps to calibrate the color response of the sensor:

[0107] def cal ibrate_color_response():

[0108] #Assume function for color correction

[0109] print("Calibration color response")

[0110] #Actual operation may involve complex algorithms and test pattern analysis

[0111] cal ibrate_color_response()

[0112] Specific steps to identify and correct dead pixels and color distortion on the sensor:

[0113] def cal ibrate_bad_pixels_and_color_distort ion():

[0114] #Assumption function for calibrating bad pixels and color distortion

[0115] print("Cal ibrat ing bad pixels and color distort ion")

[0116] #The actual operation may involve image processing algorithms to identify and correct bad pixels

[0117] cal ibrate_bad_pixels_and_color_distort ion()

[0118] Furthermore, the specific optimization process of finally optimizing the dynamic gimbal system through the optimization module according to the threshold comparison result of the jitter evaluation coefficient of the gimbal device is as follows: the final optimization refers to the introduction of an inertia compensation algorithm in the electronic image stabilization. The inertia compensation algorithm calculates the speed and position changes of the object and predicts its future motion trajectory, thereby compensating for the instability caused by the motion.

[0119] In this embodiment, the inertia compensation algorithm uses a navigation coordinate system and a carrier coordinate system. Navigation coordinate system (n system): Generally, the local horizontal coordinate system (g system) is selected, and XYZ points to the east, north, and sky respectively. Carrier coordinate system (b system): The center of the carrier is the coordinate origin, the carrier's horizontal axis to the right is the X axis, the carrier's vertical axis forward is the Y axis, and the carrier's vertical axis upward is the Z axis.

[0120] Inertial sensors are generally directly connected to the target carrier. The gyroscope measures the carrier's angular velocity information and calculates the attitude transformation matrix of the carrier coordinate system relative to the navigation coordinate system. On this basis, the acceleration information in the accelerometer is converted to the navigation coordinate system, and the system motion speed and position parameters are integrated. At the same time, the acceleration is used to correct the carrier's attitude and extract the carrier's attitude information to achieve position and attitude estimation of the target carrier.

[0121] The errors of the inertial compensation algorithm include zero bias error, proportional factor error, i.e. scale error, and non-orthogonal error, i.e. installation error. Error compensation techniques include zero bias correction: Accelerometers and gyroscopes often have zero bias errors, i.e., the value output by the sensor is not zero without external force or angular velocity. The influence of these errors can be reduced by performing zero bias calibration in a static state. Scale factor correction: The measured values ​​of accelerometers and gyroscopes are sometimes affected by scale factor errors, i.e., there is a proportional relationship between the measured value and the actual value. The influence of this error can be reduced by performing scale factor correction. Static alignment: Static alignment refers to estimating the attitude by measuring the gravity acceleration and the geomagnetic field strength in a static state, and then compensating the sensor output according to the estimated attitude. This can reduce the influence of attitude error on navigation results. Dynamic alignment: Dynamic alignment refers to estimating the attitude and sensor error and compensating for it by using the position and velocity information provided by other auxiliary sensors (such as GPS) in a moving state. Kalman filter: Kalman filter is a commonly used state estimation method that can use prior information and measurement values ​​to estimate the state of the system and effectively compensate for sensor errors. In inertial navigation, Kalman filtering can be used to fuse inertial measurement and other auxiliary sensor information to improve navigation accuracy. This application example mainly uses zero bias correction. By performing zero bias calibration in a stationary state, the impact of errors can be reduced.

[0122] Furthermore, the specific method for obtaining the jitter evaluation coefficient of the PTZ device is: JEC = ln[sec -1 (SD*σ1+VOG*σ2+1)]; where JEC is the jitter evaluation coefficient of the gimbal device, SD is the impact evaluation coefficient of the sway degree on the gimbal device, VOG is the impact evaluation coefficient of the vibration on the gimbal device, σ1 is the weight factor of the impact evaluation coefficient of the sway degree on the gimbal device, and σ2 is the weight factor of the impact evaluation coefficient of the vibration on the gimbal device.

[0123] In this embodiment, the weight factor of the impact assessment coefficient of the sway degree on the pan-tilt device and the weight factor of the impact assessment coefficient of the vibration on the pan-tilt device can be obtained through a real-time dynamic database of the jitter of the pan-tilt device, which represents the proportion of the impact assessment coefficient of the sway degree on the pan-tilt device and the weight factor of the impact assessment coefficient of the vibration on the pan-tilt device in the jitter assessment coefficient of the pan-tilt device. The weight factor of the impact assessment coefficient of the sway degree on the pan-tilt device and the weight factor of the impact assessment coefficient of the vibration on the pan-tilt device can also be obtained through a mapping relationship. For example, a mapping set of the impact assessment coefficient of the sway degree on the pan-tilt device and the impact assessment coefficient of the vibration on the pan-tilt device and their corresponding weights is established respectively through the relationship between the jitter assessment coefficient of the pan-tilt device and the impact assessment coefficient of the sway degree on the pan-tilt device and the impact assessment coefficient of the vibration on the pan-tilt device in the historical data. The weight factor of the impact assessment coefficient of the sway degree on the pan-tilt device and the weight factor of the impact assessment coefficient of the vibration on the pan-tilt device corresponding to the mapping set are obtained by inputting the real-time impact assessment coefficient of the sway degree on the pan-tilt device and the impact assessment coefficient of the vibration on the pan-tilt device.

[0124] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0125] The present invention is described with reference to flowcharts and / or block diagrams of systems, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0126] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1A function specified in one or more boxes.

[0127] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0128] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0129] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. The Skynet system dynamic PTZ control system integrating MEMS optics and inertia is characterized by: Including data collection and preprocessing module, data analysis module, threshold comparison module and optimization module: Data collection and preprocessing module: collects ship sway amplitude related data and ship vibration related data, preprocesses the ship sway amplitude related data and ship vibration related data to obtain preprocessed ship sway amplitude related data and preprocessed ship vibration related data, and obtains the impact assessment threshold of the sway degree on the pan-tilt device and the jitter assessment threshold of the pan-tilt device from the database; Data analysis module: Analyze the pre-processed ship sway amplitude related data to obtain the impact assessment coefficient of the sway degree on the PTZ equipment; analyze the pre-processed ship vibration related data to obtain the impact assessment coefficient of the vibration on the PTZ equipment; conduct a comprehensive assessment of the impact assessment coefficient of the sway degree on the PTZ equipment and the impact assessment coefficient of the vibration on the PTZ equipment to obtain the jitter assessment coefficient of the PTZ equipment; Threshold comparison module: compare the impact assessment coefficient of the sway degree on the PTZ device with the impact assessment threshold of the sway degree on the PTZ device to obtain the threshold comparison result of the impact assessment coefficient of the sway degree on the PTZ device; compare the jitter assessment coefficient of the PTZ device with the jitter assessment threshold of the PTZ device to obtain the threshold comparison result of the jitter assessment coefficient of the PTZ device; Optimization module: The dynamic gimbal system is initially optimized through the optimization module according to the threshold comparison result of the evaluation coefficient of the impact of the sway degree on the gimbal device, and the dynamic gimbal system is finally optimized through the optimization module according to the threshold comparison result of the jitter evaluation coefficient of the gimbal device.

2. The Skynet system dynamic PTZ control system integrating MEMS optics and inertia as claimed in claim 1, characterized in that: The specific process of collecting the ship sway amplitude related data and the ship vibration related data is as follows: The data related to the ship's sway amplitude include roll angle, pitch angle, vertical roll angle, roll displacement, pitch displacement and vertical roll displacement; Ship vibration related data include vibration velocity, vibration period, natural frequency and dynamic amplification factor.

3. The Skynet system dynamic PTZ control system integrating MEMS optics and inertia as claimed in claim 1, characterized in that: The specific process of preprocessing the ship sway amplitude related data and the ship vibration related data is as follows: Check the timestamps of the sensors on the ship, identify and eliminate outliers, fill in missing values, use a low-pass filter to remove high-frequency noise, retain low-frequency ship swing amplitude-related data and ship vibration-related data, and normalize the ship swing amplitude-related data and ship vibration-related data.

4. The Skynet system dynamic PTZ control system integrating MEMS optics and inertia as claimed in claim 1, characterized in that: The specific process of analyzing the pre-processed ship swing amplitude related data is as follows: The ship's roll angle, pitch angle, vertical roll angle, roll displacement, pitch displacement and vertical roll displacement are transmitted from the data acquisition and preprocessing module to the data analysis module, and the ship's roll angle, pitch angle, vertical roll angle, roll displacement, pitch displacement and vertical roll displacement are summed and processed to obtain the impact evaluation coefficient of the sway degree on the pan / tilt equipment.

5. The Skynet system dynamic pan-tilt control system integrating MEMS optics and inertia as claimed in claim 1, characterized in that: The specific process of analyzing the pre-processed ship swing amplitude related data is as follows: The vibration velocity, vibration period, natural frequency and dynamic amplification factor of the ship are transmitted from the data acquisition and preprocessing module to the data analysis module, and the vibration velocity, vibration period, natural frequency and dynamic amplification factor of the ship are summed and processed to obtain the impact evaluation coefficient of vibration on the pan-tilt equipment.

6. The Skynet system dynamic pan-tilt control system integrating MEMS optics and inertia as claimed in claim 1, characterized in that: The specific comparison process of comparing the impact assessment coefficient of the swing degree on the pan-tilt device with the impact assessment threshold of the swing degree on the pan-tilt device is as follows: The impact assessment coefficient of the sway degree on the gimbal device is compared with the impact assessment threshold of the sway degree on the gimbal device. If the impact assessment coefficient of the sway degree on the gimbal device is greater than or equal to the impact assessment threshold of the sway degree on the gimbal device, the dynamic gimbal system is preliminarily optimized through the optimization module according to the threshold comparison result of the impact assessment coefficient of the sway degree on the gimbal device. If the impact assessment coefficient of the sway degree on the gimbal device is less than the impact assessment threshold of the sway degree on the gimbal device, the gimbal device is marked as not requiring optimization.

7. The Skynet system dynamic pan-tilt control system integrating MEMS optics and inertia as claimed in claim 1, characterized in that: The specific comparison process of comparing the jitter evaluation coefficient of the pan / tilt device with the jitter evaluation threshold of the pan / tilt device is as follows: The jitter assessment coefficient of the gimbal device is compared with the jitter assessment threshold of the gimbal device. If the jitter assessment coefficient of the gimbal device is greater than or equal to the jitter assessment threshold of the gimbal device, the dynamic gimbal system is finally optimized through the optimization module according to the threshold comparison result of the jitter assessment coefficient of the gimbal device. If the jitter assessment coefficient of the gimbal device is less than the jitter assessment threshold of the gimbal device, the gimbal device is marked as not requiring optimization.

8. The Skynet system dynamic pan-tilt control system integrating MEMS optics and inertia as claimed in claim 1, characterized in that: The specific optimization process of performing preliminary optimization of the dynamic pan-tilt system through the optimization module according to the threshold comparison result of the impact evaluation coefficient of the swing degree on the pan-tilt device is as follows: Initial optimization includes optical image stabilization of the gimbal equipment and adjustment and calibration of the sensor; Optical image stabilization for gimbal equipment: Actively adjust optical components to achieve adaptive adjustment of the optical path, thereby compensating for jitter caused by high-frequency vibration of the camera; Adjust and calibrate the sensor: adjust the sensor’s exposure time, color correction, calibrate dead pixels and color distortion on the sensor.

9. The Skynet system dynamic PTZ control system integrating MEMS optics and inertia as claimed in claim 1, characterized in that: The specific optimization process of finally optimizing the dynamic pan-tilt system through the optimization module according to the threshold comparison result of the jitter evaluation coefficient of the pan-tilt device is as follows: The final optimization refers to the introduction of an inertia compensation algorithm in the electronic image stabilization. The inertia compensation algorithm calculates the speed and position changes of the object and predicts its future motion trajectory, thereby compensating for the instability caused by the motion.

10. The Skynet system dynamic pan-tilt control system integrating MEMS optics and inertia as claimed in claim 1, characterized in that: The specific method for obtaining the jitter evaluation coefficient of the pan / tilt device is as follows: JEC=ln [ seconds -1 (SD*σ1+VOG*σ2+1) ] ; Wherein, JEC represents the jitter evaluation coefficient of the gimbal device, SD represents the impact evaluation coefficient of the sway degree on the gimbal device, VOG represents the impact evaluation coefficient of the vibration on the gimbal device, σ1 represents the weight factor of the impact evaluation coefficient of the sway degree on the gimbal device, and σ2 represents the weight factor of the impact evaluation coefficient of the vibration on the gimbal device.

Images