FPGA-based multi-sensor data efficient acquisition and analysis processing method and system

Abstract

The application discloses a multi-sensor data efficient acquisition and analysis processing method and system based on FPGA, and belongs to the technical field of data processing, and specifically comprises the following steps: after starting data acquisition to obtain multi-sensor data, synchronous data is obtained by using PLL clock compensation, valid data is obtained through preprocessing and abnormal separation; then, data fusion reasoning is carried out based on the valid data, temporarily stored abnormal data and a preset function module, the hierarchical results are transmitted to an external PLC and WCS, and the abnormal data are stored to an FRAM; and according to the control instructions and optimization requirements fed back by the PLC and the WCS, model switching and data supplement verification are realized, and efficient acquisition, accurate processing and intelligent application of multi-sensor data are realized.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of data processing, and in particular relates to a multi-sensor data efficient acquisition and analysis processing method and system based on FPGA. BACKGROUND

[0002] In many fields such as industrial production and environmental monitoring, it is often necessary to simultaneously collect data from multiple types of sensors to obtain comprehensive and accurate information. Traditional data acquisition and analysis methods have problems such as difficulty in data synchronization, low processing efficiency, and difficulty in effectively fusing different modal data when facing multi-sensor data, which leads to the inability to timely and accurately extract valuable information, affecting the decision-making and control accuracy of the system. Therefore, it is of great practical significance to develop an efficient multi-sensor data acquisition and analysis processing method. SUMMARY

[0003] In view of the deficiencies of the prior art, the present application provides a multi-sensor data efficient acquisition and analysis processing method and system based on FPGA. After starting data acquisition to obtain multi-sensor data, synchronized data is obtained using PLL clock compensation, and valid data is obtained after preprocessing and abnormal separation. Then, based on the valid data, temporarily stored abnormal data, and preset function modules, data fusion reasoning is performed, the hierarchical results are transmitted to external PLC and WCS, and abnormal data is stored to FRAM. According to the control instructions and optimization requirements fed back by PLC and WCS, model switching and data supplement verification are realized, and efficient acquisition, accurate processing, and intelligent application of multi-sensor data are realized.

[0004] To achieve the above-mentioned purpose, the present application provides the following technical scheme:

[0005] The multi-sensor data efficient acquisition and analysis processing method based on FPGA comprises:

[0006] A hardware architecture is built. The hardware architecture includes a FPGA control unit, a multi-modal sensor group, a conveying line encoder, a two-dimensional code scanning module, a FRAM, and a redundant sensor. The FPGA control unit is pre-set with clock domain division, preprocessing logic, weighted gated attention module, INT8 quantization circuit, and lightweight model.

[0007] According to the hardware architecture, data acquisition is started and multi-sensor data is obtained.

[0008] According to the multi-sensor data and the clock domain division pre-set by the FPGA control unit, clock compensation is performed through PLL to obtain synchronized multi-sensor data.

[0009] According to the synchronized multi-sensor data and the pre-set pre-processing logic of the FPGA control unit, data pre-processing and abnormal separation are performed to obtain effective data; the effective data includes SWIR feature data, millimeter wave scattering map, RGB compressed map and filtered environmental data;

[0010] According to the effective data, the temporarily stored abnormal data and the function module pre-set by the FPGA control unit, data fusion reasoning is performed to obtain a hierarchical result, the hierarchical result is transmitted to the PLC and the WCS outside the hardware architecture, and the abnormal data temporarily stored in the temporary storage area is stored in the FRAM in the hardware architecture;

[0011] According to the control instructions and optimization requirements fed back by the PLC and the WCS based on the hierarchical result, model switching and data supplement verification are performed.

[0012] Specifically, according to the hardware architecture, data acquisition is started and multi-sensor data is acquired, including:

[0013] A trigger signal is generated by the two-dimensional code scanning module in the hardware architecture and is transmitted to the FPGA control unit, triggering the FPGA control unit to enter a data acquisition ready state;

[0014] The FPGA control unit acquires the conveying speed data transmitted by the conveying line encoder in the hardware architecture, adjusts the acquisition frequency of the multi-modal sensor group in the hardware architecture according to the conveying speed data, and sends a synchronization instruction to the multi-modal sensor group;

[0015] The multi-modal sensor group responds after receiving the synchronization instruction sent by the FPGA control unit, synchronously acquires RGB images, SWIR spectral data, millimeter wave A-scan data and environmental data, and obtains multi-sensor data.

[0016] Specifically, in the process of adjusting the acquisition frequency of the multi-modal sensor group in the hardware architecture according to the conveying speed data, a proportional-integral control algorithm is used, including:

[0017] The FPGA control unit calculates the speed deviation value by differentiating the real-time conveying speed data transmitted by the conveying line encoder from the pre-set reference speed; the pre-set reference speed is stored in the parameter storage area pre-set by itself;

[0018] The speed deviation value is converted into an acquisition frequency adjustment amount based on the proportional-integral control algorithm, wherein the proportional term coefficient is set according to the sensor response speed;

[0019] The calculated acquisition frequency adjustment amount is superimposed on the initial acquisition frequency of the multi-modal sensor group to obtain the adjusted target acquisition frequency, and the adjusted target acquisition frequency is sent to the multi-modal sensor group as one of the parameters of the synchronization instruction.

[0020] Specifically, according to the clock domain division preset by the FPGA control unit and the multi-sensor data, clock compensation is performed by the PLL to obtain synchronized multi-sensor data, including:

[0021] The FPGA control unit loads clock domain division parameters from a parameter storage area preset by itself, initializes the PLL based on the clock domain division parameters, and makes the PLL enter a state to be adjusted, and outputs the initialized PLL and the loaded clock domain division parameters; the clock domain division parameters include reference frequencies, phase relationships, and maximum deviation thresholds of the camera clock domain and the radar clock domain;

[0022] The FPGA control unit receives multi-sensor data, extracts clock characteristics of actual operation of each sensor from synchronization signals associated with the multi-sensor data in combination with the clock domain division parameters, compares them with reference clocks, analyzes frequency deviation and phase offset information, and outputs clock deviation data; the synchronization signals associated with the multi-sensor data include camera line synchronization signals and radar sampling trigger signals;

[0023] The FPGA control unit converts the output clock deviation data into control instructions and sends them to the initialized PLL, controls the PLL to adjust through internal phase detection, filtering, and voltage-controlled oscillation, and generates a compensation clock signal consistent with the reference clock frequency and phase alignment;

[0024] The FPGA control unit distributes the output compensation clock signal to the data processing module of the corresponding sensor, and the data processing module adjusts the sampling timing based on the compensation clock signal and outputs each sensor data with aligned sampling edges;

[0025] The FPGA control unit integrates the output sampling edge aligned data, adds a unified timestamp based on the compensation clock signal, and verifies the actual deviation of the data timestamp through an internal delay measurement circuit. If the actual deviation is within the preset deviation threshold, the synchronized multi-sensor data is output; otherwise, the clock characteristics are extracted again and the clock deviation is analyzed.

[0026] Specifically, in the process of generating a compensation clock signal consistent with the reference clock frequency and phase alignment by the control PLL through internal phase detection, filtering, and voltage-controlled oscillation adjustment, the phase adjustment of the PLL adopts a dynamic step mechanism, including:

[0027] The FPGA control unit first calls the deviation threshold and step parameters corresponding to the PLL phase adjustment from the parameter storage area preset by itself, including preset large deviation threshold, small deviation threshold, and corresponding preset large adjustment step, medium adjustment step, and small adjustment step;

[0028] The FPGA control unit extracts a phase offset value of the current multi-sensor clock and the reference clock from the clock deviation data, and compares the phase offset value with preset large deviation threshold and small deviation threshold;

[0029] When the phase offset value is greater than the preset large deviation threshold, the PLL uses a large adjustment step for phase compensation;

[0030] When the phase offset value is less than or equal to the large deviation threshold and greater than the small deviation threshold, a medium adjustment step is used;

[0031] When the phase offset value is less than or equal to the small deviation threshold, a small adjustment step is used;

[0032] The FPGA control unit converts the output target adjustment step into a control instruction recognized by the PLL and sends it to the initialized PLL. After receiving the control instruction, the PLL adjusts the phase of the internal voltage-controlled oscillator according to the target adjustment step, generates and outputs a preliminary compensated clock signal;

[0033] The FPGA control unit starts the real-time monitoring module, inputs the output preliminary compensated clock signal and the preset reference clock into the phase detection circuit, measures the phase difference between the two in real time, calculates a new phase offset value at a fixed period, updates the phase information in the clock deviation data, and outputs the updated phase offset value;

[0034] The FPGA control unit takes the output updated phase offset value as a new current phase offset value again, triggers the phase offset value comparison, step selection and PLL adjustment again until the updated phase offset value is stable and less than or equal to the small deviation threshold.

[0035] Specifically, the generation process of the effective data includes:

[0036] According to the synchronized multi-sensor data and the pre-processing logic preset by the FPGA control unit, the RGB image in the synchronized multi-sensor data is extracted, and the RGB image is adaptively compressed to obtain an RGB compressed image. The SWIR spectral data is extracted and sequentially subjected to smoothing filtering and PCA dimension reduction processing to obtain SWIR feature data. The millimeter wave A-scan data is extracted and subjected to parallel FFT processing to obtain a millimeter wave scatter diagram. The RGB compressed image, the SWIR feature data, the millimeter wave scatter diagram, and the environment data in the synchronized multi-sensor data are integrated to form full-quantity pre-processing data to be filtered. Based on the pre-processing logic preset by the FPGA control unit, the full-quantity pre-processing data to be filtered is subjected to outlier filtering, and the abnormal data is separated and temporarily stored in the temporary storage area of the FPGA control unit. The remaining data is the effective data. The outlier filtering adopts a combination of 3σ criterion and isolated forest algorithm.

[0037] Specifically, the temporary storage area of the FPGA control unit adopts a double-buffer queue design, including a first buffer partition and a second buffer partition.

[0038] The FPGA control unit writes the separated abnormal data into the first buffer partition, and when the storage amount of the first buffer partition reaches a preset threshold, automatically switches to the second buffer partition to continue writing the abnormal data, while batch-transferring the abnormal data in the first buffer partition to the FRAM; after the data transmission of the first buffer partition is completed, the first buffer partition is emptied, and waits for the next round of abnormal data writing, wherein the preset threshold is determined based on the writing rate of the FRAM and the generation rate of the abnormal data.

[0039] Specifically, according to the effective data, the temporarily stored abnormal data, and the function modules preset by the FPGA control unit, data fusion inference is performed to obtain a hierarchical result, including:

[0040] The SWIR feature data, the millimeter wave scattering map, and the RGB compressed map in the effective data are transmitted into the weighted gate attention module preset by the FPGA control unit, the contribution of the SWIR feature data, the millimeter wave scattering map, and the RGB compressed map to target classification is analyzed, the respective weight proportions are calculated and the fusion features are output;

[0041] Based on the filtered environmental data in the effective data, the data standardization processing logic in the FPGA control unit is called to perform numerical normalization operation on the filtered environmental data, and then the normalized environmental data is converted into an agricultural vector according to a preset feature mapping rule;

[0042] Based on the fusion features, the agricultural vector and the fusion features are combined and spliced according to a preset dimension order to form a spliced vector containing multi-dimensional information;

[0043] Taking the spliced vector as input, the INT8 quantization circuit in the FPGA control unit is called to compress the precision and convert the data format of the numerical values in the spliced vector, to obtain a quantized spliced vector, and the quantized spliced vector is transmitted into the lightweight model;

[0044] The lightweight model performs feature extraction, correlation analysis and category judgment on the multi-dimensional information in the quantized spliced vector according to the built-in inference algorithm, generates a hierarchical result reflecting the hierarchical situation of the target object through the model output layer, and transmits the output hierarchical result to the PLC and WCS outside the hardware architecture.

[0045] Specifically, according to the control instructions and optimization requirements fed back by the PLC and WCS based on the hierarchical result, model switching and data supplement verification are performed, including:

[0046] The FPGA control unit receives feedback information of the PLC and the WCS; the feedback information includes control instructions and optimization requirements for a current data processing flow;

[0047] The FPGA control unit calls multiple sets of lightweight models preset by itself based on the feedback information, filters out a lightweight model matched with the current feedback information through built-in model adaptability judgment logic, completes switching of the preset lightweight models, and obtains a target lightweight model adapted to the current scene;

[0048] After completing the model switching, the FPGA control unit sends a start instruction to a redundant sensor in the hardware architecture based on a requirement of the target lightweight model for data acquisition accuracy, and the redundant sensor collects supplementary data meeting the accuracy requirement of the target lightweight model for the current scene in response to the instruction and transmits the supplementary data to the FPGA control unit;

[0049] The FPGA control unit inputs the supplementary data into the target lightweight model as a verification benchmark, judges whether the supplementary data and effective data collected by the multiple sensors are matched in feature consistency and numerical accuracy through model inference, verifies the adaptation effect of the target lightweight model, and outputs a verification result including a data consistency conclusion and a model adaptability conclusion.

[0050] The FPGA-based multiple-sensor data efficient acquisition and analysis processing system comprises a data driving module, a clock synchronization module, a data processing module, a fusion inference module and an optimization and storage module.

[0051] The data driving module is configured to dynamically drive a multi-modal sensor group to collect multiple-sensor data according to conveying line encoder conveying speed data.

[0052] The clock synchronization module is configured to generate a compensation clock through a PLL to control sampling edge deviation of each sensor within a preset range and output synchronized multiple-sensor data.

[0053] The data processing module is configured to perform targeted processing on the synchronized multiple-sensor data, extract effective features and filter abnormal data, output effective data, and temporarily store abnormal data.

[0054] The fusion inference module is configured to fuse and infer the effective data, generate a hierarchical result reflecting a target object state, and transmit the hierarchical result to the PLC and the WCS outside the hardware architecture.

[0055] The optimization and storage module is configured to receive feedback information and optimize a running state, and store abnormal data.

[0056] Compared with the prior art, the present application has the following advantages:

[0057] 1.The application provides a FPGA-based multi-sensor data efficient acquisition and analysis processing system, and optimization and improvement are made on the architecture, operation steps and flow, and the system has the advantages of simple flow, low investment and operation cost, and low production cost.

[0058] 2.The application provides a FPGA-based multi-sensor data efficient acquisition and analysis processing method, which can efficiently realize the synchronization of multi-sensor data through the clock domain division preset by the FPGA control unit and the clock compensation by PLL, effectively solve the time deviation problem of data acquisition and processing under different clock domains, and improve the accuracy of data processing; at the same time, with the help of the preset preprocessing logic, the synchronized multi-sensor data can be quickly filtered, normalized and operated, the abnormal data can be accurately separated, and high-quality effective data such as SWIR feature data can be obtained, and the real-time performance of data processing is improved.

[0059] 3.The application provides a FPGA-based multi-sensor data efficient acquisition and analysis processing method, which uses the weighted gate attention module, INT8 quantization circuit and lightweight model in the FPGA control unit to intelligently fuse and infer the effective data and temporary abnormal data, can deeply mine the potential features of the data, accurately obtain the grading results and transmit them to the external PLC and WCS, and improves the intelligent level of the system; moreover, according to the control instructions and optimization requirements fed back by the PLC and WCS, the model switching and data supplement verification can be flexibly carried out, so that the system can quickly adapt to different operating environments and demand changes, and effectively improves the adaptability and reliability of the system. BRIEF DESCRIPTION OF DRAWINGS

[0060] Figure 1 Fig. 1 is a schematic diagram of the FPGA-based multi-sensor data efficient acquisition and analysis processing method of the application;

[0061] Figure 2 Fig. 2 is a principle flowchart of the FPGA-based multi-sensor data efficient acquisition and analysis processing method of the application. DETAILED DESCRIPTION

[0062] Example 1

[0063] Please refer to Figure 1 and Figure 2 , the application provides an embodiment: a FPGA-based multi-sensor data efficient acquisition and analysis processing method, the method comprises S1-S6, including the following steps:

[0064] S1: build a hardware architecture; the hardware architecture includes an FPGA control unit, a multi-modal sensor group, a conveying line encoder, a two-dimensional code scanning module, an FRAM, and a redundant sensor; the FPGA control unit is pre-configured with a clock domain division, a preprocessing logic, a weighted gated attention module, an INT8 quantization circuit, and a lightweight model;

[0065] S2: according to the hardware architecture, start data acquisition and obtain multi-sensor data;

[0066] S3: according to the multi-sensor data and the clock domain division pre-configured by the FPGA control unit, clock compensation is performed through the PLL to obtain synchronized multi-sensor data;

[0067] S4: according to the synchronized multi-sensor data and the preprocessing logic pre-configured by the FPGA control unit, data preprocessing and abnormal separation are performed to obtain valid data; the valid data includes SWIR feature data, millimeter wave scattering map, RGB compressed map, and filtered environmental data;

[0068] S5: according to the valid data, the temporarily stored abnormal data, and the function module pre-configured by the FPGA control unit, data fusion reasoning is performed to obtain hierarchical results, the hierarchical results are transmitted to the PLC and the WCS outside the hardware architecture, and the abnormal data temporarily stored in the temporary storage area is stored in the FRAM in the hardware architecture;

[0069] S6: according to the control instructions and optimization requirements fed back by the PLC and the WCS based on the hierarchical results, model switching and data supplement verification are performed.

[0070] PLL (Phase Locked Loop) is a phase-locked loop, FRAM (Ferroelectric Random Access Memory) is a memory, PLC (Programmable Logic Controller) is a programmable logic controller, and WCS (Warehouse Control System) is a warehouse control system, which is used to control intelligent factory equipment to perform tasks such as handling and production.

[0071] According to the hardware architecture, starting data acquisition and obtaining multi-sensor data includes:

[0072] S2.1: generate a trigger signal through the two-dimensional code scanning module in the hardware architecture and transmit it to the FPGA control unit to trigger the FPGA control unit to enter a data acquisition ready state;

[0073] Further, the specific steps of S2.1 include:

[0074] (1) The two-dimensional code scanning module calls initialization parameters from its own preset configuration storage area after power-on, and the initialization parameters include scanning frequency, two-dimensional code recognition accuracy threshold, trigger signal output format, and GPIO interface parameters connected with the FPGA control unit;

[0075] (2) After loading, the two-dimensional code scanning module starts internal self-checking, and performs function verification on the image sensor, light source driving circuit, and decoding chip, such as detecting whether the image sensor can output normal image signals and whether the light source can stably emit light. After confirming that there is no fault, it enters the scanning state, and at the same time, the state indicator light is lit to indicate readiness, and the scanning state and the corresponding state identifier are output;

[0076] (3) When the target object carrying the preset two-dimensional code enters the detection area of the two-dimensional code scanning module, the two-dimensional code scanning module starts working based on the output scanning state: the light source driving circuit emits 650nm infrared light according to the preset parameters to illuminate the surface of the two-dimensional code, and the image sensor captures the two-dimensional code image according to the scanning frequency, converts the analog image signal into a digital image signal, and transmits the digital image signal to the internal decoding chip. The digital image signal output to the decoding chip;

[0077] (4) The decoding chip receives the output digital image signal and performs two-dimensional code positioning and analysis: the edge detection algorithm is used to identify the position detection pattern of the two-dimensional code to determine the boundary, and then the black and white module sequence in the image is decoded according to the two-dimensional code encoding rule to extract the information contained in the two-dimensional code, such as the preset start identifier string. If the decoding result completely matches the preset start identifier, it is determined as a valid two-dimensional code. If the decoding fails or the information does not match, it is determined as invalid, no trigger signal is generated, and the scanning continues in step (3);

[0078] (5) When the output determination result is a valid two-dimensional code, the decoding chip sends a start instruction to the signal generation unit inside the two-dimensional code scanning module; the signal generation unit generates a pulse trigger signal according to the output format of the trigger signal, and transmits it to the preset GPIO input pin of the FPGA control unit through the special shielded twisted pair line. If the determination result is an invalid two-dimensional code, return to step (3) and continue to wait for the two-dimensional code of the target object to enter the detection area;

[0079] (6) The input interface module of the FPGA control unit monitors the GPIO pin connected to the two-dimensional code scanning module in real time. When the output pulse trigger signal is detected, the signal verification is started: the pulse duration is measured by the internal timer, and the level amplitude is detected at the same time. If both meet the standard, it is determined as a valid trigger signal, and the valid trigger signal verification result is output. If it does not meet the standard, it is determined as an interference signal, and the input is ignored and the pin state continues to be monitored;

[0080] (7) The FPGA control unit verifies the result based on the output valid trigger signal, and starts state switching: the internal state machine switches from the idle state to the data collection ready state, activates the SPI, I2C interface connected with the multi-modal sensor group and the conveying line encoder, initializes the 1MB internal data buffer for temporary storage of raw data, and writes the ready state identifier into the state register. At the same time, the confirmation signal is sent to the two-dimensional code scanning module through another GPIO pin, and the data collection ready state and the confirmation signal sent to the two-dimensional code scanning module are output;

[0081] (8) After receiving the output confirmation signal, the two-dimensional code scanning module flashes the status indicator light once to indicate that the trigger signal has been received, and then returns to the scanning state output in (2) to wait for the next target object two-dimensional code to enter the detection area.

[0082] S2.2: The FPGA control unit acquires the conveying speed data transmitted by the conveying line encoder in the hardware architecture in the data collection ready state, adjusts the collection frequency of the multi-modal sensor group in the hardware architecture according to the conveying speed data, and sends a synchronization instruction to the multi-modal sensor group;

[0083] Further, the specific steps of S2.2 include:

[0084] (1) After the FPGA control unit enters the data collection ready state, the differential signal receiving interface module connected with the conveying line encoder in the hardware architecture is activated; the differential signal receiving interface module is pre-set to adapt to the AB phase pulse output format of the encoder;

[0085] (2) The differential signal receiving interface module starts the internal clock synchronization mechanism to align the receiving clock with the encoder output clock. For example, if the encoder output clock is 1MHz, the differential signal receiving interface module is configured to have a 1MHz sampling frequency. At the same time, a 512-byte data buffer is initialized for temporary storage of pulse signals, completing the hardware preparation before receiving, and outputting the ready encoder interface state;

[0086] (3) Based on the output ready encoder interface state, the conveying line encoder generates real-time pulse signals as it operates with the conveying line. For example, 1024 pulses per revolution, the direction of rotation is distinguished by the phase difference of the AB phase pulses. These pulse signals are transmitted to the encoder interface module through shielded differential lines. The encoder interface module detects the edges of the pulse signals at the sampling frequency configured in (2), records the number of pulses within 10 milliseconds, and determines the conveying direction according to the AB phase relationship to ensure that the speed data is positive. The pulse count result per unit time is output;

[0087] (4) The speed analysis module of the FPGA control unit receives the output pulse count result, and calculates the speed in combination with the preset parameters of the conveying line encoder: first, the number of pulses per unit time is converted into the number of rotations, then the conveying distance per unit time is calculated, and finally the conveying speed per second is converted, such as 3.14 cm per 10 ms, the speed is 3.14 m / s, and the real-time conveying speed data is output, wherein the preset parameters include the wheel diameter corresponding to the conveying line, the number of pulses per rotation, such as 31.4 cm of conveying distance per rotation when the wheel diameter is 10 cm, the number of rotations is equal to the ratio of the number of pulses and the number of pulses per rotation, and the conveying distance is equal to the product of the number of rotations and the distance per rotation;

[0088] (5) The FPGA control unit calls the reference configuration data of the multi-modal sensor group from the preset parameter storage area of itself, including the initial acquisition frequency of each sensor, the speed-frequency adjustment reference, the upper and lower limits of the hardware acquisition frequency, and the preset reference speed, and outputs the sensor reference configuration data, wherein the speed-frequency adjustment reference is, for example, 5 frames / second adjustment amount per meter / second for the camera and 20 times / second adjustment amount per meter / second for the SWIR sensor; the upper and lower limits of the hardware acquisition frequency are, for example, 60 frames / second and 10 frames / second for the camera and 200 times / second for the SWIR; and the preset reference speed is set according to the target object detection requirement, such as 0.5 m / s;

[0089] (6) The frequency adjustment module of the FPGA control unit receives the output real-time conveying speed data and sensor reference configuration data, calculates the speed deviation, that is, the difference between the real-time speed and the preset reference speed; if the real-time speed is higher than the reference speed, the positive adjustment amount is calculated according to the adjustment reference; if it is lower than the reference speed, the negative adjustment amount is calculated; if it is equal to the reference speed, the adjustment amount is 0; the adjustment amount is added to the initial acquisition frequency to obtain the target acquisition frequency of each sensor, and the target acquisition frequency parameter is output;

[0090] (7) The synchronization instruction generation module of the FPGA control unit constructs a synchronization instruction frame based on the output target acquisition frequency parameter: the frame header is a preset synchronization identifier, which is used for sensor identification instruction, the frame contains the target acquisition frequency and acquisition trigger delay distinguished by sensor ID, and the frame tail is a check code calculated based on the data in the frame, which is used for sensor to verify the integrity of the instruction, and the complete synchronization instruction frame is output;

[0091] (8) The FPGA control unit sends the synchronization instruction frame to each sensor in turn through the SPI bus interface connected with the multi-modal sensor group according to the sensor ID, each sensor verifies the frame header and the check code through the internal verification module after receiving the synchronization instruction, configures the internal clock generator according to the target acquisition frequency after confirming the validity of the synchronization instruction, and returns the confirmation signal containing the sensor ID and the configuration completion identifier to the FPGA control unit, and the configuration confirmation signal of each sensor is output;

[0092] (9) The FPGA control unit receives the output of each sensor configuration confirmation signal, checks whether all sensors return confirmation, if there is no return, retransmit synchronization instruction, after confirming that all configurations are completed, start the internal synchronization timer, according to the trigger delay in the instruction, when the timer ends, send a unified trigger signal to the bus interface, after receiving the trigger signal, the multi-modal sensor group starts data acquisition at the same time, to ensure that all kinds of data are generated synchronously in time dimension.

[0093] S2.3: After receiving the synchronization instruction sent by the FPGA control unit, the multi-modal sensor group responds to synchronize the acquisition of RGB images, SWIR spectral data, millimeter wave A-scan data and environmental data, and obtains multi-sensor data.

[0094] The adjustment process adopts a proportional-integral control algorithm, including:

[0095] S2.2.1: The FPGA control unit calculates the speed deviation value by differentiating the real-time conveying speed data transmitted by the conveying line encoder from the preset reference speed; the preset reference speed is stored in the parameter storage area preset by itself;

[0096] Further, if the real-time conveying speed is greater than the reference speed, the deviation value is positive, representing that the conveying is too fast and the sensor acquisition frequency needs to be increased to avoid data omission; if the real-time conveying speed is less than the reference speed, the deviation value is negative, representing that the conveying is too slow and the acquisition frequency can be appropriately reduced to save resources; if both are equal, the deviation value is 0, and the acquisition frequency does not need to be adjusted.

[0097] S2.2.2: Convert the speed deviation value into the acquisition frequency adjustment amount based on the proportional-integral control algorithm, wherein the proportional term coefficient is set according to the sensor response speed;

[0098] Further, the specific steps of S2.2.2 include:

[0099] For the response speed of different types of sensors in the multi-modal sensor group, a preset proportional term coefficient is called, such as the camera response is faster, the proportional term coefficient is set to 0.2 Hz / (m / s), to ensure that the frequency can be adjusted in time when the speed increases slightly; the millimeter wave radar response is slower, the proportional term coefficient is set to 0.1 kHz / (m / s), to avoid large frequency fluctuations, the speed deviation value is multiplied by the corresponding proportional term coefficient to obtain the proportional adjustment component; the preset integral term coefficient is called to integrate the speed deviation value to eliminate static deviation, such as long-term existence of small speed deviation, the integral adjustment component can gradually accumulate the adjustment amount, so that the collection frequency finally adapts to the actual conveying speed, and the integral adjustment component is obtained; the proportional adjustment component and the integral adjustment component are added to obtain the collection frequency adjustment amount for each sensor, in the present application, the preset integral term coefficient is set to .

[0100] S2.2.3: The calculated collection frequency adjustment amount is superimposed on the initial collection frequency of the multi-modal sensor group to obtain the adjusted target collection frequency, and the adjusted target collection frequency is sent to the multi-modal sensor group as one of the parameters of the synchronization instruction.

[0101] Further, if the collection frequency adjustment amount is positive, the target collection frequency is higher than the initial collection frequency; if the collection frequency adjustment amount is negative, the target collection frequency is lower than the initial collection frequency; if the collection frequency adjustment amount is 0, the target collection frequency is consistent with the initial collection frequency; at the same time, it is ensured that the calculated target collection frequency does not exceed the hardware working range of each sensor, such as the upper limit of the camera frame rate is not more than 60 fps, and the lower limit is not less than 10 fps, if it exceeds, the hardware limit value is taken as the final target collection frequency.

[0102] According to the multi-sensor data and the clock domain division preset by the FPGA control unit, the clock is compensated through the PLL to obtain the synchronized multi-sensor data, including:

[0103] S3.1: The FPGA control unit loads the clock domain division parameters from the preset parameter storage area of itself, and initializes the PLL based on the clock domain division parameters, so that the PLL enters the adjustment state, and outputs the initialized PLL and the loaded clock domain division parameters; the clock domain division parameters include the reference frequency, phase relationship and maximum deviation threshold of the camera clock domain and the radar clock domain;

[0104] Further, the specific steps of S3.1 include:

[0105] (1) After the FPGA control unit enters the data processing flow, the internal storage interface module is started, the storage interface module is positioned to the preset parameter storage area storing the clock domain division parameters through the preset address decoding logic, the storage interface module reads the storage area data in address order, and integrates the clock domain division parameters read initially;

[0106] (2) The parameter verification module of the FPGA control unit receives the output preliminary read clock domain division parameters, and carries out effectiveness verification: checking whether the camera clock domain reference frequency is within 20 to 30 MHz, whether the radar clock domain reference frequency is within 8 to 12 MHz, verifying whether the phase relationship value is within -10 to 10 ns, and confirming whether the maximum deviation threshold is a positive number and less than one-tenth of the reference period. If all pass, the clock domain division parameters that pass the verification are output; if there is invalid parameter, the storage area re-reading mechanism is triggered, and (1) is returned to re-read until valid parameters are obtained;

[0107] (3) The PLL configuration module of the FPGA control unit receives the output clock domain division parameters that pass the verification, and starts the PLL initialization process: according to the reference frequencies of the camera clock domain and the radar clock domain, the 20 MHz system clock inside the FPGA control unit is set as the PLL input reference clock; the frequency division and frequency multiplication coefficients are calculated based on the reference frequencies, for example, to obtain a 25 MHz camera clock, the 20 MHz reference clock is multiplied by 1.25 times; the PLL internal phase offset register is configured according to the phase relationship parameters, and the initial phase difference is preset to match the clock domain relationship; the PLL adjustment sensitivity is set according to the maximum deviation threshold, and the initialization configuration signal of the PLL is output;

[0108] (4) The PLL receives the output initialization configuration signal, and starts internal self-checking: checking whether the phase difference between the input and output clocks can be normally detected by the phase detector, whether the capacitor-resistor parameters of the loop filter match the configuration requirements, and whether the voltage-controlled oscillator can stably oscillate within the target frequency range. After the self-checking passes, the PLL enters the adjustment-ready state, and returns an initialization completion signal to the FPGA control unit through a special state pin, and outputs the initialization completion signal, wherein the initialization completion signal is high level;

[0109] (5) The FPGA control unit receives the output initialization completion signal, confirms that the PLL has entered the adjustment-ready state, at the same time, stores the clock domain division parameters that pass the verification to the internal cache, and finally outputs the initialized PLL and the loaded clock domain division parameters.

[0110] S3.2: The FPGA control unit receives multi-sensor data, extracts the clock characteristics of each sensor actually working from the synchronization signals associated with the multi-sensor data in combination with the clock domain division parameters, and compares them with the reference clock to analyze the frequency deviation and phase offset information, and outputs the clock deviation data; the synchronization signals associated with the multi-sensor data include camera line synchronization signals and radar sampling trigger signals;

[0111] Further, the specific steps of S3.2 include:

[0112] (1) FPGA control unit receives multi-sensor data through a special data interface connected with multi-modal sensor group, such as MIPI interface of camera and LVDS interface of radar, while accompanying with associated synchronization signals; the synchronization signals include line synchronization signal output by camera and sampling trigger signal output by radar, wherein the line synchronization signal output by camera is pulse signal generated at the start of each row of pixel collection, and the sampling trigger signal output by radar is trigger pulse before each A-scan data collection;

[0113] (2) The special data interface performs level conversion and noise filtering on the received associated synchronization signals, and outputs pre-processed multi-sensor data and associated synchronization signals, wherein the level conversion and noise filtering are prior art contents in the field and are not the inventive scheme of the present application, and will not be described here;

[0114] (3) The clock parameter calling module of the FPGA control unit takes out the clock domain division parameters loaded previously from the internal cache and transmits them to the clock feature extraction module, and outputs the reference clock parameters to be compared;

[0115] (4) The clock feature extraction module receives the output pre-processed synchronization signals and the reference clock parameters to be compared, and extracts actual clock features for the synchronization signals of different sensors: for the camera line synchronization signal, the time interval between two consecutive rising edges is measured by an internal high-frequency counter, and the reciprocal of the interval is the actual working clock frequency of the camera; at the same time, the time point of the rising edge relative to the rising edge of the internal reference clock of the FPGA control unit is recorded as a phase reference; for the radar sampling trigger signal, the same method is used to measure the time interval between two consecutive rising edges to obtain the actual clock frequency of the radar, and the time point of the rising edge relative to the internal reference clock of the FPGA control unit is recorded, and the actual clock frequency of the camera and the radar and the phase reference point are output;

[0116] (5) The frequency deviation analysis unit receives the actual clock frequency and the reference frequency in the reference clock parameters to be compared, and calculates the frequency deviation: the camera frequency deviation is the difference between the actual clock frequency and the reference frequency, and the ratio of the difference to the reference frequency; the radar frequency deviation is calculated in the same way, and it is checked whether the deviation is within the preset adjustable range, such as ±1%, and if it is beyond, it is marked as abnormal, and the frequency deviation value and the deviation state of the camera and the radar are output, wherein the deviation state includes normal or abnormal;

[0117] (6) The phase offset analysis unit receives the preset phase relationship in the phase reference point and the reference clock parameter to be compared, calculates the phase offset: taking the internal reference clock of the FPGA control unit as the time reference, respectively calculating the actual time difference between the rising edge of the camera line synchronization signal and the rising edge of the radar sampling trigger signal, and then comparing with the preset phase relationship, such as 5 nanoseconds, the difference is the phase offset, and the final phase offset is output by reducing the random error through multiple measurements;

[0118] (7) The clock deviation data integration unit receives the frequency deviation value, deviation state and phase offset output, combines the maximum deviation threshold in the reference clock parameter to be compared, and integrates the clock deviation data in the preset format, including sensor type, frequency deviation, phase offset and timestamp: if the frequency deviation state is abnormal, an abnormal identifier is added in the data; if the phase offset exceeds the maximum deviation threshold, an out-of-limit identifier is added, and the complete clock deviation data is finally output.

[0119] S3.3: The FPGA control unit converts the output clock deviation data into control instructions and sends them to the initialized PLL, which generates a compensation clock signal consistent with the reference clock frequency and phase alignment through internal phase discrimination, filtering and voltage-controlled oscillation adjustment;

[0120] Further, the FPGA control unit converts the output clock deviation data into control instructions, including:

[0121] (1) The instruction conversion module of the FPGA control unit receives the output clock deviation data, including sensor type, frequency deviation value, phase offset, deviation state and out-of-limit identifier, which is generated by the clock deviation data integration unit;

[0122] (2) The instruction conversion module structurally analyzes the clock deviation data and extracts key information according to the preset field: separates the frequency deviation values of the camera and the radar, confirms the specific value of the phase offset, identifies the abnormal or out-of-limit identifier, determines the type and degree of the clock deviation to be adjusted, and outputs the analyzed clock deviation key information;

[0123] (3) The instruction conversion module calls the internal preset adjustment strategy table according to the deviation information based on the analyzed clock deviation key information; the preset adjustment strategy table records the corresponding relationship between different deviation types and adjustment methods;

[0124] (4) The instruction conversion module calculates the specific adjustment parameters according to the adjustment mode of the output: for frequency adjustment, the frequency deviation value and the PLL preset adjustment sensitivity are used, such as 0.01 frequency coefficient adjustment corresponding to each 0.1% deviation, to calculate the frequency division / multiplier coefficient correction amount, such as 0.04 coefficient adjustment corresponding to 0.4% deviation of the camera; for phase adjustment, the phase offset and the PLL phase adjustment step are used, such as 10 adjustment units corresponding to each nanosecond offset, to convert the phase offset register write value, such as 20 adjustment units corresponding to 2 nanosecond offset; the joint adjustment calculates the two types of parameters respectively, and outputs the specific adjustment parameter value;

[0125] (5) The instruction conversion module converts the output adjustment parameter value into a standardized control instruction frame according to the PLL preset interface protocol: the frame header is a fixed identifier, the data field is the adjustment parameter value formatted according to the PLL register bit width, and the frame tail is a check code calculated based on the frame header and the data field. The output is a formatted control instruction frame;

[0126] (6) The instruction conversion module performs validity check on the output control instruction frame: check whether the frame header conforms to the preset format, whether the data field parameter is within the PLL hardware adjustment range, such as within ±0.1 of the frequency coefficient correction amount, within 100 of the phase adjustment unit, and whether the check code is correct. If the check is passed, it is marked as a valid control instruction; otherwise, return to (4) to recalculate the adjustment parameters and generate a control instruction frame until a valid control instruction is output;

[0127] (7) The instruction conversion module transmits the output verified control instruction to the control interface connected between the FPGA and the PLL, and the control interface sends the control instruction to the PLL in time sequence, completing the conversion and output from the clock deviation data to the control instruction, and providing operation instructions for the PLL to perform clock compensation.

[0128] S3.4: The FPGA control unit distributes the output compensation clock signal to the data processing module of the corresponding sensor, and the data processing module adjusts the sampling timing based on the compensation clock signal and outputs the sensor data with aligned sampling edges;

[0129] S3.5: The FPGA control unit integrates the output sampling edge aligned data, adds a unified timestamp based on the compensation clock signal, and verifies the actual deviation of the data timestamp through the internal delay measurement circuit. If the actual deviation is within the preset deviation threshold, the synchronized multi-sensor data is output; otherwise, the clock features are extracted again and the clock deviation is analyzed.

[0130] The control PLL generates a compensation clock signal consistent with the reference clock frequency and phase alignment through internal phase detection, filtering and voltage-controlled oscillation adjustment. The phase adjustment of the PLL adopts a dynamic step mechanism, including:

[0131] S3.3.1: The FPGA control unit first calls the deviation threshold and step size parameters required for PLL phase adjustment from the parameter storage area preset by itself, including the preset large deviation threshold, small deviation threshold, and corresponding preset large adjustment step size, medium adjustment step size, and small adjustment step size;

[0132] Further, in the present application, the large deviation threshold is set to 5ns, the small deviation threshold is set to 1ns, the corresponding large adjustment step size is set to 1ps / step, the medium adjustment step size is set to 0.5ps / step, and the small adjustment step size is set to 0.1ps / step.

[0133] S3.3.2: The FPGA control unit extracts the phase offset value of the current multi-sensor clock and the reference clock from the clock deviation data, and compares the phase offset value with the preset large deviation threshold and small deviation threshold;

[0134] When the phase offset value is greater than the preset large deviation threshold, the PLL uses the large adjustment step size for fast phase compensation;

[0135] When the phase offset value is less than or equal to the large deviation threshold and greater than the small deviation threshold, the medium adjustment step size is used;

[0136] When the phase offset value is less than or equal to the small deviation threshold, the small adjustment step size is used;

[0137] S3.3.3: The FPGA control unit converts the output target adjustment step size into a control instruction recognizable by the PLL and sends it to the initialized PLL. After receiving the control instruction, the PLL adjusts the phase of the internal voltage-controlled oscillator according to the target adjustment step size, generates and outputs the preliminary compensated clock signal;

[0138] Further, the specific steps of S3.3.3 include:

[0139] (1) The phase adjustment instruction generation module of the FPGA control unit receives the output target adjustment step size, analyzes the numerical value and type of the target adjustment step size, and determines the phase accuracy level to be adjusted, while calling the preset PLL register mapping table to output the parsed step size information and register configuration parameters; the preset PLL register mapping table records the addresses and bit widths of the phase adjustment related registers;

[0140] (2) The phase adjustment instruction generation module converts the target adjustment step into a digital adjustment amount recognizable by the PLL based on the output parsed step information and the register configuration parameter: according to the phase adjustment unit conversion relationship of the PLL, such as 1 register count unit corresponding to 0.01 picosecond phase adjustment, the target adjustment step is converted into the corresponding register count value, such as 1 picosecond large adjustment step corresponding to 100 count units, 0.5 picosecond medium adjustment step corresponding to 50 count units, ensuring that the count value is within the effective range of the register, and outputting the converted register adjustment count value;

[0141] (3) The phase adjustment instruction generation module constructs a control instruction frame according to the communication protocol of the PLL, such as the SPI protocol, in combination with the register configuration parameter and the register adjustment count value: the frame header is a fixed phase adjustment instruction identifier, the address field is the phase adjustment register address, the data field is the 16-bit binary format register adjustment count value, and the frame tail is the checksum, and the complete phase adjustment control instruction frame is output;

[0142] (4) The FPGA control unit sends the phase adjustment control instruction frame through the SPI control interface connected with the PLL: the control interface operates according to the SPI timing: first pull down the chip select signal to enable the PLL to receive, then send the serial data of the frame header, address field, data field, and frame tail in turn, finally pull up the chip select signal to end the transmission, and output the instruction transmission completion signal;

[0143] (5) The instruction receiving module of the initialized PLL in the standby adjustment state starts data reception after monitoring the instruction transmission completion signal: serial data is collected according to the SPI timing and converted into a parallel instruction frame, the internal verification module is used to verify whether the frame header matches the phase adjustment identifier and whether the checksum is correct, the address field and the data field are extracted after verification, the adjustment count value in the data field is temporarily stored in the internal buffer register after confirming that the address points to the phase adjustment register; verification fails, the instruction is discarded and an error signal is output, and finally the adjustment count value verified is output;

[0144] (6) The phase adjustment module of the PLL reads the adjustment count value verified, combines the internally preset phase-voltage conversion coefficient, such as 100 count units corresponding to 0.1 volt control voltage change, calculates the control voltage adjustment amount required by the voltage controlled oscillator, the phase adjustment module transmits the adjustment amount to the loop filter, and after RC circuit smoothing processing, a stable control voltage is output to the voltage controlled oscillator, so that the oscillation phase of the voltage controlled oscillator is offset by the target adjustment step, such as 1 picosecond step corresponding to 1 picosecond phase advance or lag, and the preliminary adjusted voltage controlled oscillator clock signal is output;

[0145] (7) The output buffer module of the PLL receives the preliminarily adjusted voltage-controlled oscillator clock signal, enhances the signal driving capability through an internal driving circuit, ensures that the output clock amplitude and rise / fall time meet the requirements of the multi-sensor clock domain, and finally generates and outputs the preliminarily compensated clock signal, while outputting an adjustment completion signal to the FPGA control unit through a special feedback pin.

[0146] S3.3.4: The FPGA control unit starts the real-time monitoring module, inputs the output preliminarily compensated clock signal and the preset reference clock into the phase detection circuit, measures the phase difference between the two in real time, calculates a new phase offset value at a fixed period, updates the phase information in the clock deviation data, and outputs the updated phase offset value.

[0147] Further, the specific steps of S3.3.4 include:

[0148] (1) The FPGA control unit starts the internal real-time monitoring module, which calls key configuration information from its own preset parameter storage area, completes the configuration, and outputs the configuration completion state of the real-time monitoring module and the preset reference clock signal.

[0149] (2) Based on the output configuration completion state, the real-time monitoring module starts the signal input unit, transmits the preliminarily compensated clock signal output by the PLL and the preset reference clock signal to the signal conditioning subunit, the signal conditioning subunit filters out high-frequency noise through an RC low-pass filter, unifies the amplitudes of the two signals to 3.3V standard level through a level matching circuit, and optimizes the steepness of the signal edges through an edge shaping circuit, and outputs the conditioned preliminarily compensated clock signal and the preset reference clock signal, wherein the RC low-pass filter, the level matching circuit, and the edge shaping circuit are prior art contents in the field and are not the inventive scheme of the present application, and will not be described here.

[0150] (3) The phase detection circuit receives the two conditioned clock signals and starts real-time phase difference measurement: an XOR gate combined with a 1 gigahertz high-frequency counter is used, the two inputs of the XOR gate are connected to the two conditioned clock signals, and a high-level pulse is output when the phases are not synchronized; the high-frequency counter counts the pulse width, and outputs the real-time phase difference count result every time a period comparison is completed.

[0151] (4) The phase offset value calculation unit of the real-time monitoring module reads the real-time phase difference count result at a preset 10 microsecond fixed period, processes the multiple count results in the period: eliminates abnormal values exceeding the 0.1 nanosecond precision threshold, takes the arithmetic mean of the valid results, converts the new phase offset value in the time dimension according to the 1 gigahertz count frequency, records the time stamp, and outputs the new phase offset value in the period.

[0152] (5) The clock bias data updating unit receives the new phase offset value, calls the stored clock bias data, updates the phase information therein: keeps the frequency bias value and other unchanged parameters unchanged, replaces the old value with the new phase offset value, and updates the bias state according to the comparison between the new phase offset value and the preset maximum bias threshold value, wherein the new phase offset value is less than or equal to the preset maximum bias threshold value, and the output updated complete clock bias data and the updated phase offset value alone.

[0153] S3.3.5: The FPGA control unit reuses the output updated phase offset value as a new current phase offset value, triggers the phase offset value comparison, step selection and PLL adjustment again until the updated phase offset value is stable and less than or equal to the small bias threshold value.

[0154] The generation process of the effective data includes:

[0155] S4.1: According to the synchronized multi-sensor data and the pre-processing logic preset by the FPGA control unit, the RGB image in the synchronized multi-sensor data is extracted, and the adaptive compression processing is performed on the RGB image to obtain the RGB compressed image; the SWIR spectral data is extracted and sequentially subjected to smoothing filtering and PCA dimension reduction processing to obtain the SWIR feature data; the millimeter wave A-scan data is extracted and subjected to parallel FFT processing to obtain the millimeter wave scattering map;

[0156] S4.2: The RGB compressed image, the SWIR feature data, the millimeter wave scattering map, and the environment data in the synchronized multi-sensor data are integrated to form the full amount of pre-processing data to be filtered;

[0157] S4.3: Based on the pre-processing logic preset by the FPGA control unit, the full amount of pre-processing data to be filtered is subjected to outlier filtering, and the abnormal data is separated and temporarily stored in the temporary storage area of the FPGA control unit, and the remaining data is the effective data.

[0158] Further, in the outlier filtering of the full amount of pre-processing data to be filtered based on the pre-processing logic preset by the FPGA control unit, the outlier filtering adopts a combination of 3σ criterion and isolated forest algorithm, including:

[0159] (1) The FPGA control unit calls the historical effective data set from the storage partition preset by itself, and the historical effective data set contains the verified normal multi-sensor data in the past 24 hours, and the benchmark data pool for outlier filtering is clear;

[0160] (2) The FPGA control unit calculates the mean and standard deviation of each data type based on the called historical effective data set, and outputs the mean and standard deviation parameters of each type, wherein the calculation formula of the mean and standard deviation is a prior art in the field, and is not the creative scheme of the present application, and is not described here;

[0161] (3) The FPGA control unit obtains the full amount of preprocessed data to be filtered, and compares each data point with the mean and standard deviation according to the corresponding type: if any data point is outside the 3σ range, it is marked as preliminary abnormal data; otherwise, it is classified as normal data, and the preliminary abnormal data set and the normal data set are output, wherein σ represents the standard deviation;

[0162] (4) The FPGA control unit starts the preset isolated forest model, and the isolated forest model is realized based on the parallel computing unit of the FPGA, wherein 10 isolated trees are configured, and the sample division feature number of each tree is set to 5. Specifically, the isolated forest model is a prior art in the field, and is not the creative scheme of the present application, and is not described here;

[0163] (5) The output preliminary abnormal data set and normal data set are jointly input into the isolated forest model, and the isolated forest model generates the abnormal score of each preliminary abnormal data by calculating the path length of each data point in the isolated tree, wherein the score range is 0 to 1, and the higher the value, the greater the abnormal probability;

[0164] (6) The FPGA control unit compares the output abnormal score with the preset score threshold: the preliminary abnormal data with an abnormal score greater than the preset score threshold is determined as the final abnormal data, and is temporarily stored in the temporary storage area of the FPGA control unit; the rest, i.e. the preliminary abnormal data and the normal data set with an abnormal score less than or equal to the preset score threshold, are integrated into effective data, and in the present application, the preset score threshold is set to 0.8;

[0165] (7) The FPGA control unit updates the historical effective data set every 2 hours: the output effective data is replaced by new data in proportion, such as 20% new data, to supplement the original historical effective data set, and the expired data exceeding 24 hours is removed, so that the historical effective data set always reflects the latest normal data characteristics of the scene.

[0166] The temporary storage area of the FPGA control unit adopts a double-buffer queue design, including a first buffer partition and a second buffer partition;

[0167] The FPGA control unit writes the separated abnormal data into the first buffer partition, when the storage amount of the first buffer partition reaches a preset threshold, automatically switches to the second buffer partition to continue writing the abnormal data, and simultaneously batch-transmits the abnormal data in the first buffer partition to the FRAM; after the data transmission of the first buffer partition is completed, the first buffer partition is emptied, and waits for the next round of abnormal data writing, wherein the preset threshold is determined based on the writing rate of the FRAM and the generation rate of the abnormal data, and in the present application, it is set to 80% of the storage capacity.

[0168] According to the effective data, the temporarily stored abnormal data and the function module preset by the FPGA control unit, data fusion inference is performed to obtain a hierarchical result, including:

[0169] S5.1: The SWIR feature data, the millimeter wave scattering map and the RGB compressed map in the effective data are transmitted into the weighted gate attention module preset by the FPGA control unit, the contribution degrees of the SWIR feature data, the millimeter wave scattering map and the RGB compressed map to target classification are analyzed, the respective weight proportions are calculated and the fusion features are outputted;

[0170] Further, the specific steps of S5.1 include:

[0171] (1) The FPGA control unit calls the effective data obtained by preprocessing, extracts the SWIR feature data, the millimeter wave scattering map and the RGB compressed map therein, clearly defines the three types of data objects whose feature variances need to be calculated, and outputs the three types of feature data to be processed;

[0172] (2) The FPGA control unit calculates the feature variances of the outputted three types of feature data respectively: the mean value of the variances of each spectral feature dimension of the SWIR feature data is calculated to obtain the SWIR feature data variance; the mean value of the variances of each pixel point feature value of the millimeter wave scattering map is calculated to obtain the millimeter wave scattering map variance; the mean value of the variances of each channel pixel value of the RGB compressed map is calculated to obtain the RGB compressed map variance, and the feature variances of the three types of data are outputted;

[0173] (3) The FPGA control unit adds the outputted three types of feature variances to obtain the total variance, and then calculates the variance proportion of each type of data respectively, that is, the SWIR variance proportion is equal to the ratio of the SWIR feature data variance to the total variance, the millimeter wave variance proportion is equal to the ratio of the millimeter wave scattering map variance to the total variance, and the RGB variance proportion is equal to the ratio of the RGB compressed map variance to the total variance, and the variance proportions of the three types of data are outputted;

[0174] (4) The FPGA control unit inputs the output three types of variance proportions into a preset Softmax function, performs normalization processing on the proportions through function operation, obtains weight coefficients with a sum of 1, and outputs the weight coefficients corresponding to the three types of data, wherein the Softmax function is prior art content in the field and is not the inventive scheme of the present application, and will not be described here.

[0175] (5) The FPGA control unit multiplies the output SWIR feature data, millimeter wave scattering map and RGB compressed map respectively with the output corresponding weight coefficients, then superimposes and sums the three weighted feature data according to the feature dimension to obtain a fusion feature that fuses the key features of the three types of data, and outputs the fusion feature.

[0176] S5.2: Based on the filtered environmental data in the effective data, the data standardization processing logic in the FPGA control unit is called to perform numerical normalization operation on the filtered environmental data, and then the normalized environmental data is converted into a farming vector according to a preset feature mapping rule;

[0177] Further, in the conversion of the normalized environmental data into the farming vector according to the preset feature mapping rule, the feature mapping rule is constructed based on the correlation degree between the environmental parameters and the farming demand, including: determining the parameter types contained in the environmental data, including temperature, humidity, light intensity and soil moisture content; setting the mapping dimension and weight for each parameter, wherein the temperature is mapped to the 1st dimension of the farming vector, the weight is set to 0.3, the humidity is mapped to the 2nd dimension, the weight is set to 0.25, the light intensity is mapped to the 3rd dimension, the weight is set to 0.25, and the soil moisture content is mapped to the 4th dimension, the weight is set to 0.2; multiplying the normalized values of each environmental parameter with the corresponding weight to obtain the farming vector, and the dimension of the farming vector is fixed to 4 dimensions to ensure the compatibility of splicing with the fusion feature, wherein the weight of each parameter is obtained through training of agricultural production data set, and the weight value is updated once every quarter to adapt to the farming demand in different growth stages.

[0178] S5.3: Based on the fusion feature, the farming vector and the fusion feature are combined and spliced according to a preset dimension order to form a splicing vector containing multi-dimensional information;

[0179] S5.4: Taking the splicing vector as input, the INT8 quantization circuit in the FPGA control unit is called to perform precision compression and data format conversion on the values in the splicing vector to obtain a quantized splicing vector, and the quantized splicing vector is input into the lightweight model;

[0180] Further, in the precision compression and data format conversion of the numerical values in the spliced vector, the quantization processing adopts an asymmetric quantization method. The asymmetric quantization method is prior art in the field and is not the creative scheme of the present application, and thus is not described herein.

[0181] S5.5: The lightweight model performs feature extraction, correlation analysis and category judgment on the multi-dimensional information in the quantized spliced vector according to the built-in inference algorithm, generates a grading result reflecting the grading situation of the target object through the model output layer, and transmits the output grading result to the PLC and WCS outside the hardware architecture. The lightweight model is prior art in the field and is not the creative scheme of the present application, and thus is not described herein.

[0182] According to the control instructions and optimization requirements fed back by the PLC and WCS based on the grading result, model switching and data supplement verification are performed, including:

[0183] S6.1: The FPGA control unit receives the feedback information of the PLC and WCS; the feedback information includes control instructions and optimization requirements for the current data processing flow;

[0184] Further, the transmission of the feedback information adopts the Modbus-TCP protocol, and the FPGA control unit is built-in with a protocol analysis module, specifically including:

[0185] (1) The protocol analysis module of the FPGA control unit is started, and it is clear that the feedback information to be received is from the PLC and WCS outside the hardware architecture. The feedback information is generated based on the grading result transmitted to the PLC and WCS. The protocol analysis module prepares to receive the TCP data packet sent by the PLC / WCS through a preset TCP port, and outputs the TCP data packet to be analyzed;

[0186] (2) The protocol analysis module performs header analysis on the output TCP data packet, extracts the function code in the TCP data packet, distinguishes the type of feedback information through the function code, and outputs the identified function code;

[0187] (3) The protocol analysis module analyzes the data domain content of the TCP data packet according to the output function code: if the function code is 0x03, the control instructions are parsed, including adjusting the sensor acquisition frequency, such as instruction identifier 0x01, improving the inference accuracy, such as instruction identifier 0x02, and reducing the processing delay, such as instruction identifier 0x03; if the function code is 0x10, the optimization requirements are parsed, including the target inference delay, the target accuracy, and the upper limit of resource occupation, and the parsed control instructions and optimization requirements are output. In the present application, the target inference delay is set to be less than or equal to 8 ms, the target accuracy is set to be greater than or equal to 95%, and the upper limit of resource occupation is set to be less than or equal to 70% FPGA logical resources.

[0188] (4) The protocol analysis module converts the output control instructions and optimization requirements into parameters recognizable by the FPGA control unit: for the control instructions, the instruction identifier is converted into an 8-bit binary operation code, such as 0x01 into 00000001; for the optimization requirements, the target reasoning time delay of 8ms is converted into a 32-bit integer parameter of 8000μs, the target accuracy of 95% is converted into a single-precision floating-point parameter of 0.95, and the upper limit of resource occupation of 70% is converted into an 8-bit integer parameter of 70; the converted internal parameters are output;

[0189] (5) The protocol analysis module stores the output internal parameters in the feedback information buffer area of the FPGA control unit, and the parameters in the feedback information buffer area are used as the input basis for the subsequent model switching unit to screen the target lightweight model and the parameter adjustment unit to optimize the sensor collection frequency, and the effective parameters in the buffer area are output; the feedback information buffer area is a pre-set ring queue with a capacity of 10 records;

[0190] (6) The protocol analysis module starts a periodic detection mechanism, which detects TCP data every 100ms: if new TCP data packets are detected, return to (1); if no new data is detected, keep the current parameters in the feedback information buffer area unchanged to ensure real-time updating of the feedback information without omission.

[0191] S6.2: The FPGA control unit uses the feedback information as a basis to call multiple pre-set lightweight models, selects a lightweight model that matches the current feedback information through the built-in model adaptability judgment logic, completes the switching of the pre-set lightweight model, and obtains a target lightweight model that adapts to the current scene, wherein the lightweight model includes MobileNet-V2, EfficientNet-Lite1, and ResNet-18, and the lightweight model is a prior art in the field and is not part of the inventive concept of the present application, and will not be described here.

[0192] S6.3: After completing the model switching, the FPGA control unit sends a start instruction to the redundant sensor in the hardware architecture based on the demand for data collection accuracy of the obtained target lightweight model, and the redundant sensor collects supplementary data that meets the accuracy requirements of the target lightweight model for the current scene in response to the instruction, and transmits the supplementary data to the FPGA control unit;

[0193] S6.4: The FPGA control unit uses the obtained target lightweight model as a verification benchmark, inputs the supplementary data into the target lightweight model, and judges whether the supplementary data and the effective data collected by the multi-sensor are matched in terms of feature consistency and numerical accuracy, while verifying the adaptation effect of the target lightweight model, and outputs a verification result containing data consistency conclusion and model adaptability conclusion.

[0194] Embodiment 2:

[0195] Another embodiment provided by the application is a FPGA-based multi-sensor data efficient acquisition and analysis processing system, comprising:

[0196] a data driving module, a clock synchronization module, a data processing module, a fusion inference module, and an optimization and storage module;

[0197] The data driving module is configured to dynamically drive the multi-modal sensor group to collect multi-sensor data at an adaptive frequency according to the conveying speed data of the conveying line encoder, so as to ensure that the collection process is dynamically matched with the scene.

[0198] The clock synchronization module is configured to generate a compensation clock through a PLL, so that the sampling edge deviation of each sensor is controlled within a preset range, and the synchronized multi-sensor data is output.

[0199] The data processing module is configured to perform targeted processing on the synchronized multi-sensor data, extract effective features and filter abnormal data, output effective data meeting the subsequent fusion inference requirements, and temporarily store abnormal data.

[0200] The fusion inference module is configured to fuse and infer the effective data, generate a hierarchical result reflecting the state of a target object, and transmit the hierarchical result to the PLC and the WCS outside the hardware architecture.

[0201] The optimization and storage module is configured to receive feedback information from the PLC and the WCS, optimize the system running state, and realize long-term storage of abnormal data.

[0202] The embodiments of the application are described above with reference to the accompanying drawings, but the application is not limited to the above specific embodiments, and the above specific embodiments are only illustrative rather than limiting. Those skilled in the art can make changes, modifications, replacements and variations to the above embodiments without departing from the purpose and scope of the application, and these are all within the protection scope of the application.

Claims

1. A method for efficient acquisition and analysis of multi-sensor data based on FPGA, characterized in that, include: Build the hardware architecture; The hardware architecture includes an FPGA control unit, a multimodal sensor group, a conveyor encoder, a QR code scanning module, an FRAM, and redundant sensors; the FPGA control unit is pre-configured with clock domain partitioning, preprocessing logic, a weighted gating attention module, an INT8 quantization circuit, and a lightweight model. Based on the hardware architecture, initiate data acquisition and acquire data from multiple sensors; Based on the multi-sensor data and the clock domain division preset by the FPGA control unit, clock compensation is performed through PLL to obtain synchronized multi-sensor data. Based on the synchronized multi-sensor data and the preprocessing logic preset by the FPGA control unit, data preprocessing and anomaly separation are performed to obtain valid data; the valid data includes SWIR feature data, millimeter-wave scattering map, RGB compressed map and filtered environmental data; Based on the valid data, the temporarily stored abnormal data, and the preset functional modules of the FPGA control unit, data fusion reasoning is performed to obtain a classification result. The classification result is transmitted to the PLC and WCS outside the hardware architecture, and the abnormal data temporarily stored in the temporary storage area is stored in the FRAM in the hardware architecture. The WCS is a warehouse control system used to control the intelligent factory equipment to perform handling and production tasks. Based on the control instructions and optimization requirements of the PLC and WCS based on the feedback of hierarchical results, model switching and data supplementation verification are performed.

2. The method for efficient acquisition and analysis of multi-sensor data based on FPGA as described in claim 1, characterized in that, Based on the aforementioned hardware architecture, data acquisition is initiated and multi-sensor data is acquired, including: The QR code scanning module in the hardware architecture generates a trigger signal and transmits it to the FPGA control unit, triggering the FPGA control unit to enter the data acquisition ready state. When the FPGA control unit is in the data acquisition ready state, it acquires the conveying speed data transmitted by the conveyor encoder in the hardware architecture, adjusts the acquisition frequency of the multimodal sensor group in the hardware architecture according to the conveying speed data, and sends a synchronization command to the multimodal sensor group. After receiving the synchronization command sent by the FPGA control unit, the multimodal sensor group responds and synchronously acquires RGB images, SWIR spectral data, millimeter-wave A-scan data, and environmental data to obtain multi-sensor data.

3. The method for efficient acquisition and analysis of multi-sensor data based on FPGA as described in claim 2, characterized in that, In adjusting the acquisition frequency of the multimodal sensor group in the hardware architecture based on the conveying speed data, the adjustment process employs a proportional-integral control algorithm, including: The FPGA control unit calculates the difference between the real-time conveying speed data transmitted by the conveyor encoder and the preset reference speed to obtain the speed deviation value; the preset reference speed is stored in its own preset parameter storage area. Based on the proportional-integral control algorithm, the speed deviation value is converted into the sampling frequency adjustment value, wherein the proportional coefficient is set according to the sensor response speed. The calculated acquisition frequency adjustment is superimposed on the initial acquisition frequency of the multimodal sensor group to obtain the adjusted target acquisition frequency, and the adjusted target acquisition frequency is sent to the multimodal sensor group as one of the parameters of the synchronization command.

4. The method for efficient acquisition and analysis of multi-sensor data based on FPGA as described in claim 1, characterized in that, Based on the multi-sensor data and the preset clock domain division of the FPGA control unit, clock compensation is performed through PLL to obtain synchronized multi-sensor data, including: The FPGA control unit loads clock domain partitioning parameters from its own preset parameter storage area, initializes the PLL based on the clock domain partitioning parameters, puts the PLL into an adjustment state, and outputs the initialized PLL and the loaded clock domain partitioning parameters; the clock domain partitioning parameters include the reference frequency, phase relationship and maximum deviation threshold of the camera clock domain and the radar clock domain. The FPGA control unit receives data from multiple sensors, combines it with clock domain partitioning parameters, extracts the actual clock characteristics of each sensor from the synchronization signals associated with the multi-sensor data, compares it with a reference clock, analyzes and obtains frequency deviation and phase offset information, and outputs clock deviation data; the synchronization signals associated with the multi-sensor data include camera line synchronization signals and radar sampling trigger signals; The FPGA control unit converts the output clock deviation data into control commands and sends them to the initialized PLL. The PLL then uses internal phase detection, filtering, and voltage-controlled oscillation adjustment to generate a compensation clock signal that is consistent with the reference clock frequency and phase-aligned. The FPGA control unit distributes the output compensation clock signal to the data processing module of the corresponding sensor. The data processing module adjusts the sampling timing based on the compensation clock signal and outputs the sensor data with the sampling edge aligned. The FPGA control unit integrates the output sampling edge aligned data, adds a unified timestamp based on the compensation clock signal, and then verifies the actual deviation of the data timestamp through the internal delay measurement circuit. If the actual deviation is within the preset deviation threshold, the synchronized multi-sensor data is output; otherwise, the clock features are re-extracted and the clock deviation is analyzed.

5. The FPGA-based method for efficient acquisition and analysis of multi-sensor data as described in claim 4, characterized in that, The control PLL generates a compensated clock signal that is consistent with the reference clock frequency and phase-aligned through internal phase detection, filtering, and voltage-controlled oscillator adjustment. The phase adjustment of the PLL adopts a dynamic step size mechanism, including: The FPGA control unit first calls the deviation threshold and step size parameters corresponding to the PLL phase adjustment from its own preset parameter storage area, including the preset large deviation threshold, small deviation threshold, and the corresponding preset large adjustment step size, medium adjustment step size, and small adjustment step size. The FPGA control unit extracts the phase offset value between the current multi-sensor clock and the reference clock from the clock offset data, and compares the phase offset value with the preset large offset threshold and small offset threshold. When the phase offset value is greater than the preset large deviation threshold, the PLL uses a large adjustment step size for phase compensation. When the phase offset value is less than or equal to the large deviation threshold and greater than the small deviation threshold, a medium adjustment step size is used. When the phase offset value is less than or equal to the small deviation threshold, a small adjustment step size is used; The FPGA control unit converts the output target adjustment step size into a control command that the PLL can recognize and sends it to the initialized PLL. After receiving the control command, the PLL adjusts the phase of the internal voltage-controlled oscillator according to the target adjustment step size, and generates and outputs a clock signal after preliminary compensation. The FPGA control unit starts the real-time monitoring module, connects the output pre-compensated clock signal and the preset reference clock to the phase detection circuit, measures the phase difference between the two in real time, calculates the new phase offset value at a fixed period, updates the phase information in the clock deviation data, and outputs the updated phase offset value. The FPGA control unit will use the updated phase offset value as the new current phase offset value, and trigger the phase offset value comparison, step size filtering and PLL adjustment again until the updated phase offset value is stably less than or equal to the small deviation threshold.

6. The method for efficient acquisition and analysis of multi-sensor data based on FPGA as described in claim 1, characterized in that, The process of generating the valid data includes: Based on the synchronized multi-sensor data and the preprocessing logic preset by the FPGA control unit, RGB images are extracted from the synchronized multi-sensor data, and adaptive compression is performed on the RGB images to obtain RGB compressed images; SWIR spectral data is extracted and sequentially processed with smoothing filtering and PCA dimensionality reduction to obtain SWIR feature data; millimeter-wave A-scan data is extracted and parallel FFT processing is performed to obtain millimeter-wave scattering images; the RGB compressed images, SWIR feature data, millimeter-wave scattering images, and environmental data from the synchronized multi-sensor data are integrated to form the full preprocessed data to be filtered; based on the preprocessing logic preset by the FPGA control unit, outlier filtering is performed on the full preprocessed data to be filtered, outlier data is separated and temporarily stored in the temporary storage area of ​​the FPGA control unit, and the remaining data is the valid data; the outlier filtering adopts a combination of the 3σ criterion and the isolated forest algorithm.

7. The method for efficient acquisition and analysis of multi-sensor data based on FPGA as described in claim 6, characterized in that, The temporary storage area of ​​the FPGA control unit adopts a dual-buffered queue design, including a first buffer partition and a second buffer partition; The FPGA control unit writes the separated abnormal data into the first buffer partition. When the storage capacity of the first buffer partition reaches a preset threshold, it automatically switches to the second buffer partition to continue writing abnormal data. At the same time, it transfers the abnormal data in the first buffer partition to the FRAM in batches. After the data transfer in the first buffer partition is completed, the first buffer partition is cleared and waits for the next round of abnormal data writing. The preset threshold is determined based on the writing rate of the FRAM and the generation rate of abnormal data.

8. The method for efficient acquisition and analysis of multi-sensor data based on FPGA as described in claim 1, characterized in that, Based on the valid data, the temporarily stored abnormal data, and the preset functional modules of the FPGA control unit, data fusion reasoning is performed to obtain a hierarchical result, including: The SWIR feature data, millimeter-wave scattering map, and RGB compressed map from the effective data are input into the weighted gating attention module preset by the FPGA control unit. By analyzing the contribution of the SWIR feature data, millimeter-wave scattering map, and RGB compressed map to the target classification, the weight ratio of each is calculated and the fused feature is output. Based on the filtered environmental data in the valid data, the data standardization processing logic in the FPGA control unit is called to perform numerical normalization on the filtered environmental data, and then the standardized environmental data is converted into agricultural vectors according to the preset feature mapping rules. Based on the fusion features, the agricultural vector and the fusion features are combined and spliced ​​in a preset dimensional order to form a spliced ​​vector containing multi-dimensional information; Using the spliced ​​vector as input, the INT8 quantization circuit in the FPGA control unit is called to perform precision compression and data format conversion on the values ​​in the spliced ​​vector to obtain the quantized spliced ​​vector, and then the quantized spliced ​​vector is passed into the lightweight model. The lightweight model uses a built-in inference algorithm to extract features, perform correlation analysis and class determination on the multi-dimensional information in the quantized spliced ​​vector. It generates a classification result reflecting the classification status of the target object through the model output layer and transmits the output classification result to the PLC and WCS outside the hardware architecture.

9. The method for efficient acquisition and analysis of multi-sensor data based on FPGA as described in claim 1, characterized in that, Based on the control instructions and optimization requirements of the PLC and WCS based on the hierarchical result feedback, model switching and data supplementation verification are performed, including: The FPGA control unit receives feedback information from the PLC and WCS; the feedback information includes control instructions and optimization requirements for the current data processing flow. Based on the feedback information, the FPGA control unit calls up multiple preset lightweight models and uses the built-in model adaptability judgment logic to select the lightweight model that matches the current feedback information, thus completing the switching of its preset lightweight model and obtaining the target lightweight model that is adapted to the current scenario. After completing the model switching, the FPGA control unit sends a start command to the redundant sensors in the hardware architecture based on the data acquisition accuracy requirements of the obtained target lightweight model. After responding to the command, the redundant sensors acquire supplementary data that meets the accuracy requirements of the target lightweight model for the current scene and transmit the supplementary data to the FPGA control unit. The FPGA control unit uses the obtained target lightweight model as a verification benchmark, inputs supplementary data into the target lightweight model, and judges whether the supplementary data matches the effective data collected by multiple sensors in terms of feature consistency and numerical accuracy through model inference. At the same time, it verifies the adaptation effect of the target lightweight model and outputs the verification results containing the data consistency conclusion and the model adaptation conclusion.

10. A multi-sensor data acquisition and analysis system based on FPGA, used to implement the multi-sensor data acquisition and analysis method based on FPGA as described in any one of claims 1-9, characterized in that, include: Data-driven module, clock synchronization module, data processing module, fusion inference module, optimization and storage module; The data driving module is used to dynamically drive the multi-modal sensor group to collect multi-sensor data based on the conveying speed data of the conveyor line encoder. The clock synchronization module is used to generate a compensation clock through a PLL, so that the sampling edge deviation of each sensor is controlled within a preset range, and the synchronized multi-sensor data is output. The data processing module is used to perform targeted processing on the synchronized multi-sensor data, extract effective features and filter abnormal data, output effective data, and temporarily store abnormal data. The fusion reasoning module is used to fuse and reason about effective data, generate a hierarchical result reflecting the state of the target object, and transmit the hierarchical result to the PLC and WCS outside the hardware architecture. The optimization and storage module is used to receive feedback information and optimize the operating status, while also storing abnormal data.

Images