Intelligent sensor data fusion method and system based on deep learning

Abstract

The invention discloses an intelligent sensor data fusion method and system based on deep learning, and the method comprises the steps: obtaining a historical image collected by an image collection device, and extracting edge feature information; acquiring motion parameters of a target acquired by motion parameter detection equipment in real time; based on the edge feature information, predicting image states of subsequent frames of the image acquisition equipment under various candidate acquisition frequencies, and generating a plurality of state prediction results; calculating a matching degree evaluation value of each state prediction result according to the motion parameters; selecting the acquisition frequency corresponding to the state prediction result with the highest matching degree evaluation value from the candidate acquisition frequencies as an optimized acquisition frequency; controlling the image acquisition equipment to adjust the acquisition frequency according to the optimized acquisition frequency; monitoring the adjusted real-time image data, and when the image quality is reduced, adjusting the acquisition frequency again; therefore, future image states under different frequencies are generated, evaluation is performed in combination with the motion parameters, the optimal acquisition frequency is extracted, and the data fusion effect can be effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of sensors, in particular to an intelligent sensor data fusion method and system based on deep learning. BACKGROUND

[0002] In modern industrial production, especially in the production of lithium batteries, image acquisition devices play a crucial role. In order to obtain high-quality image data, image acquisition devices usually need to dynamically adjust their acquisition parameters according to the motion state and environmental conditions of the target object, and the acquisition frequency is one of the key parameters that affect image quality and data processing efficiency.

[0003] In the production process of battery winding machines, the data acquisition frequencies and delays of visual sensors (used for detecting winding alignment) and laser displacement sensors (used for measuring winding thickness) are different. Visual sensor processing images requires milliseconds, while laser displacement sensor response is microseconds. When the data feedback frequencies of the two are not adapted, time sequence misalignment occurs when the data is fused. For example, when the winding speed increases, the time sequence misalignment may cause the thickness detection and image recognition to be mismatched, causing the winding to be loose or too tight.

[0004] In the prior art, the acquisition frequency of the image acquisition device is usually adjusted by manual experience. Chinese Patent No. CN110658482B discloses a multi-sensor data acquisition fusion method for a magnetic detection system. The method acquires original sensor data of multiple magnetic detection systems located at different positions in three-dimensional space, selects one of the multiple magnetic detection systems as a magnetic detection system base station, processes the original sensor data of each magnetic detection system to unify the sampling frequency of each sensor original data, extracts data features and performs data fusion processing on the multi-sensor data of each magnetic detection system after the sampling frequency is unified, and differentiates the feature vector values of any magnetic detection system at a spatial position in the multiple magnetic detection systems from the feature vector values of the magnetic detection system base station to complete multi-sensor data acquisition fusion. The technical solution of the application solves the technical problem of different sampling results caused by different sampling frequencies of multiple sensors in the prior art.

[0005] However, this fixed or manual adjustment method has significant limitations. For example, when the target object moves at high speed, fixed low-frequency acquisition may cause motion blur, resulting in a decrease in image quality and an inability to accurately capture target details; and when the target object is stationary or moving at low speed, fixed high-frequency acquisition will generate a large amount of redundant data, increasing the storage and processing burden, and wasting computing resources and bandwidth.

[0006] Therefore, how to realize intelligent and adaptive optimization of image acquisition frequency to adapt to complex application scenarios is an urgent problem to be solved in the current technical field. SUMMARY

[0007] The technical problem solved by the present application is that the prior art cannot solve the problems of inaccurate and lagging image acquisition frequency adjustment in the prior art.

[0008] To solve the above technical problems, the present application provides the following technical solutions: An intelligent sensor data fusion method based on deep learning, comprising: Step S100, acquiring historical images collected by an image acquisition device and extracting edge feature information in the historical images; Step S200, acquiring motion parameters of a target collected by a motion parameter monitoring device in real time; Step S300, based on the edge feature information, predicting the image state of subsequent frames of the image acquisition device under multiple candidate acquisition frequencies, and generating multiple state prediction results; Step S400, calculating the matching degree evaluation value of each state prediction result according to the real-time data of the target motion parameters; Step S500, selecting the acquisition frequency corresponding to the state prediction result with the highest matching degree evaluation value from the candidate acquisition frequencies as the optimized acquisition frequency; Step S600, controlling the image acquisition device to adjust the acquisition frequency according to the optimized acquisition frequency; Step S700, monitoring the real-time image data collected after adjustment, and when it is judged that the image quality has decreased, re-executing steps S100 to S600.

[0009] Preferably, the step S100 of extracting the edge feature information in the historical images specifically comprises the following sub-steps: Step S110, preprocessing the historical images, converting color images into grayscale images through grayscale processing, and eliminating random noise interference in the images by using a median filter algorithm; Step S120, processing the preprocessed grayscale images through an edge detection algorithm to generate a preliminary edge feature map; Step S130, extracting features from the preliminary edge feature map to obtain contour feature information; the contour feature information includes edge position, edge length and edge direction data.

[0010] Preferably, the step S300 specifically comprises the following sub-steps: Step S310, according to the extracted edge feature information, calculating the position change amount of the edge features in the continuous frames by using the inter-frame difference method, and combining the frame time interval to calculate the motion speed and motion direction of the target; Step S320, establishing a motion trend corresponding relationship of the target based on the motion speed and the motion direction, the motion trend corresponding relationship including a uniform motion corresponding relationship and a uniform acceleration motion corresponding relationship, for describing a displacement trajectory of the target in an image coordinate system; Step S330, randomly setting a plurality of different candidate acquisition frequencies, predicting a change state of the edge feature in a subsequent frame under a time interval corresponding to the different candidate acquisition frequencies according to the motion trend corresponding relationship, and generating a plurality of state prediction results; the change state including an offset of an edge position, a deformation degree of an edge shape, and a change trend of edge sharpness.

[0011] Preferably, the step S400 specifically includes the following sub-steps: Step S410, establishing an image coordinate system centered on the target, and converting the target motion parameters collected by the motion parameter detection device into motion parameters corresponding to the image coordinate system, including a motion speed vector and a motion acceleration vector; Step S420, for each state prediction result, calculating a spatial consistency between the predicted motion change and the converted motion parameters, the spatial consistency including a direction consistency and an amplitude consistency; Step S430, generating a matching degree evaluation value of each state prediction result according to the direction consistency and the amplitude consistency; wherein the direction consistency is obtained by calculating a cosine value of an included angle between a predicted motion direction and a measured motion direction, and the amplitude consistency is obtained by calculating a ratio of a predicted motion amplitude to a measured motion amplitude.

[0012] Preferably, the step S600 specifically includes the following sub-steps: Step S610, generating an adjustment instruction containing a target acquisition frequency parameter, the adjustment instruction being sent to a drive controller of the image acquisition device through a device control interface; Step S620, analyzing the adjustment instruction through the drive controller, and reconfiguring a sampling timing circuit of the image acquisition device, including adjusting an exposure time, a frame readout period, and a pixel clock frequency; Step S630, verifying a stability of an image data stream after the acquisition frequency adjustment, and if data loss or timing error is detected, falling back to a previous acquisition frequency and generating an adjustment instruction again.

[0013] Preferably, the step S700 specifically includes the following sub-steps: Step S710, collecting real-time image data of the target through the image acquisition device, and calculating a sharpness index of the real-time image data; wherein a gradient amplitude of the image is calculated by using a gradient operator, and an average value of the gradient amplitude is calculated as the sharpness index; Step S720, comparing the definition index with a preset quality threshold; wherein the quality threshold is dynamically adjusted based on historical definition data; Step S730, if the definition index of the real-time image data is lower than the quality threshold within a preset time period, determining that the image quality is degraded, and re-executing steps S100 to S600.

[0014] Preferably, the calculation of the definition index in step S710 further comprises: Step S711, performing convolution processing on the real-time image data using a Laplacian operator, calculating the second derivative of the image, and obtaining a Laplacian response map; Step S712, calculating the variance value of the Laplacian response map as an auxiliary definition index, and combining the gradient amplitude average value for weighted fusion to generate a comprehensive definition index; Step S713, dynamically adjusting the weight of the definition index according to the image content complexity.

[0015] Preferably, it further comprises an image quality degradation reason identification step, which comprises the following sub-steps: Step S740, recording the corresponding time period when the image quality is determined to be degraded as the determination time, and obtaining the real-time motion parameter change rate collected by the motion parameter detection device at the determination time, including motion acceleration and motion jitter; Step S750, if the motion parameter change rate exceeds a preset motion threshold, it is determined that the image quality degradation is caused by motion blur; wherein the motion blur includes image degradation caused by high-speed motion of the target or vibration of the collection device; Step S760, if the motion parameter change rate does not exceed the motion threshold, and the definition index presents irregular fluctuations or mutations, it is determined that the image quality degradation is caused by environmental interference; the environmental interference includes light change, sensor noise or electromagnetic interference.

[0016] Preferably, in step S760, when it is determined that the image quality degradation is caused by environmental interference, the following sub-steps are further included: Step S771, maintaining the current collection frequency of the image collection device unchanged; Step S772, processing the collected real-time image data through a filtering algorithm; Step S773, monitoring the image quality change of the real-time image data, if the definition index recovers to above the quality threshold, the current processing flow is continued, otherwise the adjustment flow of re-executing steps S100 to S600 is triggered.

[0017] An intelligent sensor data fusion system based on deep learning, which is applied to the intelligent sensor data fusion method based on deep learning, comprising: An image acquisition module for acquiring image data of the target, comprising an image sensor with adjustable acquisition frequency and a matching optical lens; A motion parameter detection module for acquiring motion parameters of the target, comprising a laser displacement sensor, an encoder or an inertial measurement unit; A data processing module connected with the image acquisition module and the motion parameter detection module, for acquiring historical images collected by the image acquisition device and extracting edge feature information in the historical images; acquiring motion parameters of the target collected by the motion parameter detection device in real time; predicting image states of subsequent frames of the image acquisition device under multiple candidate acquisition frequencies based on the edge feature information, generating multiple state prediction results; calculating matching degree evaluation values of the state prediction results according to real-time data of the target motion parameters; selecting an acquisition frequency corresponding to a state prediction result with the highest matching degree evaluation value from the candidate acquisition frequencies as an optimized acquisition frequency; A control module connected with the data processing module and the image acquisition module, for generating an acquisition frequency adjustment instruction according to the optimized acquisition frequency and sending the acquisition frequency adjustment instruction to a drive controller of the image acquisition module; A quality monitoring module connected with the image acquisition module, for monitoring real-time image data collected after adjustment in real time, judging whether the image quality decreases by calculating a definition index and comparing the definition index with a preset threshold, and sending a re-execution instruction to the data processing module when it is judged that the image quality decreases.

[0018] Advantages of the present application: Firstly, the intelligent sensor data fusion method based on deep learning provided by the present application fully perceives the motion state of the target by fusing the edge feature information of the image and the real-time motion parameters of the target, generates predicted future image states under different frequencies based on the edge feature information, and then combines the real-time motion parameters for matching degree evaluation, extracts the most matched frequency as the acquisition frequency of the image, so that the adjustment of the acquisition frequency is more accurate; and then, when the sensor data fusion is performed, the fusion effect of the data can be effectively improved.

[0019] Secondly, the intelligent sensor data fusion method based on deep learning provided by the present application can automatically reduce the acquisition frequency when the target moves slowly or is stationary, avoid collecting a large amount of redundant image data, thereby effectively reducing the data storage space and the computational burden of subsequent image processing, and improving the resource utilization rate.

[0020] Thirdly, the intelligent sensor data fusion method based on deep learning provided by the present application can distinguish whether the image quality decreases due to motion blur or environmental interference by further providing an image quality decrease reason identification mechanism, and take targeted processing measures, such as maintaining the acquisition frequency and performing filtering processing under environmental interference to avoid unnecessary frequency adjustment; thereby, the processing efficiency of the system is further improved. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the basic process of a deep learning-based intelligent sensor data fusion method provided by the present invention. Detailed Implementation

[0022] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0023] Step S100: Acquire historical images acquired by the image acquisition device and extract edge feature information from the historical images.

[0024] In this step, the image acquisition device continuously acquires image data of the target object. This historical image data is transmitted to the data processing module. The data processing module processes these historical images to extract edge feature information from the images. Edge features are areas in the image where brightness changes drastically; they typically represent the outline and boundaries of an object and are crucial for describing the object's shape and movement.

[0025] Specifically, step S100, which involves extracting edge feature information from historical images, includes the following sub-steps: Step S110: Preprocess the historical image by converting the color image to a grayscale image through grayscale processing, and use a median filtering algorithm to eliminate random noise interference in the image.

[0026] First, to simplify subsequent processing and highlight brightness information, the color historical image is converted into a grayscale image. Then, to remove random noise (such as salt-and-pepper noise) that may have been introduced during image acquisition, a median filtering algorithm is used to process the grayscale image.

[0027] Step S120: The preprocessed grayscale image is processed by an edge detection algorithm to generate a preliminary edge feature map.

[0028] Edge detection algorithms are applied to noise-removed grayscale images to identify edges in the image. For example, the Canny operator is used with a multi-level detection algorithm to detect edges over a wide range.

[0029] Step S130: Extract features from the preliminary edge feature map to obtain contour feature information; the contour feature information includes edge position, edge length and edge direction data.

[0030] On the basis of the preliminary edge feature map, more specific contour feature information is further extracted through a contour finding algorithm (such as Suzuki algorithm or findContours function in OpenCV). For each detected contour, its geometric properties are calculated, including: Edge position: the coordinates of each pixel point on the contour, or the centroid coordinates of the contour.

[0031] Edge length: the perimeter of the contour, which can be obtained by calculating the distance between adjacent pixel points on the contour and accumulating.

[0032] Edge direction data: the gradient direction of each point on the contour, or the overall direction of the contour (such as calculating the principal axis direction of the contour through principal component analysis PCA).

[0033] These contour feature information will serve as the basis data for subsequent prediction of target motion state. In this way, the present application can ensure the perfection of contour feature information and improve the accuracy of subsequent prediction.

[0034] Step S200, acquiring the motion parameter monitoring device to collect the motion parameters of the target in real time.

[0035] The system collects the physical motion parameters of the target object in real time. These parameters include the speed, acceleration, angular velocity, angular acceleration, displacement, etc. of the target. For example, a laser displacement sensor can provide the distance change between the target and the sensor, thereby calculating the speed; an encoder can measure the displacement and speed of rotary or linear motion; an inertial measurement unit (IMU) can provide the attitude, angular velocity and acceleration information of the target in three-dimensional space. These real-time motion parameters are transmitted to the data processing module as objective basis for evaluating the image state prediction results.

[0036] Step S300, based on the edge feature information, predicting the image state of the subsequent frame of the image acquisition device under multiple candidate acquisition frequencies, generating multiple state prediction results.

[0037] In the data processing module, the edge feature information extracted in step S100 is used in combination with a deep learning model (such as a recurrent neural network RNN, a long short-term memory network LSTM or a Transformer sequence prediction model) to predict the future image state of the target under different acquisition frequencies. Specifically, in step S330, a long short-term memory network (LSTM) model is used for state prediction. The model input is the historical edge feature sequence (including edge position, length and direction), and the output is the edge change state of the subsequent frame. The model training uses mean square error loss function, the optimizer is Adam, and the training data comes from historical images and corresponding motion parameters.

[0038] Specifically, the step S300 includes the following sub-steps: Step S310, according to the extracted edge feature information, the position change amount of the edge feature in the continuous frames is calculated by the inter-frame difference method, and the motion speed and motion direction of the target are calculated in combination with the frame time interval.

[0039] The application uses the edge feature information extracted in the continuous historical frames to obtain the displacement of the edge feature in the image coordinate system by the inter-frame difference method (for example, calculating the difference of the centroid position or the key point position of the same edge feature in adjacent frames). In combination with the known frame time interval, the motion speed (pixel / frame) and motion direction of the target on the image plane can be calculated.

[0040] Step S320, based on the motion speed and motion direction, the motion trend corresponding relationship of the target is established, which includes uniform motion corresponding relationship and uniform acceleration motion corresponding relationship, and is used to describe the displacement trajectory of the target in the image coordinate system.

[0041] According to the historical motion speed and motion direction data, the motion trend of the target is modeled by a motion model (such as Kalman filter or particle filter). These motion trend corresponding relationships can include: Uniform motion corresponding relationship: it is assumed that the target maintains constant speed and direction motion in a short time.

[0042] Uniform acceleration motion corresponding relationship: it is assumed that the target maintains constant acceleration motion in a short time.

[0043] The uniform speed and uniform acceleration motion trend corresponding relationship is used to predict the displacement trajectory of the target at future time.

[0044] Step S330, a plurality of different candidate acquisition frequencies are randomly set, according to the motion trend corresponding relationship, the change state of the edge feature in the subsequent frames under the time interval corresponding to the different candidate acquisition frequencies is predicted, and a plurality of state prediction results are generated; the change state includes the offset of the edge position, the deformation degree of the edge shape and the change trend of the edge definition.

[0045] In the application, a plurality of different candidate acquisition frequencies are randomly set, which can be set according to the specific application scene and the performance of the image acquisition device, for example, it can be set to 5Hz, 10Hz, 15Hz and a series of discrete values, or it can be randomly sampled in a certain continuous interval, the range of the candidate acquisition frequency is set based on the minimum acquisition frequency and the maximum acquisition frequency of the image acquisition device, and it is uniformly sampled or dynamically selected according to the historical optimization record in the acquisition frequency interval. That is, the person skilled in the art can set the appropriate candidate frequency range and sampling strategy based on the minimum and maximum acquisition frequencies of the image acquisition device, the target motion speed range and the acceptable image blur degree and other factors according to the actual needs.

[0046] For each candidate acquisition frequency, the specific change state of the target edge feature in the next frame is predicted according to its corresponding time interval (i.e. the inverse of the candidate acquisition frequency) and the motion trend correspondence established in step S320. These change states include: The offset of the edge position: the displacement of the predicted target edge in the image coordinate system relative to the current position.

[0047] The deformation degree of the edge shape: the shape change that the target edge may undergo during the movement, such as stretching, compression or rotation due to perspective effect or target shape deformation. This can be quantified by comparing the similarity (such as Hausdorff distance or shape context matching) of the predicted edge contour and the current contour.

[0048] The change trend of the edge sharpness: the degree of decline in edge sharpness that may occur due to motion blur effect at different acquisition frequencies. For example, at a higher acquisition frequency, the motion blur is lower and the edge sharpness is higher; while at a lower acquisition frequency, the motion blur is higher and the edge sharpness is lower. This can be estimated by convolving the predicted image with a simulated motion blur kernel and calculating its sharpness index (such as gradient amplitude or Laplacian variance).

[0049] Each candidate acquisition frequency generates a corresponding state prediction result, which will be used for subsequent matching degree evaluation.

[0050] Step S400, according to the real-time data of the target motion parameters, calculate the matching degree evaluation value of each state prediction result.

[0051] Therefore, the present application compares the multiple state prediction results generated in step S300 with the real-time motion parameters obtained in step S200 through the data processing module, to evaluate the accuracy of each prediction result.

[0052] Specifically, the step S400 includes the following sub-steps: Step S410, establish an image coordinate system centered on the target, and convert the target motion parameters collected by the motion parameter detection device into motion parameters corresponding to the image coordinate system, including the motion velocity vector and the motion acceleration vector.

[0053] Since the motion parameter detection device (such as IMU) usually provides motion parameters in the physical world coordinate system, while the image state prediction is carried out in the image coordinate system, coordinate conversion is needed. Through the camera intrinsic and extrinsic parameters (if known), the velocity, acceleration and other motion parameters in the physical world coordinate system are projected into the image coordinate system to obtain the motion velocity vector and the motion acceleration vector of the target in the image plane.

[0054] Step S420, for each state prediction result, calculate the spatial consistency between its predicted motion change and the converted motion parameter, which includes direction consistency and amplitude consistency.

[0055] For each state prediction result (including edge position offset, shape deformation and definition change) under each candidate acquisition frequency, it is converted into a predicted motion vector (for example, a predicted edge centroid displacement vector). Then, this predicted motion vector is compared with the real-time motion parameter (such as real-time motion velocity vector) converted in step S410, and the spatial consistency between them is calculated: Direction consistency: obtained by calculating the cosine value of the included angle between the predicted motion direction and the measured motion direction. The closer the cosine value is to 1, the higher the direction consistency is.

[0056] Amplitude consistency: obtained by calculating the ratio of the predicted motion amplitude (for example, the predicted displacement) to the measured motion amplitude (for example, the real-time velocity multiplied by the predicted time interval). The closer the ratio is to 1, the higher the amplitude consistency is.

[0057] Step S430, according to the direction consistency and the amplitude consistency, generate a matching degree evaluation value of each state prediction result.

[0058] The direction consistency and the amplitude consistency are weighted and fused to generate a comprehensive matching degree evaluation value. For example, the following formula can be used: matching degree evaluation value = w1*direction consistency + w2*amplitude consistency. Wherein, w1 and w2 are weight coefficients, which can be adjusted according to actual application scenarios, for example, w1 and w2 can be set to 0.5 to reflect the importance of different consistency indicators. The higher the matching degree evaluation value is, the more consistent the image state prediction result under the candidate acquisition frequency is with the actual motion parameter.

[0059] Step S500, select the acquisition frequency corresponding to the state prediction result with the highest matching degree evaluation value from the candidate acquisition frequencies as the optimized acquisition frequency.

[0060] In this way, the application compares the matching degree evaluation values corresponding to all candidate acquisition frequencies through the data processing module, and selects the candidate acquisition frequency with the highest evaluation value as the current optimized acquisition frequency. This optimized acquisition frequency represents the acquisition frequency that can most accurately capture the target motion and has the best image quality under the current target motion state, thereby improving the efficiency of the application in debugging work.

[0061] Step S600, according to the optimized acquisition frequency, control the image acquisition device to adjust the acquisition frequency.

[0062] After the data processing module determines the optimized acquisition frequency, an adjustment instruction is sent to the image acquisition module according to the optimized acquisition frequency. In this way, the present application realizes adaptive optimization of the acquisition frequency through the fusion prediction of the motion parameters and the image features.

[0063] Specifically, the step S600 includes the following sub-steps: Step S610, generating an adjustment instruction containing a target acquisition frequency parameter, and sending the adjustment instruction to a drive controller of the image acquisition device through a device control interface.

[0064] The control module generates a standard device control instruction according to the optimized acquisition frequency. The instruction contains a new acquisition frequency parameter and is sent to the drive controller inside the image acquisition device through the device control interface (such as USB, Ethernet, MIPI, etc.) of the image acquisition device.

[0065] Step S620, parsing the adjustment instruction through the drive controller and reconfiguring the sampling timing circuit of the image acquisition device, including adjusting the exposure time, frame readout period and pixel clock frequency.

[0066] The drive controller of the image acquisition device receives and parses the adjustment instruction. According to the optimized acquisition frequency in the instruction, the drive controller will reconfigure the sampling timing circuit inside the image acquisition device. This usually involves: Adjusting the exposure time: The exposure time determines the time for the sensor to receive light. At a higher acquisition frequency, the exposure time is usually shortened to avoid motion blur; at a lower acquisition frequency, the exposure time can be appropriately lengthened to improve image brightness.

[0067] Frame readout period: The frame readout period is the time required for the sensor to read a frame of image. The adjustment of the acquisition frequency directly affects the frame readout period.

[0068] Pixel clock frequency: The pixel clock frequency determines the speed of data transmission for each pixel. Adjusting the acquisition frequency may require adjusting the pixel clock frequency accordingly to ensure the synchronization and integrity of data transmission.

[0069] In this way, through the coordinated adjustment of these parameters, the image acquisition device can perform image acquisition at the new optimized acquisition frequency, improving the debugging efficiency.

[0070] Step S630, verifying the stability of the image data stream after the adjustment of the acquisition frequency, and if data loss or timing error is detected, falling back to the previous acquisition frequency and re-generating the adjustment instruction.

[0071] After adjusting the acquisition frequency, the control module or the firmware inside the image acquisition device monitors the adjusted image data stream. If data loss (e.g. frame loss) or timing error (e.g. abnormal frame synchronization signal) is detected, it indicates that the current hardware configuration or transmission bandwidth cannot support the new acquisition frequency. In this case, the system will immediately fall back to the previous acquisition frequency before adjustment and regenerate the adjustment instruction, which may try a more conservative acquisition frequency or trigger a fault alarm. In this way, the present application ensures the stability and reliability of system adjustment.

[0072] Step S700, monitoring the real-time image data collected after adjustment, when judging that the image quality is decreased, re-executing steps S100 to S600.

[0073] After the image acquisition device works at the optimized acquisition frequency, the quality monitoring module continuously monitors the real-time image data collected.

[0074] Specifically, the step S700 includes the following sub-steps: Step S710, collecting real-time image data of the target by the image acquisition device, and calculating the definition index of the real-time image data; wherein the gradient amplitude of the image is calculated by using a gradient operator, and the average value of the gradient amplitude is calculated as the definition index.

[0075] The quality monitoring module receives the real-time image data. In order to quantify the image quality, the definition index of the image is calculated. The present application calculates the gradient amplitude of the image by using a gradient operator (such as Sobel, Prewitt or Scharr operator). The gradient amplitude reflects the degree of change in pixel gray scale in the image, and the clearer the edge, the larger the gradient amplitude. The average value of the gradient amplitude of all pixel points in the image is calculated as the definition index of the image. The higher the average gradient amplitude, the clearer the image.

[0076] Step S711, performing convolution processing on the real-time image data by using a Laplace operator, calculating the second derivative of the image and obtaining a Laplace response map; In order to more comprehensively evaluate the image definition, the present application uses a Laplace operator in addition to the gradient operator. By performing Laplace convolution processing on the real-time image data, a Laplace response map can be obtained, in which the size of the response value reflects the degree of change in gray scale of the local area of the image.

[0077] Step S712, calculating the variance value of the Laplace response map as an auxiliary definition index, and combining the average value of the gradient amplitude to generate a comprehensive definition index by weighted fusion; The variance value of the Laplacian response map can be used as an auxiliary sharpness indicator. The sharper the image, the larger the variance of its Laplacian response map. The gradient amplitude average calculated in step S710 and the Laplacian response map variance value calculated in this step are weighted and fused to generate a comprehensive sharpness indicator. For example: comprehensive sharpness indicator = w(gradient) * Laplacian response map variance value Gradient amplitude Mean + w (Laplace) * Laplacian response map variance value; Where w(gradient) and w(Lap) are the weight coefficients of (gradient amplitude average) and (Laplacian response map variance value), respectively. The weight system can be adjusted according to the actual application scenario, for example, w(gradient) and w(Lap) can be set to 0.5.

[0078] Step S713, dynamically adjusting the weight of the sharpness indicator according to the image content complexity.

[0079] Image content complexity can affect the sensitivity of the sharpness indicator. For example, for images with rich texture, the gradient amplitude and Laplacian variance may be generally high. In order to more accurately evaluate, the weights of the gradient amplitude average and the Laplacian response map variance can be dynamically adjusted according to the image content complexity. Image content complexity can be measured by image entropy (measuring image information and texture richness) or texture analysis (such as gray level co-occurrence matrix GLCM) and other conventional methods. For example, when the image complexity is high, the weight of the gradient amplitude average can be appropriately reduced, and the weight of the Laplacian variance can be increased, and vice versa.

[0080] Step S720, comparing the sharpness indicator with a preset quality threshold; wherein the quality threshold is dynamically adjusted based on historical sharpness data.

[0081] In the present application, the quality monitoring module compares the comprehensive sharpness indicator calculated with a preset quality threshold. This quality threshold is not fixed, but is dynamically adjusted based on historical image sharpness data. For example, the statistical distribution (such as mean, standard deviation) of historical sharpness indicators or through machine learning models (such as support vector machine SVM or neural network) can be trained to adapt to different scenes and targets. When the sharpness indicator is lower than the quality threshold, it indicates that the image quality may have decreased.

[0082] Step S730, if the sharpness indicator of the real-time image data is lower than the quality threshold within a preset time period, it is determined that the image quality has decreased, and steps S100 to S600 are re-executed.

[0083] To avoid misjudgment caused by instantaneous fluctuation, the system continuously monitors the definition index within a preset time period (e.g. N consecutive frames or T seconds). If the definition index continuously falls below the quality threshold within the time period, the image quality is finally determined to be degraded. Once the image quality is determined to be degraded, the system triggers the execution of steps S100 to S600 again, i.e. edge feature extraction, motion parameter acquisition, state prediction, matching degree evaluation and acquisition frequency optimization, in order to find a new optimal acquisition frequency to restore the image quality. Embodiments

[0084] This embodiment is based on embodiment one, and further provides an image quality degradation cause identification step to achieve more intelligent fault diagnosis and processing.

[0085] After determining the image quality degradation in step S730, the image quality degradation cause identification step is further included, which specifically includes the following sub-steps: In step S740, the corresponding time period when the image quality is determined to be degraded is recorded as the determination time, and the real-time motion parameter change rate of the motion parameter detection device at the determination time is acquired, including the motion acceleration and the motion jitter amount.

[0086] When the quality monitoring module determines that the image quality is degraded, the time point or time period when the degradation occurs is recorded as the determination time. At the same time, the real-time motion parameter change rate at the determination time is obtained from the motion parameter detection module. These change rates include: Motion acceleration: change rate of target speed.

[0087] Motion jitter amount: unexpected and high-frequency micro-movement of the target within a short time, which can be obtained by high-pass filtering or calculating the variance of the motion parameters (such as displacement and speed).

[0088] These parameters help to determine whether the image quality degradation is related to the dramatic change of the target motion state.

[0089] In step S750, if the motion parameter change rate exceeds a preset motion threshold, it is determined that the image quality degradation is caused by motion blur; wherein the motion blur includes image degradation caused by high-speed motion of the target or vibration of the acquisition device.

[0090] The present application compares the obtained motion parameter change rate with the preset motion threshold. If the motion acceleration or the motion jitter amount exceeds the preset threshold (e.g. the acceleration exceeds a certain critical value, or the jitter amount exceeds a certain allowed range), it is determined that the main cause of the image quality degradation is motion blur. Motion blur may be due to high-speed motion of the target itself, resulting in image smearing on the sensor within the exposure time; or it may be due to vibration of the image acquisition device itself, resulting in image relative motion and blur.

[0091] If the motion parameter change rate does not exceed the motion threshold, it indicates that the target motion is relatively stable, and the image quality degradation is not mainly caused by motion blur. At this time, the change pattern of the sharpness index will be further analyzed. If the sharpness index presents irregular fluctuations (e.g., high and low in a short time, without obvious trend decline) or sudden sharp decline, it is determined that the image quality degradation is caused by environmental interference. Environmental interference may include:

[0092] If the motion parameter change rate does not exceed the motion threshold, it indicates that the target motion is relatively stable, and the image quality degradation is not mainly caused by motion blur. At this time, the change pattern of the sharpness index will be further analyzed. If the sharpness index presents irregular fluctuations (e.g., high and low in a short time, without obvious trend decline) or sudden sharp decline, it is determined that the image quality degradation is caused by environmental interference. Environmental interference may include: Light changes: sudden darkening or brightening of environmental light, causing underexposure or overexposure of the image.

[0093] Sensor noise: random noise generated by the image sensor itself, especially noticeable under low light or high gain.

[0094] Electromagnetic interference: interference of the surrounding electromagnetic environment to the image acquisition device, causing the image to appear stripes, snowflakes, etc. Embodiment

[0095] This embodiment is based on Embodiment Two, and further provides a targeted processing strategy when it is determined that the image quality degradation is caused by environmental interference.

[0096] In step S760, when it is determined that the image quality degradation is caused by environmental interference, the following sub-steps are included: Step S771, maintain the current acquisition frequency of the image acquisition device unchanged.

[0097] Unlike motion blur, environmental interference is usually unrelated to the acquisition frequency. Therefore, when it is determined that the image quality degradation is caused by environmental interference, the system does not immediately re-adjust the acquisition frequency, but maintains the current acquisition frequency of the image acquisition device unchanged. This avoids unnecessary frequency adjustment and reduces system computing resource consumption.

[0098] Step S772, processing the collected real-time image data by a filtering algorithm.

[0099] In order to cope with environmental interference, the quality monitoring module will filter the real-time collected image data. For example: For light changes: adaptive histogram equalization, gamma correction or local brightness enhancement algorithm can be used to improve the brightness and contrast of the image.

[0100] For sensor noise: Gaussian filter, bilateral filter, non-local mean filter (NLM) or deep learning denoising model can be used to remove random noise in the image while preserving image details as much as possible.

[0101] For electromagnetic interference: Fourier transform can be used to analyze the interference frequency, and a corresponding notch filter or band-stop filter can be designed to remove periodic interference.

[0102] The present application effectively improves the image quality through these filtering algorithms.

[0103] Step S773, monitor the image quality change of real-time image data, if the clarity index recovers to above the quality threshold, continue to maintain the current processing flow, otherwise trigger the adjustment process of steps S100 to S600.

[0104] After filtering, the quality monitoring module will continue to monitor the clarity index of real-time image data. If the clarity index can recover to above the preset quality threshold after filtering, it indicates that the environmental interference has been effectively suppressed and the image quality has returned to normal, and the system will continue to maintain the current acquisition frequency and processing flow. However, if the clarity index still cannot recover to above the quality threshold after filtering, or the environmental interference continues to worsen, it indicates that the current filtering is not enough to solve the problem. At this time, the system will trigger the adjustment process of steps S100 to S600, i.e. re-perform comprehensive sensor data fusion and acquisition frequency optimization to seek a more fundamental solution. Embodiment

[0105] The present embodiment provides an intelligent sensor data fusion system based on deep learning, comprising: Image acquisition module: for acquiring image data of the target, including an image sensor with adjustable acquisition frequency and a matching optical lens. The image sensor is responsible for converting optical signals into electrical signals, and the optical lens is responsible for focusing light. The module also includes a drive controller and a sampling timing circuit for receiving control instructions and adjusting acquisition frequency, exposure time and other parameters.

[0106] Motion parameter detection module: for acquiring motion parameters of the target, including laser displacement sensor, encoder or inertial measurement unit. These sensors acquire real-time physical motion information of the target such as speed, acceleration and displacement, and transmit data to the data processing module.

[0107] Data processing module: connected with the image acquisition module and the motion parameter detection module, is the core processing unit of the system. It includes: Edge feature extraction unit: for acquiring historical images collected by the image acquisition device and extracting edge feature information in the historical images, such as edge position, length and direction.

[0108] Motion parameter conversion unit: for converting the target motion parameter collected by the motion parameter detection device into a motion parameter corresponding to the image coordinate system.

[0109] State prediction unit: based on the edge feature information, predict the image state of the subsequent frame of the image collection device under multiple candidate collection frequencies, generate multiple state prediction results, including edge position offset, shape deformation degree and clarity change trend.

[0110] Matching degree evaluation unit: according to the real-time data of the target motion parameter, calculate the matching degree evaluation value of each state prediction result, evaluate the consistency degree of the prediction result and the actual motion.

[0111] Optimized frequency selection unit: select the collection frequency corresponding to the state prediction result with the highest matching degree evaluation value from the candidate collection frequency as the optimized collection frequency.

[0112] Control module: connected with the data processing module and the image collection module, used for generating collection frequency adjustment instruction according to the optimized collection frequency, and sending to the driving controller 130 of the image collection module. It includes adjustment instruction generation unit, responsible for converting optimized frequency into specific hardware control instruction.

[0113] Quality monitoring module: connected with the image collection module, used for real-time monitoring of the real-time image data collected after adjustment, judging whether the image quality is decreased by calculating the clarity index and comparing with the preset threshold value, and sending the re-execution instruction to the data processing module when judging the image quality is decreased. It includes: Clarity calculation unit: for calculating the clarity index of real-time image data, such as the weighted fusion of gradient amplitude average value and Laplace response map variance.

[0114] Quality threshold management unit: for managing and dynamically adjusting the preset quality threshold.

[0115] Cause identification unit: for identifying the cause of image quality decline, distinguishing motion blur and environmental interference.

[0116] Filter processing unit: when it is determined that it is environmental interference, filter processing is performed on the image data to improve the quality.

[0117] Those skilled in the art will appreciate that embodiments of the present application can be readily used as a method, a system or a computer program product. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-usable storage media (or computer- readable storage media) having computer-usable program code embodied in the medium. The medium can be any available storage media that can be accessed by a computer. By way of example, and not limitation, such computer-usable storage media can include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other storage medium(s) that can be used to carry or store desired computer program code in the form of instructions or data structures and that can be accessed by a computer. Also, the present application can be embodied in a computer program product that can be traded as goods or merchandise, through a computer-based platform or via a network. Figure 1 one or more flows and / or blocks Figure 1 one or more flows and / or blocks

[0118] It should be noted that the above-mentioned embodiments are only used to illustrate but not to limit the technical solutions of the present application. Although the present application is described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or replaced equivalently without departing from the spirit and scope of the technical solutions of the present application, and they should be covered in the scope of the claims of the present application.

Claims

1. A deep learning-based intelligent sensor data fusion method, characterized in that, The method comprises the following steps: Step S100, acquiring historical images collected by an image collection device and extracting edge feature information in the historical images; Step S200, acquiring motion parameters of a target collected by a motion parameter monitoring device in real time; Step S300, predicting image states of subsequent frames of the image collection device under a plurality of candidate collection frequencies based on the edge feature information, and generating a plurality of state prediction results; Step S400, calculating matching degree evaluation values of the state prediction results according to real-time data of the target motion parameters; Step S500, selecting a collection frequency corresponding to a state prediction result with the highest matching degree evaluation value from the candidate collection frequencies as an optimized collection frequency; Step S600, controlling the image collection device to adjust the collection frequency according to the optimized collection frequency; Step S700, monitoring real-time image data collected after the adjustment, and re-executing steps S100 to S600 when it is judged that the image quality is reduced. 2.The deep learning based intelligent sensor data fusion method of claim 1, wherein, The step S100 of extracting the edge feature information in the historical images specifically comprises the following sub-steps: Step S110, pre-processing the historical images, converting color images into grayscale images through grayscale processing, and eliminating random noise interference in the images by using a median filter algorithm; Step S120, processing the pre-processed grayscale images by using an edge detection algorithm to generate a preliminary edge feature map; Step S130, extracting features from the preliminary edge feature map to acquire contour feature information; the contour feature information comprises edge position, edge length and edge direction data.

3. The deep learning based intelligent sensor data fusion method of claim 2, wherein, The step S300 specifically comprises the following sub-steps: Step S310, calculating position variation amounts of edge features in continuous frames by using an inter-frame difference method according to the extracted edge feature information, and calculating motion speed and motion direction of the target in combination with frame time intervals; Step S320, establishing a motion trend corresponding relationship of the target based on the motion speed and the motion direction; the motion trend corresponding relationship comprises uniform motion corresponding relationship and uniform acceleration motion corresponding relationship, and is used for describing displacement trajectories of the target in an image coordinate system; Step S330, randomly setting a plurality of different candidate collection frequencies, predicting variation states of the edge features in subsequent frames under time intervals corresponding to the different candidate collection frequencies according to the motion trend corresponding relationship, and generating a plurality of state prediction results; the variation states comprise offset amounts of edge positions, deformation degrees of edge shapes and variation trends of edge sharpness.

4. The deep learning based intelligent sensor data fusion method of claim 3, wherein, The step S400 specifically comprises the following sub-steps: Step S410, establishing an image coordinate system with the target as the center, and converting the target motion parameters collected by the motion parameter monitoring device into motion parameters corresponding to the image coordinate system, including a motion speed vector and a motion acceleration vector; Step S420, for each state prediction result, calculating spatial consistency between a predicted motion variation and the converted motion parameters; the spatial consistency comprises direction consistency and amplitude consistency; Step S430, according to the direction consistency and the amplitude consistency, a matching degree evaluation value of each state prediction result is generated; wherein, the direction consistency is obtained by calculating the cosine value of the included angle between the predicted motion direction and the measured motion direction, and the amplitude consistency is obtained by calculating the ratio of the predicted motion amplitude and the measured motion amplitude.

5. The deep learning based intelligent sensor data fusion method of claim 4, wherein, The step S600 specifically includes the following sub-steps: Step S610, an adjustment instruction containing the target acquisition frequency parameter is generated, and the adjustment instruction is sent to the driving controller of the image acquisition device through the device control interface; Step S620, the adjustment instruction is parsed by the driving controller, and the sampling timing circuit of the image acquisition device is reconfigured, including adjusting the exposure time, the frame readout period and the pixel clock frequency; Step S630, the stability of the image data stream after the acquisition frequency adjustment is verified, and if data loss or timing error is detected, the last acquisition frequency is rolled back and the adjustment instruction is re-generated.

6. The deep learning based intelligent sensor data fusion method of claim 5, wherein, The step S700 specifically includes the following sub-steps: Step S710, real-time image data of the target is acquired by the image acquisition device, and the definition index of the real-time image data is calculated; wherein, the gradient amplitude of the image is calculated by using the gradient operator, and the average value of the gradient amplitude is calculated as the definition index; Step S720, the definition index is compared with the preset quality threshold; wherein, the quality threshold is dynamically adjusted based on historical definition data; Step S730, if the definition index of the real-time image data is lower than the quality threshold within a preset time period, it is determined that the image quality is degraded, and steps S100 to S600 are re-executed.

7. The deep learning based intelligent sensor data fusion method of claim 6, wherein, The calculation of the definition index in step S710 further includes: Step S711, the real-time image data is convoluted by using the Laplace operator, the second derivative of the image is calculated, and the Laplace response map is obtained; Step S712, the variance value of the Laplace response map is calculated as an auxiliary definition index, and the gradient amplitude average value is combined for weighted fusion to generate a comprehensive definition index; Step S713, the weight of the definition index is dynamically adjusted according to the image content complexity. 8.The deep learning based intelligent sensor data fusion method of claim 6, wherein, Further comprising an image quality degradation reason identification step, specifically including the following sub-steps: Step S740, the corresponding time period when the image quality is determined to be degraded is recorded as the determination time, and the real-time motion parameter change rate, including the motion acceleration and the motion jitter, collected by the motion parameter detection device at the determination time is obtained; Step S750, if the motion parameter change rate exceeds the preset motion threshold, it is determined that the image quality degradation is caused by motion blur; wherein, the motion blur includes image degradation caused by high-speed motion of the target or vibration of the acquisition device; Step S760, if the motion parameter change rate does not exceed the motion threshold, and the definition index presents irregular fluctuations or mutations, it is determined that the image quality degradation is caused by environmental interference; the environmental interference includes light change, sensor noise or electromagnetic interference. 9.The deep learning based intelligent sensor data fusion method of claim 8, wherein, In step S760, when it is determined that the image quality degradation is caused by environmental interference, the following sub-steps are further included: Step S771, the current acquisition frequency of the image acquisition device is maintained unchanged; Step S772, processing the collected real-time image data through a filtering algorithm; Step S773, monitoring the image quality change of the real-time image data, if the definition index recovers to above the quality threshold, the current processing flow is continued to maintain, otherwise the adjustment flow of steps S100 to S600 is triggered to re-execute.

10. A deep learning based intelligent sensor data fusion system applied in the deep learning based intelligent sensor data fusion method according to any one of claims 1-9, characterized in that, Comprise; An image acquisition module for acquiring image data of a target, comprising an image sensor with adjustable acquisition frequency and a matching optical lens; A motion parameter detection module for acquiring motion parameters of the target, comprising a laser displacement sensor, an encoder or an inertial measurement unit; A data processing module connected with the image acquisition module and the motion parameter detection module, for acquiring historical images collected by the image acquisition device and extracting edge feature information in the historical images; Acquiring motion parameters of the target collected by the motion parameter detection device in real time; Based on the edge feature information, predicting the image state of the subsequent frame of the image acquisition device under multiple candidate acquisition frequencies, generating multiple state prediction results; calculating the matching degree evaluation value of each state prediction result according to the real-time data of the target motion parameters; Selecting the acquisition frequency corresponding to the state prediction result with the highest matching degree evaluation value from the candidate acquisition frequencies as the optimized acquisition frequency; A control module connected with the data processing module and the image acquisition module, for generating an acquisition frequency adjustment instruction according to the optimized acquisition frequency and sending it to the driving controller of the image acquisition module; A quality monitoring module connected with the image acquisition module, for monitoring the real-time image data collected after adjustment in real time, judging whether the image quality has decreased by calculating the definition index and comparing it with the preset threshold, and sending a re-execution instruction to the data processing module when judging that the image quality has decreased.

Images