Data processing method, device, equipment and medium

Abstract

The present invention relates to the technical field of automated control, and discloses a data processing method, device, equipment, and medium. The method includes obtaining photographic data, point cloud data, and weight data of a material, establishing a data quality model to output quality scores for the photographic data, point cloud data, and weight data, and performing weighted calculations based on basic weights to obtain a total quality score. If the total quality score meets the requirements, the operating parameters of the sensor are adjusted to continuously improve the total quality score. When the total quality score meets the requirements, the photographic data, point cloud data, and weight data are fused and processed to obtain material information. The present invention can improve the credibility of collected data based on multi-sensor collaboration and dynamic quality optimization, and can automatically adjust the multi-sensor operating parameters in a timely manner for dynamic scenarios with changing working conditions, thereby ensuring the ability to identify materials and efficiently obtaining accurate material information.

Description

Technical Field

[0001] The present invention relates to the field of automation control technology, and in particular to a data processing method, device, equipment and medium. Background Art

[0002] In high-precision material detection scenarios such as industrial automation and intelligent warehousing, accurate acquisition of material information is the key to achieving efficient operations. Existing material detection often relies on a single sensor, or although multiple sensors are used, there is a lack of effective coordination mechanism, resulting in errors in the acquired data and insufficient complementarity, making it difficult to accurately obtain material information, and it is even more difficult to effectively process dynamic scenarios. For example, in dynamic scenarios such as material movement changes and weight fluctuations, sensors are prone to data distortion under fixed operating parameters, but lack adaptive adjustment capabilities and still output invalid data. Moreover, during the data collection process, manual and frequent debugging of sensor operating parameters is often required, which is not only manpower-consuming, but also difficult to meet the needs of real-time or long-term operation scenarios. Summary of the Invention

[0003] The purpose of the present invention is to provide a data processing method, device, equipment and medium for the detection of materials in related technologies, which lacks an effective coordination mechanism between multiple sensors and makes it difficult to accurately obtain material information. The method can improve the credibility of collected data based on multi-sensor collaboration and dynamic quality optimization, automatically adjust the working parameters of multiple sensors in a timely manner for dynamic scenarios with changing working conditions, ensure the ability to identify materials, and efficiently obtain accurate material information.

[0004] In order to achieve the above-mentioned purpose, in a first aspect of the present invention, a data processing method is provided, comprising: S1, using a camera, a laser radar and a weight sensor to detect the material, and respectively obtaining the shooting data, point cloud data and weight data of the material; S2, establishing a data quality model for separating the background area and the material area of ​​the shooting data, and performing optical flow calculation on the background area and the material area respectively to obtain the average displacement, so as to output the quality score of the shooting data; obtaining the point cloud overlap rate of the point cloud data in the current frame and the previous frame, so as to output the quality score of the point cloud data; calculating the volatility of the weight data, so as to output the quality score of the weight data; S3, based on the quality scores of the shooting data, the point cloud data and the weight data, The basic weights are weightedly calculated to obtain the total quality score, and the basic weights are dynamically determined based on the average displacement and the point cloud overlap rate to determine whether the total quality score is less than the first threshold. If so, proceed to step S4, otherwise proceed to step S5; S4, based on the quality scores of the shooting data, point cloud data and weight data, adjust the working parameters of the camera, lidar and weight sensor accordingly, and obtain the updated shooting data, point cloud data and weight data, and return to step S2 to update the quality scores of the shooting data, point cloud data and weight data; S5, based on the quality scores of the shooting data, point cloud data and weight data, and combined with the total quality score, fuse the shooting data, point cloud data and weight data to obtain material information.

[0005] In the second aspect of the present invention, a data processing device is provided, comprising: an acquisition module for detecting materials using a camera, a laser radar and a weight sensor, and respectively acquiring shooting data, point cloud data and weight data of the materials; a modeling module for establishing a data quality model for separating the background area and the material area of ​​the shooting data, and performing optical flow calculation on the background area and the material area respectively to acquire the average displacement, so as to output the quality score of the shooting data; acquiring the point cloud overlap rate of the point cloud data in the current frame and the previous frame, so as to output the quality score of the point cloud data; calculating the volatility of the weight data, so as to output the quality score of the weight data; a judgment module for judging the quality scores of the shooting data, the point cloud data and the weight data through a basic The weights are weightedly calculated to obtain the total quality score. The basic weights are dynamically determined based on the average displacement and the point cloud overlap rate to determine whether the total quality score is less than the first threshold. If so, the adjustment module is entered; otherwise, the fusion module is entered; the adjustment module adjusts the working parameters of the camera, lidar and weight sensor based on the quality scores of the shooting data, point cloud data and weight data, and obtains the updated shooting data, point cloud data and weight data, and returns to the modeling module to update the quality scores of the shooting data, point cloud data and weight data; the fusion module fuses the shooting data, point cloud data and weight data based on the quality scores of the shooting data, point cloud data and weight data, and combines them with the total quality score to obtain material information.

[0006] In a third aspect of the present invention, an electronic device is provided, comprising a memory and a processor coupled to the memory, wherein the processor is configured to execute the method of the first aspect based on instructions stored in the memory.

[0007] In a fourth aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method of the first aspect is implemented.

[0008] The technical solution of the present invention can improve the reliability of material information through a closed-loop process of multi-sensor data acquisition → quality assessment → dynamic adjustment → fusion output. It is based on multi-sensor collaboration and dynamic quality optimization, specifically identifying low-quality data such as blurred images, sparse point clouds, and fluctuating weights through quality assessment, and optimizing the acquisition process by adjusting the sensor working parameters, thereby reducing invalid data caused by improper sensor configuration or environmental interference, improving the quality of single-source data, and being able to adapt to dynamic scenarios such as material movement changes, weight fluctuations, etc., to avoid the problem of data distortion caused by sensors under fixed working parameters. Multi-sensor fusion is combined with weighted calculation of quality scores to make full use of high-reliability data, reduce the error of a single sensor, and improve the comprehensive accuracy of obtaining material information. Through automated quality assessment and parameter adjustment, the need for manual debugging of sensors is effectively reduced. It is suitable for scenarios that require real-time or long-term operation, especially automated scenarios that require high-precision material detection, such as industrial automation, smart warehousing and other applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the technical solutions in the embodiments of the present application, a brief introduction will be given below to the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0010] Figure 1 is a flow chart of a method for processing data provided in an embodiment;

[0011] Figure 2 is a processing flow chart of the data quality model provided in the embodiment;

[0012] Figure 3 is a schematic structural diagram of a data processing device provided in an embodiment;

[0013] Figure 4 It is a schematic structural diagram of an electronic device provided in an embodiment. DETAILED DESCRIPTION

[0014] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0015] The present application is described below with reference to specific embodiments in conjunction with the accompanying drawings.

[0016] like Figure 1 As shown, a data processing method of the present embodiment includes: S1, using a camera, a laser radar and a weight sensor to detect the material, and respectively obtain the shooting data, point cloud data and weight data of the material; S2, establishing a data quality model for separating the background area and the material area of ​​the shooting data, and performing optical flow calculation on the background area and the material area respectively to obtain the average displacement, so as to output the quality score of the shooting data; obtaining the point cloud overlap rate of the point cloud data in the current frame and the previous frame, so as to output the quality score of the point cloud data; calculating the volatility of the weight data, so as to output the quality score of the weight data; S3, weighting the quality scores of the shooting data, the point cloud data and the weight data by the basic weight The total quality score is calculated, and the basic weight is dynamically determined based on the average displacement and the point cloud overlap rate. It is judged whether the total quality score is less than the first threshold. If so, step S4 is performed, otherwise step S5 is performed; S4, based on the quality scores of the shooting data, point cloud data and weight data, the working parameters of the camera, lidar and weight sensor are adjusted accordingly, and the updated shooting data, point cloud data and weight data are obtained, and the process returns to step S2 to update the quality scores of the shooting data, point cloud data and weight data; S5, based on the quality scores of the shooting data, point cloud data and weight data, the shooting data, point cloud data and weight data are fused in combination with the total quality score to obtain material information.

[0017] In step S2, a background subtraction method (such as the MOG2 model) can be used to separate the background area and the material area of ​​the captured data. Optical flow calculations are performed on the background area and the material area respectively, and the pixel displacements of adjacent frames of the corresponding areas can be obtained. Furthermore, the average displacement of all valid pixels in the two areas (i.e., pixels in non-edge and non-occluded areas) can be obtained by statistics. The average displacement is the vector sum of the displacements of the pixels in the X and Y directions. The quality score of the captured data can then be determined based on the average displacement. If the average displacement exceeds the minimum resolvable displacement of the camera, the captured image will be blurred. Therefore, the larger the average displacement, the lower the quality score. The background area and the material area can be weighted according to the on-site working conditions to obtain the final quality score.

[0018] For example, the quality score is out of 10. Based on the pixel size of the camera sensor and the working conditions of the optical system, 2 pixels / frame is determined as the demarcation value for slightly blurred images, and 5 pixels / frame is determined as the demarcation value for obviously blurred images. The Lucas-Kanade (LK) or Farneback algorithm is used to calculate the optical flow of the background area and the material area respectively. After obtaining the pixel displacement of adjacent frames, the average displacement of the background area and the material area can be calculated. The average displacement of the background area and the material area is weighted according to the preset weight ratio. For example, the material area and the background area can be weighted. The weight ratios are 0.7 and 0.3 respectively, and the material area is the main weight to obtain the final average displacement, that is, the material area can be used as the main judgment object for evaluation, reducing the negative impact of other external factors and improving the reliability of the quality score. If the average displacement is <2 pixels / frame, it is judged that there is basically no blurring phenomenon and the score is 9-10 points. If the average displacement is between 2-5 pixels / frame, it is judged to be a slight blurring phenomenon and the score is 6-8 points. If the average displacement is >5 pixels / frame, it is judged to be an obvious blurring phenomenon and the score is 0-5 points. Finally, the quality score of the average displacement can be directly used as the quality score of the output shooting data.

[0019] When obtaining the point cloud overlap rate of the current frame and the previous frame, the ICP algorithm (Iterative Closest Point) can be used to calculate the point cloud overlap rate of the current frame and the previous frame. The point cloud overlap rate is an important indicator to measure the credibility of the alignment result. If the point cloud overlap rate is too low, it means that the lidar may have been subject to greater interference when collecting data, and the data credibility is low. For example, the quality score is 10 points. According to the working conditions, the point cloud overlap rate is 75% as the cutoff value for drastic changes, and 85% as the cutoff value for scene stability. If the point cloud overlap rate is <75%, it is judged that the environment changes drastically and the data credibility is low, and the score is 0-5 points. If the point cloud overlap rate is between 75%-85%, it is judged to be a slight change, and the score is 6-8 points. If the point cloud overlap rate is >85%, the scene is stable and the point cloud alignment effect is good, and the score is 9-10 points. Finally, the quality score of the point cloud overlap rate can be directly used as the quality score of the output point cloud data.

[0020] The volatility of weight data can be judged by selecting a variety of different indicators according to different working conditions, such as the variance, standard deviation, mean change rate, skewness, peak value, etc. of the weight data. By selecting appropriate indicators for calculation, the volatility of the weight data can be judged to obtain the final quality score of the weight data.

[0021] In step S3, the basic weight is calculated based on the dynamics of the material. The dynamic degree of the material can be obtained based on the average displacement and the point cloud overlap rate. For example, in a static scene where materials are piled up, the dynamic degree of the material is zero, and the basic weight is a pre-set default value. At this time, the weight sensor can monitor the impact signal first due to its sensitivity to vibration, and the set weight is the largest, the weight of the camera is second, and the weight of the lidar is the lowest. If in a dynamic scene of a transportation production line, the greater the movement speed of the material, the greater the dynamic degree of the material, so the basic weight will be dynamically adjusted. At this time, it is most critical to capture the motion state with a camera, and the basic weight of the lidar is second. Therefore, the weight ratio of the camera and lidar can be increased according to the dynamic degree of the material, and the weight ratio of the weight sensor can be reduced accordingly. The basic weight dynamically adjusted based on the average displacement and the point cloud overlap rate can obtain appropriate basic weights in different dynamic scenes, thereby being able to adapt to on-site working conditions, so that the final total quality score has a high degree of trust and reference.

[0022] The first threshold serves as the boundary value for whether the shooting data, point cloud data, and weight data are to be fused, and can be determined according to specific working conditions. When the total quality score is greater than or equal to the first threshold, it indicates that the current sensor data is trustworthy and subsequent fusion processing can be performed to obtain the required material information. If the total quality score is lower than the first threshold, it indicates that the current sensor data is untrustworthy and it is necessary to adjust the working parameters of the corresponding sensors based on the quality scores of each sensor to regain higher-quality sensor data.

[0023] In step S5, material information can be obtained selectively according to working condition requirements. For example, through fusion processing of sensor data, multiple key material information such as material volume, mass, position coordinates, shape contour, motion state, and stacking density can be finally obtained based on weights. The above key material information can be measured independently by at least two sensors to avoid system paralysis caused by failure of a single sensor. Different sensors perceive the same physical phenomenon from different dimensions, forming a complete chain of direct measurement-indirect estimation-physical constraint. Based on the fusion result, the source of the abnormality can be quickly located through the quality score and measurement value deviation of each sensor, that is, a highly reliable fusion result can be output, and the abnormality can be quickly located through deviation analysis. This multi-dimensional and redundant fusion mechanism significantly improves the robustness and reliability of abnormal working condition detection in the factory.

[0024] In addition, it should be noted that in step S1, the three sensors collect data synchronously to ensure that the data timestamps are consistent. The camera can output an image as shooting data, the lidar can output a three-dimensional point cloud as point cloud data, and the weight sensor can output a time-series weight signal as weight data. The detection areas of the camera and the lidar are the same or overlap to ensure the feasibility of data fusion processing. In some optional embodiments, after obtaining the shooting data, point cloud data, and weight data, all data can be preprocessed accordingly, such as denoising, filtering, calibration, etc., to ensure that the data quality model outputs an effective quality score. Specifically, image denoising can be performed on the shooting data to eliminate random noise (such as Gaussian noise and salt and pepper noise), enhance image clarity, and retain edge and texture details. Point cloud filtering can be performed on the point cloud data to remove outliers (such as floating objects and measurement errors), smooth surface noise, and improve the uniformity of point cloud density. The weight data can be notch filtered, wavelet denoising, and calibration to suppress periodic vibration noise, eliminate random high-frequency noise, and obtain weight data that highlights the true weight change trend.

[0025] If after multiple cycles of optimization in step S4, that is, after multiple adjustments of the working parameters, the total quality score still does not increase to reach the first threshold, it can be set not to loop back to step S2 after a preset number of cycles, but a notification can be issued for manual intervention to perform adjustments. The preset number of cycles can be determined according to the working conditions.

[0026] The data processing method of the present invention can improve the reliability of material information through a closed-loop process of multi-sensor data acquisition → quality assessment → dynamic adjustment → fusion output. It is based on multi-sensor collaboration and dynamic quality optimization. Specifically, it identifies low-quality data such as blurred images, sparse point clouds, and fluctuating weights through quality assessment, and optimizes the acquisition process by adjusting the sensor working parameters, reducing invalid data caused by improper sensor configuration or environmental interference, improving the quality of single-source data, and can adapt to dynamic scenarios such as material movement changes and weight fluctuations to avoid the problem of data distortion caused by sensors under fixed working parameters. Multi-sensor fusion is combined with weighted calculation of quality scores to make full use of high-reliability data, reduce the error of a single sensor, and improve the comprehensive accuracy of obtaining material information. Through automated quality assessment and parameter adjustment, the need for manual debugging of sensors is effectively reduced. It is suitable for scenarios that require real-time or long-term operation, and is particularly suitable for automated scenarios that require high-precision material detection, such as industrial automation, smart warehousing and other applications.

[0027] In some optional embodiments, in step S2, outputting the quality score of the captured data also includes: calculating the high-frequency energy proportion and gradient variance of the captured data, determining the corresponding quality scores based on the high-frequency energy proportion and gradient variance, respectively, and outputting the quality score of the captured data in combination with the quality scores of the average displacement, high-frequency energy proportion, and gradient variance.

[0028] The average displacement is used as the corresponding judgment indicator for the quality score of the shooting data. On this basis, the high-frequency energy ratio and gradient variance are added for auxiliary judgment. This can effectively improve the accuracy of judging whether the shot is blurry and improve the reliability of the quality score. The high-frequency energy ratio refers to the proportion of the energy of the high-frequency component in the total energy of the image. The larger the high-frequency energy ratio, the clearer the shooting details. The gradient variance reflects the combined effect of pixel value changes and gradient changes in the local area of ​​the image. The larger the gradient variance, the richer and clearer the details.

[0029] For example, if the quality score is 10 points, the high-frequency component ratio is set to 30% according to the working conditions, which is a fuzzy boundary value, and the gradient variance is set to 80, which is a fuzzy boundary value. The high-frequency energy ratio can be calculated by fast Fourier transform of the shooting data, and the Sobel operator (Sobel-Ferald The gradient map is calculated using the [operator] to obtain the gradient variance of the captured data. If the high-frequency energy ratio is greater than 35%, the details are clear and the score is 9-10. If the high-frequency energy ratio is between 30% and 35%, the details are basically clear and the score is 6-8. If the high-frequency energy ratio is less than 30%, the details are lost and blurred and the score is 0-5. Similarly, if the gradient variance is greater than 120, the details are rich and the score is 9-10. If the gradient variance is between 80-120, the details are relatively clear and the score is 6-8. If the gradient variance is less than 80, the details are blurred and the score is 0-5. Finally, the quality score of the captured data is obtained by combining the quality scores of mean displacement, high-frequency energy ratio, and gradient variance. The specific combination method can be selected according to actual conditions. For example, when there is a large amount of historically annotated data, a machine learning model can be established to input the quality scores of mean displacement, high-frequency energy ratio, and gradient variance to output the quality score of the captured data. Alternatively, a weight coefficient can be set according to the specific scenario for weighted calculation, or a fuzzy comprehensive evaluation method can be used based on professional experience.

[0030] In some optional embodiments, in step S2, outputting the quality score of the point cloud data further includes: calculating the point cloud density of the point cloud data, determining the corresponding quality score based on the point cloud density, and outputting the quality score of the point cloud data by combining the quality scores of the point cloud overlap rate and the point cloud density.

[0031] On the basis of judging the point cloud data quality by the point cloud overlap rate, the point cloud density is added for combined judgment, which makes up for the limitations of a single indicator and can improve the applicability to dynamic scenes, especially scenes where materials are in motion, and can more accurately evaluate the data quality. Furthermore, when performing weighted calculation on the two indicators, the weight of the quality score corresponding to the point cloud overlap rate can be increased for static scenes, and the weight of the quality score corresponding to the point cloud density can be increased for dynamic scenes, so that the final point cloud data quality score has better robustness and applicability.

[0032] For example, the quality score is 10 points, and the point cloud density is set to 500 points / m according to the working conditions. 2 It is the low-density boundary value, and the point cloud density is 1000 points / m 2 It is the boundary value of high density. If the point cloud density is greater than 1000 points / m 2 , the score is 9-10 points, if the point cloud density is 500-1000 points / m 2 If the point cloud density is less than 500 points / m, the score is 6-8 points. 2 , with a score of 0-5. Finally, the quality score of the point cloud data is obtained by combining the quality scores of the point cloud overlap rate and the point cloud density. Specifically, machine learning models, fuzzy comprehensive evaluation or weighted calculation methods can be used.

[0033] In some optional embodiments, in step S2, the volatility of the weight data is calculated to output a quality score of the weight data, including: using a sliding window to collect weight data, calculating the variance of the weight data in the sliding window, and obtaining the mean change rate of the weight data within a preset time, determining corresponding quality scores based on the variance and the mean change rate, and outputting the quality score of the weight data in combination with the quality scores of the variance and the mean change rate.

[0034] The sliding window can capture the local change characteristics in the weight data and is suitable for real-time or quasi-real-time data processing of time series data. The volatility of the weight data of materials in the factory is mainly affected by factors such as random noise (such as mechanical vibration) and drift error (such as zero offset). Since random noise will cause data points to fluctuate randomly around the mean, the larger the variance, the stronger the noise. Therefore, the variance is selected as the judgment indicator of random noise. The drift error will cause the mean to slowly shift over time. A high mean change rate indicates a significant trend deviation, and a low change rate indicates that the drift is negligible. Therefore, the mean change rate is selected as the judgment indicator of the drift error. Combining the variance and mean change rate can effectively judge the volatility of the material weight data, and it is easy to distinguish the source of the volatility, so that the sensor can be adjusted in a targeted manner in the future.

[0035] For example, the quality score is based on a 10-point scale, and the variance is set to 0.05kg according to the working conditions. 2is the boundary value of low noise, with a variance of 0.1kg 2 It is the boundary value of high noise, the mean change rate of 0.03kg / s is the boundary value of data stability, and the mean change rate of 0.05kg / s is the boundary value of data fluctuation abnormality. We can further use all the sampled data in the one-second sliding window to calculate the variance. If the variance is greater than 0.1kg 2 , indicating that the noise is too large, the score is 0-5 points, if the variance is between 0.05-0.1kg 2 If the noise is controllable, the score is 6-8 points. If the variance is <0.05kg 2 , indicating that the noise is extremely low, with a score of 9-10 points. Similarly, the mean change rate of the weight data within 30 consecutive seconds can be calculated. If the mean change rate is >0.05kg / s, it indicates that the data fluctuates abnormally, with a score of 0-5 points. If the mean change rate is between 0.03-0.05kg / s, it indicates that the data volatility is low, with a score of 6-8 points. If the mean change rate is <0.03kg / s, it indicates that the data is stable, with a score of 9-10 points. Finally, the quality score of the weight data is output by combining the quality scores of the variance and the mean change rate. Specifically, fusion methods such as machine learning models, fuzzy comprehensive evaluation, or weighted calculation can be used.

[0036] In some optional embodiments, in step S3, the basic weight is dynamically determined based on the average displacement and the point cloud overlap rate, including: obtaining the dynamic coefficient K of the material based on the average displacement and the point cloud overlap rate, K=a*d / D+b*(1-q), where a is the displacement contribution coefficient, b is the point cloud contribution coefficient, d is the average displacement, D is the maximum displacement, and q is the point cloud overlap rate; obtaining the basic weight Wp=W1+W0*m*K of the shooting data, obtaining the basic weight Wd=W2+W0*n*K of the point cloud data, and obtaining the basic weight Wz=1-Wp-Wd of the weight data, where W1 and W2 are the default weights of the shooting data and the point cloud data in the static scene, respectively, W0 is the maximum adjustment range, m is the adjustment coefficient of the shooting data, n is the adjustment coefficient of the point cloud data, and K is the dynamic coefficient.

[0037] The dynamic coefficient K is used to quantify the degree of material movement. Without manual intervention, the weights of multiple sensor quality scores are self-adjusted, which can adapt to various dynamic scenes in the factory, so that the total quality score can be trustworthy and referenceable under various working conditions. For example, if the average displacement obtained in step S2 is 2 pixels / frame, the point cloud overlap rate is 80%, and in the process of obtaining the average displacement, the maximum displacement D can be calibrated, assuming it is 10 pixels / frame, and the displacement contribution coefficient a is set to 0.7, the point cloud contribution coefficient b is set to 0.3, that is, the dynamic of the material The degree is mainly based on the shooting data, and then the dynamic coefficient K=0.7*0.2+0.3*0.2=0.2 can be obtained. The value range of the dynamic coefficient K can be set to 0~1, that is, a+b=1. The larger the value of K, the faster the material movement speed. K=0.2 indicates that the material movement speed is slightly lower, which belongs to a low-dynamic scene. The default weights W1 and W2 of the static scene are further set to 0.3 and 0.2 respectively. That is, the weight of the shooting data in the static scene is 0.3, and the weight of the point cloud data is 0.2. The default weight of the weight data is the largest, and its effect on The corresponding weight W3 is 0.5, followed by the shooting data. That is, in static scenes, the basic weights of shooting data, point cloud data, and weight data are 0.3, 0.2, and 0.5, respectively, which meets the needs of static scenes. The maximum adjustment range can be set to 0.2 according to the needs. The adjustment coefficient of shooting data is m=1.5, and the adjustment coefficient of point cloud data is n=1.2. Specifically, m can be set greater than n to meet the dynamic scene, increase the weight of shooting data, and thus meet the measurement needs. Then obtain the dynamic adjustment weight Wp=0.3+0.2*1.5 of the shooting data. *0.2=0.36, the dynamic adjustment weight of the point cloud data is obtained as Wd=0.2+0.2*1.2*0.2=0.248, and the dynamic adjustment weight of the weight data is obtained as Wz=0.392. That is, in low-dynamic scenes, the basic weights of the shooting data, point cloud data, and weight data are calculated to be 0.36, 0.248, and 0.392, respectively. This increases the weights of the shooting data and point cloud data in low-dynamic scenes, and adaptively reduces the weight of the weight data. Similarly, in high-dynamic scenes, the weights of the shooting data and point cloud data will be further increased.

[0038] In some optional embodiments, in step S4, based on the quality scores of the shooting data, point cloud data and weight data, the operating parameters of the camera, lidar and weight sensor are adjusted accordingly, including: if the quality score of the shooting data is lower than the second threshold, increasing the shutter speed and / or shooting frame rate of the camera; if the quality score of the point cloud data is lower than the third threshold, increasing the point cloud density and / or scanning frequency of the lidar; if the quality score of the weight data is lower than the fourth threshold, increasing the sampling frequency of the weight sensor and / or performing zero point calibration.

[0039] When the total quality score is less than the first threshold, the second threshold is the critical value for the camera to be optimized, the third threshold is the critical value for the lidar to be optimized, and the fourth threshold is the critical value for the weight sensor to be optimized. The second threshold, the third threshold and the fourth threshold can all be determined according to the specific working conditions. For example, they can all be set to 6 points. That is, when the total quality score is less than the first threshold, if the quality score of the shooting data, point cloud data or weight data is lower than 6 points, corresponding adjustments are required. The adjustment standard of the shooting data can also be increased based on the working conditions. For example, when the total quality score does not meet the conditions, the camera needs to be adjusted if the quality score of the shooting data is lower than 8 points.

[0040] The captured data is divided into a background area and a material area, and the average displacement of the two different areas is obtained respectively. In step S4, the working parameters of the camera can also be adjusted in a targeted manner. For example, when the average displacement of the material area is ≥5 pixels / frame and the average displacement of the background area is <2 pixels / frame, it means that the background of the captured data is clear and the material is blurred. This is likely to be caused by the material moving too fast. Increasing the shutter speed and / or shooting frame rate of the camera can effectively reduce the average displacement of the material area, capture clear material data, and thus improve the quality score of the captured data. If the average displacement of the material area and the background area are both greater than 5 pixels / frame, it means that the background and material of the captured data are blurred. At this time, while increasing the shutter speed and / or shooting frame rate of the camera, the installation stability of the camera should also be additionally calibrated to avoid camera shake or movement that increases the blurriness of the image.

[0041] The quality score of point cloud data includes the point cloud overlap rate. If the quality score of the point cloud overlap rate is too low, the scanning frequency of the lidar can be increased to collect more scanning frames per unit time, and dynamic scenes can be captured more densely, which can effectively improve the point cloud overlap rate to improve the quality of the point cloud data. If the quality score of the point cloud density is also included, then if the quality score of the point cloud density is too low, the point cloud density of the lidar can be directly improved to make the lidar more suitable for the current working conditions. The judgment threshold for the point cloud overlap rate and point cloud density quality scores being too low can also be determined according to the specific working conditions.

[0042] The quality score of weight data is mainly affected by random noise and drift error. If the random noise is too large as determined by the quality score of variance, the sampling frequency of the weight sensor should be adjusted accordingly to improve the sensor's ability to capture transient shocks, thereby providing a data basis for denoising. In combination with a pre-set filter, the noise can be effectively reduced. If the drift error is too large as determined by the quality score of the mean change rate, zero-point calibration can effectively improve the drift error, thereby specifically improving the quality score of the weight data.

[0043] In some optional embodiments, step S5 also includes: if the quality score of the shooting data is lower than the fifth threshold, increasing the shutter speed and / or shooting frame rate of the camera; if the quality score of the point cloud data is lower than the sixth threshold, increasing the point cloud density and / or scanning frequency of the lidar; if the quality score of the weight data is lower than the seventh threshold, increasing the sampling frequency of the weight sensor and / or performing zero point calibration.

[0044] When the total quality score is greater than or equal to the first threshold, the fifth threshold is the critical value for the camera to be optimized, the sixth threshold is the critical value for the lidar to be optimized, and the seventh threshold is the critical value for the weight sensor to be optimized. That is, when the total quality score is greater than or equal to the first threshold, after completing the data fusion processing and obtaining the material information, the working parameters of the corresponding sensor can be optimized for the case where the quality score of a certain sensor data is too low, so as to improve the subsequent collection quality of the corresponding data, thereby achieving high-standard information acquisition. The fifth threshold, the sixth threshold, and the seventh threshold can be reasonably determined according to the accuracy requirements of different sensors on site. The specific optimization process is the same as the optimization of the aforementioned step S4, and will not be repeated here.

[0045] In some optional embodiments, in step S5, the shooting data, point cloud data and weight data are fused based on the quality scores of the shooting data, point cloud data and weight data and in combination with the total quality score, including: if the total quality score is greater than the eighth threshold, the shooting data, point cloud data and weight data are fused based on the basic weight; if the total quality score is greater than or equal to the first threshold and less than or equal to the eighth threshold, the shooting data, point cloud data and weight data are fused based on the dynamic weight, and the dynamic weight is determined on the basis of the basic weight and based on the quality scores of the shooting data, point cloud data and weight data.

[0046] The eighth threshold serves as the boundary value for high-credible fusion of the total quality score and can be determined according to the specific working conditions. For example, if the total quality score is also based on a 10-point system, the eighth threshold can be set to 8 points and the first threshold can be set to 6 points. When the total quality score is greater than 8, it indicates that the quality scores of all sensors are stable, without significant abnormalities, and all have high credibility. When data fusion is performed, since the basic weight is reasonably set based on the current working conditions, the shooting data, point cloud data, and weight data can continue to use the basic weight calculated in step S3 to participate in the fusion to adapt to the current working conditions. When the total quality score is between 6 and 8 points, it indicates that the quality scores of all sensors are not high or the quality score of a certain sensor is low, but it is still trustworthy overall, and dynamic fusion can be performed. That is, based on the basic weight in step S3, if there is sensor data with a low quality score, its weight is reduced and assigned to other sensor data with a higher quality score to obtain the adjusted dynamic weight. When data fusion is performed, the shooting data, point cloud data, and weight data can participate in the fusion based on the dynamic weight.

[0047] In addition, the specific method of fusing multiple sensor data based on basic weights or dynamic weights can select a fusion method suitable for the current working conditions. For example, the basic weighted averaging method is suitable for low-speed static scenes, while the fusion method using Kalman filtering is suitable for dynamic scenes. Taking the basic weighted averaging method for material volume measurement as an example, it is assumed that the material is the fusion result of the camera (image size), lidar (point cloud volume), and weight sensor (mass inverse volume), and a highly reliable volume estimate is output. The specific steps are to obtain the standardized measurement values ​​of each sensor, including: obtaining the volume data collected by the camera through the image recognition algorithm, obtaining the volume data collected by the lidar by extracting the material point set through point cloud segmentation, converting the volume data collected by the weight sensor through the material density, and finally obtaining the final volume estimate based on the basic weight or dynamic weight.

[0048] The data quality model established by combining multiple implementation methods can have more comprehensive and reliable quality scoring indicators, such as Figure 2As shown, it is a processing flow chart of the data quality model, and the above-mentioned step S2 specifically includes the following steps: S201, establishing a data quality model; S202, separating the background area and the material area of ​​the shooting data, and performing optical flow calculation on the background area and the material area respectively to obtain the average displacement, calculate the high-frequency energy ratio and gradient variance of the shooting data, and determine the corresponding quality scores based on the average displacement, high-frequency energy ratio and gradient variance to output the quality score of the shooting data; S203, obtaining the point cloud overlap rate of the point cloud data in the current frame and the previous frame, calculating the point cloud density of the point cloud data, and determining the corresponding quality scores based on the point cloud overlap rate and the point cloud density to output the quality score of the point cloud data; S204, using a sliding window to collect weight data, calculating the variance of the weight data in the sliding window, and obtaining the mean change rate of the weight data within a preset time, and determining the corresponding quality scores based on the variance and the mean change rate to output the quality score of the weight data.

[0049] The following describes a data processing device provided in an embodiment of the present application. The device described below and the method described above can refer to each other. Based on the above embodiment, Figure 3 This is a schematic diagram of the structure of a data processing device provided by this embodiment. Figure 3 As shown, the device includes: an acquisition module 10, which is used to detect materials using a camera, a laser radar and a weight sensor, and respectively obtain the shooting data, point cloud data and weight data of the materials; a modeling module 20, which is used to establish a data quality model, which is used to separate the background area and the material area of ​​the shooting data, and perform optical flow calculation on the background area and the material area respectively to obtain the average displacement to output the quality score of the shooting data; obtain the point cloud overlap rate of the point cloud data in the current frame and the previous frame to output the quality score of the point cloud data; calculate the volatility of the weight data to output the quality score of the weight data; a judgment module 30, which is based on the quality scores of the shooting data, the point cloud data and the weight data, and performs weighted calculation by basic weights to obtain To the total quality score, the basic weight is dynamically determined based on the average displacement and the point cloud overlap rate to determine whether the total quality score is less than the first threshold. If so, it enters the adjustment module, otherwise it enters the fusion module; the adjustment module 40, based on the quality scores of the shooting data, point cloud data and weight data, adjusts the working parameters of the camera, lidar and weight sensor accordingly, and obtains the updated shooting data, point cloud data and weight data, and returns to the modeling module 20 to update the quality scores of the shooting data, point cloud data and weight data; the fusion module 50, based on the quality scores of the shooting data, point cloud data and weight data, fuses the shooting data, point cloud data and weight data in combination with the total quality score to obtain material information.

[0050] Based on the above method embodiment, please see Figure 4 , Figure 4Schematic diagram of the structure of an electronic device provided in an embodiment of the present application. An electronic device 300 provided in an embodiment of the present application includes: a processor 301 and a memory 302, wherein the memory 302 stores machine-readable instructions executable by the processor 301, and when the machine-readable instructions are executed by the processor 301, the method described above is performed; the electronic device may be a physical device.

[0051] Based on the above method embodiments, a computer-readable storage medium is further provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method in any of the above embodiments are implemented.

[0052] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0053] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0054] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0055] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0056] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0057] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0058] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information, such as computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media such as modulated data signals and carrier waves.

[0059] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0060] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0061] The above are merely embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. A data processing method, characterized in that: include: S1, uses cameras, lidar and weight sensors to detect materials and obtain the material's shooting data, point cloud data and weight data respectively; S2: Build a data quality model to separate the background and material areas of the captured data. Perform optical flow calculations on the background and material areas to obtain the average displacement, which is used to output a quality score for the captured data. Obtain the point cloud overlap ratio between the current and previous frames to output a quality score for the point cloud data. Calculate the volatility of the weight data to output a quality score for the weight data. S3, based on the quality scores of the shooting data, point cloud data, and weight data, a total quality score is obtained by weighted calculation using a basic weight, where the basic weight is dynamically determined based on the average displacement and the point cloud overlap rate. It is determined whether the total quality score is less than a first threshold. If so, step S4 is performed; otherwise, step S5 is performed. S4, based on the quality scores of the captured data, point cloud data, and weight data, correspondingly adjust the operating parameters of the camera, lidar, and weight sensor, obtain updated captured data, point cloud data, and weight data, and return to step S2 to update the quality scores of the captured data, point cloud data, and weight data; S5, based on the quality scores of the shooting data, point cloud data and weight data, and combined with the total quality score, the shooting data, point cloud data and weight data are integrated and processed to obtain material information.

2. The data processing method according to claim 1, characterized in that: In step S2, the quality score of the captured data is output, which also includes: calculating the high-frequency energy ratio and gradient variance of the captured data, determining the corresponding quality scores based on the high-frequency energy ratio and gradient variance, and outputting the quality score of the captured data by combining the quality scores of the average displacement, high-frequency energy ratio, and gradient variance.

3. The data processing method according to claim 1, characterized in that: In step S2, outputting the quality score of the point cloud data further includes: calculating the point cloud density of the point cloud data, determining the corresponding quality score based on the point cloud density, and outputting the quality score of the point cloud data in combination with the quality score of the point cloud overlap rate and the point cloud density.

4. The data processing method according to claim 1, characterized in that: In step S2, the volatility of the weight data is calculated to output a quality score of the weight data, including: using a sliding window to collect weight data, calculating the variance of the weight data in the sliding window, and obtaining the mean change rate of the weight data within a preset time, determining the corresponding quality scores based on the variance and the mean change rate, and outputting the quality score of the weight data in combination with the quality scores of the variance and the mean change rate.

5. The data processing method according to claim 1, characterized in that: In step S3, the basic weight is dynamically determined based on the average displacement and the point cloud overlap rate, including: The dynamic coefficient K of the material is obtained based on the average displacement and point cloud overlap rate, K=a*d / D+b*(1-q), where a is the displacement contribution coefficient, b is the point cloud contribution coefficient, d is the average displacement, D is the maximum displacement, and q is the point cloud overlap rate; Get the basic weight of the shooting data Wp=W1+W0*m*K, get the basic weight of the point cloud data Wd=W2+W0*n*K, and get the basic weight of the weight data Wz=1-Wp-Wd, where W1 and W2 are the default weights of the shooting data and point cloud data in the static scene, respectively. W0 is the maximum adjustment range, m is the adjustment coefficient of the shooting data, n is the adjustment coefficient of the point cloud data, and K is the dynamic coefficient.

6. The data processing method according to any one of claims 1 to 5, characterized in that: In step S4, based on the quality scores of the shooting data, point cloud data, and weight data, the operating parameters of the camera, lidar, and weight sensor are adjusted accordingly, including: If the quality score of the captured data is lower than a second threshold, increasing the shutter speed and / or the shooting frame rate of the camera; If the quality score of the point cloud data is lower than a third threshold, increasing the point cloud density and / or scanning frequency of the lidar; If the quality score of the weight data is lower than a fourth threshold, the sampling frequency of the weight sensor is increased and / or a zero point calibration is performed.

7. The data processing method according to claim 6, characterized in that: Step S5 also includes: if the quality score of the shooting data is lower than the fifth threshold, increasing the shutter speed and / or shooting frame rate of the camera; if the quality score of the point cloud data is lower than the sixth threshold, increasing the point cloud density and / or scanning frequency of the lidar; if the quality score of the weight data is lower than the seventh threshold, increasing the sampling frequency of the weight sensor and / or performing zero point calibration.

8. The data processing method according to claim 1, characterized in that: In step S5, based on the quality scores of the photographic data, point cloud data, and weight data, and in combination with the total quality score, the photographic data, point cloud data, and weight data are fused, including: If the total quality score is greater than the eighth threshold, the shooting data, point cloud data, and weight data are fused based on the basic weights; If the total quality score is greater than or equal to the first threshold and less than or equal to the eighth threshold, the shooting data, point cloud data and weight data are fused based on the dynamic weight. The dynamic weight is determined based on the basic weight and the quality score of the shooting data, point cloud data and weight data.

9. A data processing device, characterized in that: include: The acquisition module is used to detect materials using cameras, lidar and weight sensors to obtain the material's shooting data, point cloud data and weight data respectively; The modeling module is used to establish a data quality model to separate the background area and material area of ​​the captured data, and perform optical flow calculation on the background area and material area to obtain the average displacement to output the quality score of the captured data; obtain the point cloud overlap rate of the current frame and the previous frame to output the quality score of the point cloud data; Calculate the volatility of the weight data to output a quality score of the weight data; The judgment module calculates the total quality score based on the quality scores of the shooting data, point cloud data, and weight data through a weighted calculation using basic weights. The basic weights are dynamically determined based on the average displacement and the point cloud overlap rate. The module determines whether the total quality score is less than a first threshold. If so, the module enters the adjustment module; otherwise, the module enters the fusion module. The adjustment module adjusts the working parameters of the camera, lidar, and weight sensor based on the quality scores of the shooting data, point cloud data, and weight data, obtains the updated shooting data, point cloud data, and weight data, and returns to the modeling module to update the quality scores of the shooting data, point cloud data, and weight data; The fusion module is based on the quality scores of the shooting data, point cloud data and weight data, and combines the shooting data, point cloud data and weight data in combination with the total quality score to obtain material information.

10. An electronic device, characterized in that: The invention comprises a memory and a processor coupled to the memory, wherein the processor is configured to execute the method according to any one of claims 1 to 7 based on instructions stored in the memory.

Images