Color image data processing method based on distributed photoelectric system

Abstract

The invention discloses a color image data processing method based on a distributed photoelectric system, and relates to the technical field of image data processing. According to the invention, by collecting original image data, effective integration of an infrared image and a visible light image is realized; preprocessing the original data, matching the image data through timestamps, and constructing an original image data set; performing feature point extraction and matching on each element to obtain a final matching point pair; based on the matching points, utilizing a deep information fusion technology to generate initial fusion image data, and optimizing the initial fusion image data to obtain color image data; a quality evaluation mechanism is set, a color image quality score is obtained through comparison with standard image data, prompt information is output according to a score result, and subsequent processing is further guided.

Description

Technical Field

[0001] The present invention relates to the technical field of image data processing, and in particular to a color image data processing method based on a distributed optoelectronic system. Background Art

[0002] With the rapid development of optoelectronic technology, infrared sensors and visible light sensors have been widely used in the field of image acquisition. Infrared sensors can capture the thermal radiation information of objects and are suitable for imaging at night or in low-light environments; while visible light sensors can capture visual information such as color and texture of objects, providing rich image details. However, the image information obtained by a single sensor is often limited and difficult to meet the application requirements in complex scenes. Therefore, a color image data processing method based on a distributed optoelectronic system has emerged, aiming to generate images with more information and rich colors by combining the advantages of infrared and visible light sensors.

[0003] The existing color image data processing methods of distributed optoelectronic systems usually include multiple steps such as image acquisition, preprocessing, image registration, image fusion and post-processing. Although the existing color image data processing methods of distributed optoelectronic systems have achieved the processing of color image data to a certain extent, they still have the following shortcomings: the existing image fusion strategies are mostly based on pixel-level or feature-level fusion, lack the understanding and utilization of deep image information, and are difficult to effectively fuse the temperature information of infrared images with the color information of visible light images, resulting in color distortion or information loss of the fused image; in addition, in a distributed system, the resource utilization of each computing node may be unbalanced, resulting in some nodes being overloaded while some nodes are idle, affecting the overall processing efficiency. Summary of the invention

[0004] The object of the present invention is to provide a color image data processing method based on a distributed optoelectronic system to solve the problems raised in the above background technology.

[0005] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0006] A color image data processing method based on a distributed optoelectronic system, the method comprising the following steps:

[0007] Step S100. Using the infrared sensor and the visible light sensor in the distributed optoelectronic system to simultaneously collect raw image data, and the raw image data includes infrared image data and visible light image data; preprocessing the raw image data, and matching the infrared image data and the visible light image data according to the timestamp, so as to form a raw image data set;

[0008] Step S200. According to the original image data set, for each element in the original image data set, feature point extraction and matching of the infrared image data and the visible light image data are performed, and the matching results of the feature points are further analyzed to obtain the final matching point pair;

[0009] Step S300. Based on the matching point pairs, perform deep information fusion on the corresponding infrared image data and visible light image data to obtain initial fused image data; perform corresponding image optimization on the initial fused image data, and obtain color image data according to the image optimization result;

[0010] Step S400. Compare the color image data with the corresponding standard image data, evaluate the quality of the color image data according to the comparison result, and obtain a corresponding quality score; output prompt information according to the size relationship between the quality score and the corresponding quality score threshold, and perform corresponding processing based on the prompt information.

[0011] Step S100 includes:

[0012] S101. In the distributed optoelectronic system, a timestamp server is set to achieve time synchronization between the infrared sensor and the visible light sensor in the system. By initializing the timestamp server, the time of the infrared sensor and the visible light sensor in the distributed optoelectronic system is kept consistent; before the infrared sensor and the visible light sensor collect raw image data, the timestamp server is requested and recorded locally; at the same time, the infrared sensor and the visible light sensor are started to collect raw image data, and the infrared sensor and the visible light sensor store the collected raw image data and their corresponding timestamps accordingly;

[0013] S102. Preprocess the collected raw image data, including noise removal, format unification, etc.; respectively obtain the timestamp set T_IR of the infrared image data and the timestamp set T_VIS of the visible light image data, and T_IR={t_IR1, t_IR2, ..., t_IRn}, T_VIS={t_VIS1, t_VIS2, ..., t_VISm}, where t_IR1 represents the first element in the timestamp set of the infrared image data, t_IR2 represents the second element in the timestamp set of the infrared image data, and so on, t_IRn represents The nth element in the timestamp set of the infrared image data; similarly, t_VIS1 represents the first element in the timestamp set of the visible light image data, t_VIS2 represents the second element in the timestamp set of the visible light image data, and t_VISm represents the mth element in the timestamp set of the visible light image data, where n and m represent the timestamp numbers of the infrared sensor and the visible light sensor, respectively; according to the timestamp set T_IR of the infrared image data and the timestamp set T_VIS of the visible light image data, the infrared image data and the visible light image data are matched to form the original image data set.

[0014] According to the timestamp set T_IR of the infrared image data and the timestamp set T_VIS of the visible light image data, the infrared image data and the visible light image data are matched to form an original image data set, which includes:

[0015] The objective function Z is set to minimize the time difference between the infrared image timestamp and the visible light image timestamp, and the objective function expression is:

[0016] Z=Σi∈[1,n],Σj∈[1,m],|t_IRi-t_VISj|·xij;

[0017] Where xij is a binary variable, indicating whether the i-th timestamp of the infrared image data matches the j-th timestamp of the visible light image data, and when the i-th timestamp of the infrared image data matches the j-th timestamp of the visible light image data, xij=1, otherwise, xij=0; for each infrared image data, a matching condition is defined so that each infrared image can only be matched with one visible light image, and the constraint condition is: Σj∈[1,m],xij≤1, For each visible light image data, a matching condition is defined so that each visible light image can only match one infrared image at most. The constraint condition is: Σi∈[1,n],xij≤1, According to the objective function and constraints, the optimization algorithm is used to solve the objective function Z to obtain the timestamp matching result xij, and the corresponding relationship between the successfully matched infrared image and the visible light image is extracted according to the matching result to form the original image data set.

[0018] Step S200 includes:

[0019] S201. According to the original image data set, for each element in the original image data set, a feature point detection algorithm is used to extract feature points from the infrared image data and the visible light image data corresponding to the element in the original image data set, and a feature point set P_IR and a feature point set P_VIS are obtained respectively, and P_IR={p_IR1, p_IR2, ..., p_IRu}, P_VIS={p_VIS1, p_VIS2, ..., p_VISv}, wherein p_IR1 represents the first feature point of the infrared image data, p_IR2 represents the second feature point of the infrared image data, and so on, p_IRu represents the uth feature point of the infrared image data; similarly, p_VIS1 represents the first feature point of the visible light image data, and p_VIS2 represents the uth feature point of the visible light image data. 2 feature points, p_VISv represents the vth feature point of visible light image data, u and v represent the number of feature points of infrared image data and visible light image data respectively; among them, the feature points of image data refer to some key points with significance, recognizability and stability in the image; for significance: for example, corner points, edge points and areas with rich texture are usually considered to be significant feature points; for recognizability: for example, SIFT (Scale Invariant Feature Transform) and SURF (Speeded Up Robust Features) algorithms generate descriptors by calculating the image information around the feature points, so that these features can be matched under different viewing angles and lighting conditions; for stability: the feature points should remain stable under transformations such as scaling, rotation and flipping of the image. This stability ensures that the same feature points can still be found in different scenes or conditions.

[0020] S202. For each feature point in the feature point set P_IR and the feature point set P_VIS, the corresponding feature descriptor is calculated, which is expressed as: d_IRx = F(p_IRx) and d_VISy = F(p_VISy), where x ranges from 1 to u, y ranges from 1 to v, and F represents the feature descriptor algorithm used, such as SIF and SURF; the feature descriptors corresponding to the feature point set P_IR and the feature point set P_VIS are summarized to form matrices D_IR and D_VIS respectively; each feature descriptor in the matrix D_IR is traversed, and the distance dist(d_IRx, d_VISy) between the feature descriptors and all the feature descriptors in the matrix D_VIS is calculated, and the matching pairs with the smallest distance are selected to form a matching result set M; the matching result set M is evaluated for slope consistency, and the final matching point pairs are obtained according to the evaluation results.

[0021] The matching result set M is evaluated for slope consistency. The specific evaluation process is as follows:

[0022] Extract all matching point pairs from the matching result set M, expressed as: M(x, y) = (d_IRx, d_VISy); calculate the corresponding slopes for any two matching point pairs, k_IR and k_VIS respectively; if |k_IR-k_VIS| < K0, the slope consistency index R = 1, otherwise, the slope consistency index R = 0, where K0 represents the slope threshold; combine the slope consistency index R with the distance dist(d_IRx, d_VISy) to obtain the comprehensive matching score S(d_IRx, d_VISy), and

[0023] S(d_IRx,d_VISy)={1 / [dist(d_IRx,d_VISy)+1]}×R,

[0024] The matching result set M' is updated according to the comprehensive matching score, and matching pairs with comprehensive matching scores higher than the threshold S0 are selected, so as to output the final matching point pair set M_final = M'.

[0025] Step S300 includes:

[0026] S301. Using matching point pairs, the corresponding pixel information in the infrared image and the visible light image is combined, and the pixel value of the initial fused image is defined as:

[0027] I_fused(a,b)=α·I_VIS(a,b)+(1-α)·I_IR(a,b),

[0028] Where I_fused(a,b) represents the value of the fused image at the pixel (a,b), I_VIS(a,b) represents the value of the visible light image at the pixel (a,b), I_IR(a,b) represents the value of the infrared image at the pixel (a,b), and α represents the weight coefficient, which ranges from [0,1];

[0029] S302. Standardize the temperature information of the infrared image, and assume that the temperature range of the infrared image is [T_min, T_max]. The specific standardization calculation formula is:

[0030] T_norm(a,b)=[T_IR(a,b)-T_min] / (T_max-T_min),

[0031] Wherein T_IR(a,b) represents the temperature value corresponding to the pixel point (a,b) of the infrared image, T_min represents the minimum temperature value of the corresponding infrared image, and T_max represents the maximum temperature value of the corresponding infrared image; according to the temperature information and visible light information in the matching point, and based on the fusion strategy of weight adjustment, the optimized fused image data is generated, and the fusion strategy based on weight adjustment is: when the temperature information of the standardized infrared image is higher than the threshold T0, the influence of the infrared information in the fused image is enhanced, and the corresponding expression is: W(a,b)=h·T_norm(a,b); otherwise, the visible light information is enhanced, and the corresponding expression is: W(a,b)=1-h; wherein W(a,b) represents the weight of the pixel point (a,b) in the optimized fused image, and h represents an adjustable parameter;

[0032] S303. Generate the final fused image by weighted averaging the fusion results of all pixels, and the corresponding calculation formula is:

[0033] I_final(a,b)=W(a,b)·I_VIS(a,b)+[1-W(a,b)]·T_norm(a,b), define an enhancement function E to optimize the final fused image, and I_optimized=E(I_final), so that the optimized fused image I_optimized is output as the final color image data I_color.

[0034] Step S400 includes:

[0035] S401. Obtain the color image data to be evaluated and the standard image data for comparison, scale the color image and the standard image to the same resolution, and calculate the difference image Y by pixel-by-pixel comparison, and the calculation formula is:

[0036] Y(a,b)=|I_color(a,b)-I_standard(a,b)|,

[0037] Where I_color(a,b) represents the pixel point (a,b) of the color image data, and I_standard(a,b) represents the pixel point (a,b) of the standard image data;

[0038] S402. Calculate the mean square error MSE, and the calculation formula of the mean square error MSE is:

[0039] MSE=(1 / N)Σa∈[1,W],Σb∈[1,H]Y(a,b)^2;

[0040] Wherein N=W×H is the total number of pixels of the image, and W and H are the width and height of the image respectively; the peak signal-to-noise ratio PSNR is calculated in combination with the mean square error MSE, and the calculation formula of the peak signal-to-noise ratio PSNR is: PSNR=10·log10(MAX^2 / MSE), wherein MAX represents the maximum value of the image pixel; according to the quality score threshold P, if PSNR≥P, the output image quality is good, if P>PSNR≥P-10, the output image quality is general, and if PSNR<P-10, the output image quality is poor; images with good image quality are saved, images with general image quality are output to relevant personnel for further processing by relevant personnel, and images with poor image quality are output with warning information to relevant personnel, and the relevant personnel reacquire the image.

[0041] The value of the quality score threshold P needs to be adaptively adjusted, including:

[0042] An initial quality score threshold P0 is set, and the relationship between the PSNR values ​​of several past evaluations and P0 is recorded, and the corresponding image quality rating is obtained by referring to the comparison result of PSNR and P in S402, and the image quality rating includes good, general, and poor; the mean μ and standard deviation σ of PSNR are calculated based on the results of several evaluations, and the threshold P is dynamically adjusted based on the mean and standard deviation of PSNR, and P_new=μ+c·σ, where c is the adjustment coefficient, c=1 means that the threshold is based on the mean and a standard deviation, c>1 means increasing the threshold, which is suitable for high-quality image requirements, and c<1 means reducing the threshold, which is suitable for low-quality image requirements;

[0043] According to the adaptively adjusted threshold value P_new, the mapping relationship of the image quality rating is redefined: if PSNR≥P_new, the output image quality is good; if P_new>PSNR≥P_new-10, the output image quality is average; if PSNR<P_new-10, the output image quality is poor.

[0044] The above method is applied to a distributed photovoltaic system, and the optimized load distribution between nodes in the distributed photovoltaic system includes the following contents:

[0045] Every selected time period, the resource utilization of each node in the system is collected, and the resource utilization includes CPU, memory and network bandwidth, etc.; based on the collected resource utilization, the load index L of each node is calculated, and the specific calculation formula is: L_q=U_q / G_q, where L_q represents the load index of the qth node, U_q represents the resource utilization of the qth node, and G_q represents the total resource available of the qth node; the resource utilization of each node in the system is obtained and the average value is calculated, and the calculated average value is used as the target load value L0, and the number of tasks that need to be migrated is determined according to the difference D_q between the target load and the current load, and D_q=L_q-L0; for nodes with D_q>0, the number of tasks that need to be migrated WORK_move=f(D_q)·WORK_total, where f(D_q) represents the migration ratio function calculated according to the difference D_q, and WORK_total represents the total number of tasks to be allocated.

[0046] Compared with the prior art, the beneficial effects of the present invention are as follows: the present invention realizes accurate time synchronization of infrared images and visible light images by introducing a timestamp server, thereby ensuring the consistency and accuracy of collected data; through a deep information fusion strategy, the temperature information of the infrared image and the color information of the visible light image are effectively combined, thereby overcoming the color distortion and information loss problems existing in the prior art and improving the overall quality of the fused image; the feature point extraction and matching method is adopted, combined with the slope consistency evaluation, to ensure the reliability of the matching result, this method can more effectively capture the spatial relationship and features between images, and reduce the degradation of the fusion effect caused by mismatching; in the image fusion process, the present invention dynamically adjusts the weight of the fused image according to the temperature information of the infrared image, enhances the influence of the infrared information of the high temperature area in the final image, thereby making the fused color image more realistic and recognizable; in view of the resource utilization of each computing node in the distributed optoelectronic system, an optimized load distribution strategy is implemented, by real-time monitoring of the resource utilization of each node, and performing task migration according to the average load, effectively reducing the overload phenomenon of some nodes, and improving the overall processing efficiency and resource utilization of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0048] Figure 1It is a schematic diagram of the steps of a color image data processing method based on a distributed optoelectronic system of the present invention. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] See also Figure 1 , the present invention provides a technical solution:

[0051] A color image data processing method based on a distributed optoelectronic system, the method comprising the following steps:

[0052] Step S100. Using the infrared sensor and the visible light sensor in the distributed optoelectronic system to simultaneously collect raw image data, and the raw image data includes infrared image data and visible light image data; preprocessing the raw image data, and matching the infrared image data and the visible light image data according to the timestamp, so as to form a raw image data set;

[0053] Step S200. According to the original image data set, for each element in the original image data set, feature point extraction and matching of the infrared image data and the visible light image data are performed, and the matching results of the feature points are further analyzed to obtain the final matching point pair;

[0054] Step S300. Based on the matching point pairs, perform deep information fusion on the corresponding infrared image data and visible light image data to obtain initial fused image data; perform corresponding image optimization on the initial fused image data, and obtain color image data according to the image optimization result;

[0055] Step S400. Compare the color image data with the corresponding standard image data, evaluate the quality of the color image data according to the comparison result, and obtain a corresponding quality score; output prompt information according to the size relationship between the quality score and the corresponding quality score threshold, and perform corresponding processing based on the prompt information.

[0056] Step S100 includes:

[0057] S101. In the distributed optoelectronic system, a timestamp server is set to achieve time synchronization between the infrared sensor and the visible light sensor in the system. By initializing the timestamp server, the time of the infrared sensor and the visible light sensor in the distributed optoelectronic system is kept consistent; before the infrared sensor and the visible light sensor collect raw image data, the timestamp server is requested and recorded locally; at the same time, the infrared sensor and the visible light sensor are started to collect raw image data, and the infrared sensor and the visible light sensor store the collected raw image data and their corresponding timestamps accordingly;

[0058] S102. Preprocess the collected raw image data, including noise removal, format unification, etc.; respectively obtain the timestamp set T_IR of the infrared image data and the timestamp set T_VIS of the visible light image data, and T_IR={t_IR1, t_IR2, ..., t_IRn}, T_VIS={t_VIS1, t_VIS2, ..., t_VISm}, where t_IR1 represents the first element in the timestamp set of the infrared image data, t_IR2 represents the second element in the timestamp set of the infrared image data, and so on, t_IRn represents The nth element in the timestamp set of the infrared image data; similarly, t_VIS1 represents the first element in the timestamp set of the visible light image data, t_VIS2 represents the second element in the timestamp set of the visible light image data, and t_VISm represents the mth element in the timestamp set of the visible light image data, where n and m represent the timestamp numbers of the infrared sensor and the visible light sensor, respectively; according to the timestamp set T_IR of the infrared image data and the timestamp set T_VIS of the visible light image data, the infrared image data and the visible light image data are matched to form the original image data set.

[0059] According to the timestamp set T_IR of the infrared image data and the timestamp set T_VIS of the visible light image data, the infrared image data and the visible light image data are matched to form an original image data set, which includes:

[0060] The objective function Z is set to minimize the time difference between the infrared image timestamp and the visible light image timestamp, and the objective function expression is:

[0061] Z=Σi∈[1,n],Σj∈[1,m],|t_IRi-t_VISj|·xij;

[0062] Where xij is a binary variable, indicating whether the i-th timestamp of the infrared image data matches the j-th timestamp of the visible light image data, and when the i-th timestamp of the infrared image data matches the j-th timestamp of the visible light image data, xij=1, otherwise, xij=0; for each infrared image data, a matching condition is defined so that each infrared image can only be matched with one visible light image, and the constraint condition is: Σj∈[1,m],xij≤1, For each visible light image data, a matching condition is defined so that each visible light image can only match one infrared image at most. The constraint condition is: Σi∈[1,n],xij≤1, According to the objective function and constraints, the optimization algorithm is used to solve the objective function Z to obtain the timestamp matching result xij, and the corresponding relationship between the successfully matched infrared image and the visible light image is extracted according to the matching result to form the original image data set.

[0063] In this embodiment, the timestamp matching process of the infrared sensor and the visible light sensor in the distributed optoelectronic system is as follows:

[0064] Initialize the timestamp server: Assume that the time for initializing the timestamp server is T_start = 2024-09-29 14:00:00;

[0065] Raw image data and timestamp collected by the infrared sensor:

[0066] t_IR1:2024-09-2914:00:01,

[0067] t_IR2:2024-09-2914:00:03,

[0068] t_IR3:2024-09-2914:00:05;

[0069] Raw image data and timestamp collected by the visible light sensor:

[0070] t_VIS1:2024-09-2914:00:02,

[0071] t_VIS2: 2024-09-29 14:00:04,

[0072] t_VIS3:2024-09-2914:00:06;

[0073] Infrared image timestamp set: T_IR = {t_IR1, t_IR2, t_IR3} = {2024-09-2914:00:01, 2024-09-2914:00:03, 2024-09-2914:00:05};

[0074] Visible light image timestamp set: T_VIS = {t_VIS1, t_VIS2, t_VIS3} = {2024-09-2914:00:02, 2024-09-2914:00:04, 2024-09-2914:00:06};

[0075] Define the objective function Z, where Z = Σi∈[1,3],Σj∈[1,3],|t_IRi-t_VISj|·xij;

[0076] Time difference calculation:

[0077] |t_IR1-t_VIS1|=|2024-09-2914:00:01-2024-09-2914:00:02|=1 (s);

[0078] |t_IR1-t_VIS2|=|2024-09-2914:00:01-2024-09-2914:00:04|=3 (s);

[0079] |t_IR1-t_VIS3|=|2024-09-2914:00:01-2024-09-2914:00:06|=5 (s);

[0080] |t_IR2-t_VIS1|=|2024-09-2914:00:03-2024-09-2914:00:02|=1 (s);

[0081] |t_IR2-t_VIS2|=|2024-09-2914:00:03-2024-09-2914:00:04|=1 (s);

[0082] |t_IR2-t_VIS3|=|2024-09-2914:00:03-2024-09-2914:00:06|=3 (s);

[0083] |t_IR3-t_VIS1|=|2024-09-2914:00:05-2024-09-2914:00:02|=3 (s);

[0084] |t_IR3-t_VIS2|=|2024-09-2914:00:05-2024-09-2914:00:04|=1 (s);

[0085] |t_IR3-t_VIS3|=|2024-09-2914:00:05-2024-09-2914:00:06|=1 (s);

[0086] The total time difference can be minimized using the Hungarian Algorithm or other optimal allocation methods. This algorithm can find the best match for each infrared image and the visible light image;

[0087] After matching processing, ideally the following matching results can be obtained:

[0088] t_IR1(2024-09-2914:00:01) matches t_VIS1(2024-09-2914:00:02);

[0089] t_IR2(2024-09-2914:00:03) matches t_VIS2(2024-09-2914:00:04);

[0090] t_IR3(2024-09-2914:00:05) matches t_VIS3(2024-09-2914:00:06);

[0091] Total time difference:

[0092] Under this matching, the total time difference Z is:

[0093] For t_IR1 and t_VIS1: (|1|=1),

[0094] For t_IR2 and t_VIS2: (|1|=1),

[0095] For t_IR3 and t_VIS3: (|1|=1);

[0096] Therefore, the total time difference is (Z=1+1+1=3) seconds.

[0097] The final timestamp matching results are as follows:

[0098] t_IR1 matches t_VIS1, t_IR2 matches t_VIS2, and t_IR3 matches t_VIS3.

[0099] Step S200 includes:

[0100] S201. According to the original image data set, for each element in the original image data set, a feature point detection algorithm is used to extract feature points from the infrared image data and the visible light image data corresponding to the element in the original image data set, and a feature point set P_IR and a feature point set P_VIS are obtained respectively, and P_IR={p_IR1, p_IR2, ..., p_IRu}, P_VIS={p_VIS1, p_VIS2, ..., p_VISv}, wherein p_IR1 represents the first feature point of the infrared image data, p_IR2 represents the second feature point of the infrared image data, and so on, p_IRu represents the uth feature point of the infrared image data; similarly, p_VIS1 represents the first feature point of the visible light image data, and p_VIS2 represents the uth feature point of the visible light image data. 2 feature points, p_VISv represents the vth feature point of visible light image data, u and v represent the number of feature points of infrared image data and visible light image data respectively; among them, the feature points of image data refer to some key points with significance, recognizability and stability in the image; for significance: for example, corner points, edge points and areas with rich texture are usually considered to be significant feature points; for recognizability: for example, SIFT (Scale Invariant Feature Transform) and SURF (Speeded Up Robust Features) algorithms generate descriptors by calculating the image information around the feature points, so that these features can be matched under different viewing angles and lighting conditions; for stability: the feature points should remain stable under transformations such as scaling, rotation and flipping of the image. This stability ensures that the same feature points can still be found in different scenes or conditions.

[0101] S202. For each feature point in the feature point set P_IR and the feature point set P_VIS, the corresponding feature descriptor is calculated, which is expressed as: d_IRx = F(p_IRx) and d_VISy = F(p_VISy), where x ranges from 1 to u, y ranges from 1 to v, and F represents the feature descriptor algorithm used, such as SIF and SURF; the feature descriptors corresponding to the feature point set P_IR and the feature point set P_VIS are summarized to form matrices D_IR and D_VIS respectively; each feature descriptor in the matrix D_IR is traversed, and the distance dist(d_IRx, d_VISy) between the feature descriptors and all the feature descriptors in the matrix D_VIS is calculated, and the matching pairs with the smallest distance are selected to form a matching result set M; the matching result set M is evaluated for slope consistency, and the final matching point pairs are obtained according to the evaluation results.

[0102] The matching result set M is evaluated for slope consistency. The specific evaluation process is as follows:

[0103] Extract all matching point pairs from the matching result set M, expressed as: M(x, y) = (d_IRx, d_VISy); calculate the corresponding slopes for any two matching point pairs, k_IR and k_VIS respectively; if |k_IR-k_VIS| < K0, the slope consistency index R = 1, otherwise, the slope consistency index R = 0, where K0 represents the slope threshold; combine the slope consistency index R with the distance dist(d_IRx, d_VISy) to obtain the comprehensive matching score S(d_IRx, d_VISy), and

[0104] S(d_IRx,d_VISy)={1 / [dist(d_IRx,d_VISy)+1]}×R,

[0105] The matching result set M' is updated according to the comprehensive matching score, and matching pairs with comprehensive matching scores higher than the threshold S0 are selected, so as to output the final matching point pair set M_final = M'.

[0106] Step S300 includes:

[0107] S301. Using matching point pairs, the corresponding pixel information in the infrared image and the visible light image is combined, and the pixel value of the initial fused image is defined as:

[0108] I_fused(a,b)=α·I_VIS(a,b)+(1-α)·I_IR(a,b),

[0109] Where I_fused(a,b) represents the value of the fused image at the pixel (a,b), I_VIS(a,b) represents the value of the visible light image at the pixel (a,b), I_IR(a,b) represents the value of the infrared image at the pixel (a,b), and α represents the weight coefficient, which ranges from [0,1];

[0110] S302. Standardize the temperature information of the infrared image, and assume that the temperature range of the infrared image is [T_min, T_max]. The specific standardization calculation formula is:

[0111] T_norm(a,b)=[T_IR(a,b)-T_min] / (T_max-T_min),

[0112] Wherein T_IR(a,b) represents the temperature value corresponding to the pixel point (a,b) of the infrared image, T_min represents the minimum temperature value of the corresponding infrared image, and T_max represents the maximum temperature value of the corresponding infrared image; according to the temperature information and visible light information in the matching point, and based on the fusion strategy of weight adjustment, the optimized fused image data is generated, and the fusion strategy based on weight adjustment is: when the temperature information of the standardized infrared image is higher than the threshold T0, the influence of the infrared information in the fused image is enhanced, and the corresponding expression is: W(a,b)=h·T_norm(a,b); otherwise, the visible light information is enhanced, and the corresponding expression is: W(a,b)=1-h; wherein W(a,b) represents the weight of the pixel point (a,b) in the optimized fused image, and h represents an adjustable parameter;

[0113] S303. Generate the final fused image by weighted averaging the fusion results of all pixels, and the corresponding calculation formula is:

[0114] I_final(a,b)=W(a,b)·I_VIS(a,b)+[1-W(a,b)]·T_norm(a,b), define an enhancement function E to optimize the final fused image, and I_optimized=E(I_final), so that the optimized fused image I_optimized is output as the final color image data I_color.

[0115] Step S400 includes:

[0116] S401. Obtain the color image data to be evaluated and the standard image data for comparison, scale the color image and the standard image to the same resolution, and calculate the difference image Y by pixel-by-pixel comparison, and the calculation formula is:

[0117] Y(a,b)=|I_color(a,b)-I_standard(a,b)|,

[0118] Where I_color(a,b) represents the pixel point (a,b) of the color image data, and I_standard(a,b) represents the pixel point (a,b) of the standard image data;

[0119] S402. Calculate the mean square error MSE, and the calculation formula of the mean square error MSE is:

[0120] MSE=(1 / N)Σa∈[1,W],Σb∈[1,H]Y(a,b)^2;

[0121] Wherein N=W×H is the total number of pixels of the image, and W and H are the width and height of the image respectively; the peak signal-to-noise ratio PSNR is calculated in combination with the mean square error MSE, and the calculation formula of the peak signal-to-noise ratio PSNR is: PSNR=10·log10(MAX^2 / MSE), wherein MAX represents the maximum value of the image pixel; according to the quality score threshold P, if PSNR≥P, the output image quality is good, if P>PSNR≥P-10, the output image quality is general, and if PSNR<P-10, the output image quality is poor; images with good image quality are saved, images with general image quality are output to relevant personnel for further processing by relevant personnel, and images with poor image quality are output with warning information to relevant personnel, and the relevant personnel reacquire the image.

[0122] The value of the quality score threshold P needs to be adaptively adjusted, including:

[0123] An initial quality score threshold P0 is set, and the relationship between the PSNR values ​​of several past evaluations and P0 is recorded, and the corresponding image quality rating is obtained by referring to the comparison result of PSNR and P in S402, and the image quality rating includes good, general, and poor; the mean μ and standard deviation σ of PSNR are calculated based on the results of several evaluations, and the threshold P is dynamically adjusted based on the mean and standard deviation of PSNR, and P_new=μ+c·σ, where c is the adjustment coefficient, c=1 means that the threshold is based on the mean and a standard deviation, c>1 means increasing the threshold, which is suitable for high-quality image requirements, and c<1 means reducing the threshold, which is suitable for low-quality image requirements;

[0124] According to the adaptively adjusted threshold value P_new, the mapping relationship of the image quality rating is redefined: if PSNR≥P_new, the output image quality is good; if P_new>PSNR≥P_new-10, the output image quality is average; if PSNR<P_new-10, the output image quality is poor.

[0125] The above method is applied to a distributed photovoltaic system, and the optimized load distribution between nodes in the distributed photovoltaic system includes the following contents:

[0126] Every selected time period, the resource utilization of each node in the system is collected, and the resource utilization includes CPU, memory and network bandwidth, etc.; based on the collected resource utilization, the load index L of each node is calculated, and the specific calculation formula is: L_q=U_q / G_q, where L_q represents the load index of the qth node, U_q represents the resource utilization of the qth node, and G_q represents the total resource available of the qth node; the resource utilization of each node in the system is obtained and the average value is calculated, and the calculated average value is used as the target load value L0, and the number of tasks that need to be migrated is determined according to the difference D_q between the target load and the current load, and D_q=L_q-L0; for nodes with D_q>0, the number of tasks that need to be migrated WORK_move=f(D_q)·WORK_total, where f(D_q) represents the migration ratio function calculated according to the difference D_q, and WORK_total represents the total number of tasks to be allocated.

[0127] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0128] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A color image data processing method based on a distributed optoelectronic system, characterized in that: The method comprises the following steps: Step S100. Using the infrared sensor and the visible light sensor in the distributed optoelectronic system to simultaneously collect raw image data, and the raw image data includes infrared image data and visible light image data; preprocessing the raw image data, and matching the infrared image data and the visible light image data according to the timestamp, so as to form a raw image data set; Step S200. According to the original image data set, for each element in the original image data set, feature point extraction and matching of the infrared image data and the visible light image data are performed, and the matching results of the feature points are further analyzed to obtain the final matching point pair; Step S300. Based on the matching point pairs, perform deep information fusion on the corresponding infrared image data and visible light image data to obtain initial fused image data; perform corresponding image optimization on the initial fused image data, and obtain color image data according to the image optimization result; Step S400. Compare the color image data with the corresponding standard image data, evaluate the quality of the color image data according to the comparison result, and obtain a corresponding quality score; output prompt information according to the size relationship between the quality score and the corresponding quality score threshold, and perform corresponding processing based on the prompt information.

2. The color image data processing method based on a distributed optoelectronic system according to claim 1, characterized in that: The step S100 includes: S101. In the distributed optoelectronic system, a timestamp server is set up, and the times of the infrared sensor and the visible light sensor in the distributed optoelectronic system are kept consistent by initializing the timestamp server; before the infrared sensor and the visible light sensor collect the original image data, the timestamp server is requested and recorded locally; at the same time, the infrared sensor and the visible light sensor are started to collect the original image data, and the infrared sensor and the visible light sensor store the collected original image data and their corresponding timestamps accordingly; S102. Preprocess the collected raw image data to obtain the timestamp set T_IR of the infrared image data and the timestamp set T_VIS of the visible light image data, and T_IR={t_IR1, t_IR2, ..., t_IRn}, T_VIS={t_VIS1, t_VIS2, ..., t_VISm}, where t_IR1 represents the first element in the timestamp set of the infrared image data, t_IR2 represents the second element in the timestamp set of the infrared image data, and so on, t_IRn represents the timestamp set of the infrared image data. Similarly, t_VIS1 represents the first element in the timestamp set of visible light image data, t_VIS2 represents the second element in the timestamp set of visible light image data, t_VISm represents the mth element in the timestamp set of visible light image data, n and m represent the timestamp numbers of the infrared sensor and the visible light sensor respectively; according to the timestamp set T_IR of the infrared image data and the timestamp set T_VIS of the visible light image data, the infrared image data and the visible light image data are corresponded to form the original image data set.

3. The color image data processing method based on a distributed optoelectronic system according to claim 2, characterized in that: According to the timestamp set T_IR of the infrared image data and the timestamp set T_VIS of the visible light image data, the infrared image data and the visible light image data are matched to form an original image data set, and the specific contents include: The objective function Z is set to minimize the time difference between the infrared image timestamp and the visible light image timestamp, and the objective function expression is: Z=Σi∈[1,n],Σj∈[1,m],|t_IRi-t_VISj|·xij; Where xij is a binary variable, indicating whether the i-th timestamp of the infrared image data matches the j-th timestamp of the visible light image data, and when the i-th timestamp of the infrared image data matches the j-th timestamp of the visible light image data, xij=1, otherwise, xij=0; for each infrared image data, a matching condition is defined so that each infrared image can only be matched with one visible light image, and the constraint condition is: For each visible light image data, a matching condition is defined so that each visible light image can only match one infrared image at most. The constraint conditions are: According to the objective function and constraints, the optimization algorithm is used to solve the objective function Z to obtain the timestamp matching result xij, and the corresponding relationship between the successfully matched infrared image and the visible light image is extracted according to the matching result to form the original image data set.

4. The color image data processing method based on a distributed optoelectronic system according to claim 1, characterized in that: The step S200 includes: S201. According to the original image data set, for each element in the original image data set, use a feature point detection algorithm to extract feature points from the infrared image data and the visible light image data corresponding to the element in the original image data set, and obtain a feature point set P_IR and a feature point set P_VIS respectively, and P_IR={p_IR1, p_IR2, ..., p_IRu}, P_VIS={p_VIS1, p_VIS2, ..., p_VISv}, wherein p_IR1 represents the first feature point of the infrared image data, p_IR2 represents the second feature point of the infrared image data, and so on, p_IRu represents the uth feature point of the infrared image data; similarly, p_VIS1 represents the first feature point of the visible light image data, p_VIS2 represents the second feature point of the visible light image data, p_VISv represents the vth feature point of the visible light image data, and u and v represent the number of feature points of the infrared image data and the visible light image data respectively; S202. For each feature point in the feature point set P_IR and the feature point set P_VIS, the corresponding feature descriptor is calculated, which can be expressed as: d_IRx = F(p_IRx) and d_VISy = F(p_VISy), where x ranges from 1 to u, y ranges from 1 to v, and F represents the feature descriptor algorithm used; the feature descriptors corresponding to the feature point set P_IR and the feature point set P_VIS are summarized to form matrices D_IR and D_VIS respectively; each feature descriptor in the matrix D_IR is traversed, and the distance dist(d_IRx, d_VISy) between the feature descriptors and all the feature descriptors in the matrix D_VIS is calculated, and the matching pairs with the smallest distance are selected to form a matching result set M; the matching result set M is evaluated for slope consistency, and the final matching point pairs are obtained according to the evaluation results.

5. The color image data processing method based on a distributed optoelectronic system according to claim 4, characterized in that: The matching result set M is evaluated for slope consistency, and the specific evaluation process is as follows: Extract all matching point pairs from the matching result set M, expressed as: M(x, y) = (d_IRx, d_VISy); calculate the corresponding slopes for any two matching point pairs, k_IR and k_VIS respectively; if |k_IR-k_VIS| < K0, the slope consistency index R = 1, otherwise, the slope consistency index R = 0, where K0 represents the slope threshold; combine the slope consistency index R with the distance dist(d_IRx, d_VISy) to obtain the comprehensive matching score S(d_IRx, d_VISy), and S(d_IRx,d_VISy)={1 / [dist(d_IRx,d_VISy)+1]}×R, The matching result set M' is updated according to the comprehensive matching score, and matching pairs with comprehensive matching scores higher than the threshold S0 are selected, so as to output the final matching point pair set M_final = M'.

6. The color image data processing method based on a distributed optoelectronic system according to claim 1, characterized in that: The step S300 includes: S301. Using matching point pairs, the corresponding pixel information in the infrared image and the visible light image is combined, and the pixel value of the initial fused image is defined as: I_fused(a,b)=α·I_VIS(a,b)+(1-α)·I_IR(a,b), Where I_fused(a,b) represents the value of the fused image at the pixel (a,b), I_VIS(a,b) represents the value of the visible light image at the pixel (a,b), I_IR(a,b) represents the value of the infrared image at the pixel (a,b), and α represents the weight coefficient, which ranges from [0,1]; S302. Standardize the temperature information of the infrared image, and assume that the temperature range of the infrared image is [T_min, T_max]. The specific standardization calculation formula is: T_norm(a,b)=[T_IR(a,b)-T_min] / (T_max-T_min), Wherein T_IR(a,b) represents the temperature value corresponding to the pixel point (a,b) of the infrared image, T_min represents the minimum temperature value of the corresponding infrared image, and T_max represents the maximum temperature value of the corresponding infrared image; according to the temperature information and visible light information in the matching point, and based on the fusion strategy of weight adjustment, the optimized fused image data is generated, and the fusion strategy based on weight adjustment is: when the temperature information of the standardized infrared image is higher than the threshold T0, the influence of the infrared information in the fused image is enhanced, and the corresponding expression is: W(a,b)=h·T_norm(a,b); otherwise, the visible light information is enhanced, and the corresponding expression is: W(a,b)=1-h; wherein W(a,b) represents the weight of the pixel point (a,b) in the optimized fused image, and h represents an adjustable parameter; S303. Generate the final fused image by weighted averaging the fusion results of all pixels, and the corresponding calculation formula is: I_final(a,b)=W(a,b)·I_VIS(a,b)+[1-W(a,b)]·T_norm(a,b), define an enhancement function E to optimize the final fused image, and I_optimized=E(I_final), so that the optimized fused image I_optimized is output as the final color image data I_color.

7. The color image data processing method based on a distributed optoelectronic system according to claim 1, characterized in that: The step S400 includes: S401. Obtain the color image data to be evaluated and the standard image data for comparison, scale the color image and the standard image to the same resolution, and calculate the difference image Y by pixel-by-pixel comparison, and the calculation formula is: Y(a,b)=|I_color(a,b)-I_standard(a,b)|, Where I_color(a,b) represents the pixel point (a,b) of the color image data, and I_standard(a,b) represents the pixel point (a,b) of the standard image data; S402. Calculate the mean square error MSE, and the calculation formula of the mean square error MSE is: MSE=(1 / N)Σa∈[1,W],Σb∈[1,H]Y(a,b)^2; Wherein N=W×H is the total number of pixels of the image, and W and H are the width and height of the image respectively; the peak signal-to-noise ratio PSNR is calculated in combination with the mean square error MSE, and the calculation formula of the peak signal-to-noise ratio PSNR is: PSNR=10·log10(MAX^2 / MSE), wherein MAX represents the maximum value of the image pixel; according to the quality score threshold P, if PSNR≥P, the output image quality is good, if P>PSNR≥P-10, the output image quality is general, and if PSNR<P-10, the output image quality is poor; images with good image quality are saved, images with general image quality are output to relevant personnel for further processing by relevant personnel, and images with poor image quality are output with warning information to relevant personnel, and the relevant personnel reacquire the image.

8. The color image data processing method based on a distributed optoelectronic system according to claim 7, characterized in that: The value of the quality score threshold P needs to be adaptively adjusted, and the specific contents include: An initial quality score threshold P0 is set, and the relationship between the PSNR values ​​evaluated several times in the past and P0 is recorded, and the corresponding image quality rating is obtained by referring to the comparison result of PSNR and P in S402, and the image quality rating includes good, average, and poor; the mean μ and standard deviation σ of PSNR are calculated based on the results of several evaluations, and the threshold P is dynamically adjusted based on the mean and standard deviation of PSNR, and P_new=μ+c·σ, where c is the adjustment coefficient; According to the adaptively adjusted threshold value P_new, the mapping relationship of the image quality rating is redefined: if PSNR≥P_new, the output image quality is good; if P_new>PSNR≥P_new-10, the output image quality is average; if PSNR<P_new-10, the output image quality is poor.

9. The color image data processing method based on a distributed optoelectronic system according to claim 1, characterized in that: The method is applied to a distributed photovoltaic system, and includes the following contents for optimizing load distribution between nodes in the distributed photovoltaic system: At every selected time period, the resource utilization of each node in the system is collected. Based on the collected resource utilization, the load index L of each node is calculated, and the specific calculation formula is: L_q=U_q / G_q, where L_q represents the load index of the qth node, U_q represents the resource utilization of the qth node, and G_q represents the total available resources of the qth node; the resource utilization of each node in the system is obtained and the average value is calculated, and the calculated average value is used as the target load value L0. According to the difference D_q between the target load and the current load, and D_q=L_q-L0, the number of tasks that need to be migrated is determined; For nodes with D_q>0, the number of tasks to be migrated WORK_move=f(D_q)·WORK_total, where f(D_q) represents the migration ratio function calculated according to the difference D_q, and WORK_total represents the total number of tasks to be allocated.

Images