A micro-change monitoring method based on a multi-task neural network

Abstract

The application discloses a target micro-change monitoring method based on a multi-task neural network, and the target scene is photographed under multiple illumination conditions at two observation time points to obtain a current image set and a reference image set respectively; the surface structure diagram and the reflectivity diagram of the scene at the current time point and the surface structure diagram and the reflectivity diagram of the scene at the reference time point are calculated by using the current image set and the reference image set respectively; the obtained two groups of surface structure diagrams and reflectivity diagrams are aligned respectively; the aligned two groups of surface structure diagrams and reflectivity diagrams are input into a multi-scale change monitoring model, and finally the micro-change detection result of the target is obtained. Compared with the prior art, the application detects the change of the scene according to the three-dimensional information of the target and the reflectivity diagram, is not affected by the difference of the illumination conditions during imaging, can more accurately find the subtle change of the target, and is especially suitable for the tasks with high monitoring precision requirements such as cultural relic monitoring and precision instrument monitoring.

Description

TECHNICAL FIELD

[0001] The present application relates to the fields of artificial intelligence, computer vision and micro change monitoring technology, and particularly relates to a micro change monitoring method based on a multi-task neural network. BACKGROUND

[0002] Micro change monitoring technology aims to monitor the micro structural changes or texture changes of a target scene in a certain time period, and plays an important role in computer vision problems such as remote sensing and cultural relic protection.

[0003] At present, the solution to the micro change monitoring problem mainly detects the changed area of the image pair of the target scene at the reference and current two time points. However, the lighting conditions at the two time points are usually inconsistent, so there will be a large color difference between the two images taken at different times, and the subtle changes of the target are easily covered by the above difference. However, the existing method directly detects the target changes from the obtained images, so it can usually detect large-scale target changes, and the detection accuracy for small changes is not ideal. SUMMARY

[0004] To solve the above technical problems, the present application provides a micro change monitoring method based on a multi-task neural network, which recovers the surface structure map and reflectivity map of the scene from the input image, and then monitors the micro changes of the scene accordingly.

[0005] The present application uses the following technical solutions:

[0006] A micro change monitoring method based on a multi-task neural network, the method comprising the following steps:

[0007] Step 1, at two observation time points, the target scene is photographed multiple times under multiple lighting conditions to obtain a current image set and a reference image set Where K and H are the number of input images;

[0008] Step 2, the current image set and the reference image set are used as input images to recover the surface structure map N cur , the reflectivity map R cur of the target scene at the current time point, and the surface structure map N ref , the reflectivity map R ref at the reference time point;

[0009] Step 3, the two sets of surface structure maps and reflectivity maps obtained are respectively aligned in pixels and scales to obtain the aligned surface structure map N and reflectivity map R at the reference time point. The specific process of the alignment processing is:

[0010] Step 4, aligning the and N cur , R cur to the multi-scale change detection model, to obtain different scale micro change detection results The micro change detection results of different scales are fused to obtain the final micro change detection result C; the formula is as follows:

[0011]

[0012] Wherein, is element-wise multiplication.

[0013] Compared with the prior art, the automatic assembling system of the capping pipe piece based on deep learning and laser of the application can achieve the following technical effects:

[0014] 1) to realize accurate monitoring of the micro change of the target scene at two observation time points, and avoid direct detection of changes from the input image;

[0015] 2) ensure the robustness of the imaging condition difference. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 It is a micro change monitoring method based on a multi-task neural network;

[0017] Figure 2 It is an algorithm block diagram of a micro change monitoring method based on a multi-task neural network;

[0018] Figure 3 It is an embodiment diagram;

[0019] Figure 4 It is an optical flow alignment module flow chart;

[0020] Figure 5 It is a multi-scale change monitoring module flow chart;

[0021] Figure 6 It is a micro change detection data set example. DETAILED DESCRIPTION

[0022] The technical solutions will be described in detail below with reference to the drawings and examples. Based on the examples in the application, all other examples and technical replacements of examples obtained by those skilled in the art without departing from the spirit of the application and without making creative efforts will fall within the scope of protection of the application.

[0023] ​The technical solutions of this invention will now be clearly and completely described with reference to the accompanying drawings. All other embodiments obtained by those skilled in the art based on the technical solutions of this invention without inventive effort are within the scope of protection of this invention.

[0024] like Figure 1 The diagram shown is an overall flowchart of a method for detecting minute changes based on a multi-task neural network according to the present invention. The specific process includes the following steps:

[0025] Step 1: At two observation time points, take multiple photos of the target scene under various lighting conditions to obtain the current image set. and reference image set (K and H are the number of input images), the specific acquisition of the current image set and the reference image set is as follows:

[0026] At a reference time point, H photos of the target scene were acquired under H different lighting conditions to obtain a reference image set.

[0027] At the current time point, acquire K photos of the target scene under K different lighting conditions to obtain the current image set.

[0028] Generally, K=13 and H=13 are selected. A light source (such as a surface light panel) is used to actively illuminate the target scene from the clockwise direction (1-12 o'clock) and the frontal direction, and scene images are acquired separately. Note that the direction of the light source is not strictly required here; only that images are captured under multi-angle lighting.

[0029] Step 2: Using the current image set and the reference image set as input images, reconstruct the surface structure map N of the target scene at the current time point using the input images. cur Reflectance diagram R cur and surface structure diagram N at the reference time point ref Reflectance diagram R ref The specific method is as follows:

[0030] Step 2.1: Transfer the current image set T cur and reference image set T ref The data are input into the photometric 3D network model to obtain the current surface structure diagram and the illumination condition N. cur , (1≤i≤K), and reference surface structure diagram and illumination condition N ref , (1≤i≤H); The photometric stereo network model used here can be any deep learning-based photometric stereo network;

[0031] Step 2.2, according to the surface structure map and the illumination condition N of the current time point cur , and the reference surface structure map and the illumination condition N ref , The current reflectance map R cur and the reference reflectance map R ref are calculated respectively by using the Lambertian rendering model.

[0032] The calculation formula of the reflectance map is as follows:

[0033]

[0034]

[0035] wherein, represents element-wise division.

[0036] Step 3, the obtained two sets of surface structure maps and reflectance maps are respectively aligned in pixels and scales, to obtain the aligned reference time point surface structure map N and reflectance map R The specific process of the alignment processing is as follows:

[0037] Step 3.1, input N cur , N ref and R cur , R ref to the optical flow estimation module respectively, to obtain the optical flow maps F N and F R ; the optical flow estimation module used here can be a deep learning-based optical flow estimation network.

[0038] Step 3.2, input N cur , N ref , R cur , R ref and the optical flow maps F N and F R to an encoder-decoder together, the encoder-decoder obtains the final optical flow map F according to the input information; wherein the encoder and the decoder of the encoder-decoder are both composed of 5 convolutional layers, and are connected by a skip connection layer between the 4th and 7th layers and the 2nd and 9th layers; according to the optical flow map F, the pixel alignment of the reference normal map and the reflectance map N ref , R ref to the current scene is obtained, to obtain the pixel-aligned The formula is as follows:

[0039]

[0040]

[0041] Wherein, Warp is a coordinate mapping operation.

[0042] The normal direction of the normal map is aligned to obtain an aligned normal map The formula is as follows:

[0043]

[0044] The global rotation matrix corresponding to the normal map is calculated by the following formula:

[0045]

[0046] Wherein, N represents the scene normal map, and p represents the index of the pixel in the image.

[0047] On the basis of the overall tone of the reflectance map is corrected to obtain an aligned reflectance map The formula is as follows:

[0048]

[0049] In the above formula respectively represent the conversion matrix and the conversion vector about the color difference of the reflectance map, is a 3x3 matrix, is a 3x1 vector, and and can convert the color information in to R represents the calculated scene reflectance map, that is, the color information of the scene itself.

[0050] Wherein and are calculated by the following formula:

[0051]

[0052] Step 4, input the aligned and N cur , R cur to the multi-scale change detection model to obtain a multi-scale micro-change detection result, and the specific process is as follows:

[0053] Step 4.1, input the aligned reference normal map reflectance map and the current N cur , R cur to the multi-scale micro-change detection module to obtain change detection results of different scales

[0054] Step 4.2, change detection result of different scales fusion to obtain the final change detection result C;

[0055] fusion of changes in to obtain the final micro-change detection result C, and the formula is as follows:

[0056]

[0057] wherein, is element-wise multiplication.

[0058] Specifically, the preliminary multi-scale change detection result marks the change area in the figure as 0, and marks the non-change area as 1.

[0059] Specifically, the structure of the multi-scale change detection model is described as follows:

[0060] The feature extraction part of the multi-scale change detection model is composed of two pyramid networks sharing weights. The two pyramid networks respectively perform multi-scale feature extraction on and N cur , R cur , and input the corresponding depth scale into the attention module to regress the change detection results of different scales

[0061] Unlike the way of directly detecting target changes from images in the previous change detection method, the present application detects changes in the scene according to the target three-dimensional information and the reflectance map, and is not affected by the difference in lighting conditions during imaging, so it can more accurately find the subtle changes of the target, and is especially suitable for monitoring of cultural relics, monitoring of precision instruments and other tasks with high monitoring precision requirements. In addition, since the surface structure map and the reflectance map both represent the scene's own information and are independent of the light, this strategy ensures the robustness of the present application to imaging condition differences, especially light differences.

[0062] The feasibility of the method of the present application is verified by combining specific examples, and the details are described below:

[0063] The verification experiment of the method of the present application is performed on the synthetic data set and the real scene data set. The experiment uses IoU (intersection over union), Precision (accuracy), Recall (recall rate) and F1-Score four indexes to quantitatively evaluate the accuracy of the method. In order to avoid the experimental fairness problem caused by the difference in input data format, this paper uses a variety of different data formats to test the comparison method, and appropriately modifies the input layer structure of the comparison method to adapt to the input of different formats.

[0064] The quantitative comparison results of the test of the method of the application and the mainstream method on the micro-change detection data set are shown in Table 1. Among them, FGCD2015_ML is a change detection method based on optimization of a traditional model, ChangeNet, STANet and SSCDNet are three mainstream deep learning-based target change detection methods, and “_SL” and “_ML” respectively represent single-illumination images and multi-illumination images as input. From the results, it can be seen that the method of the application reaches the optimal level, which proves the effectiveness of the method of the application.

[0065] Table 1

[0066]

[0067]

Claims

1. A method of monitoring for minor changes based on a multitask neural network, the method comprising: The method comprises the following steps: Step 1, at two observation time points, respectively, multiple photographs are taken under multiple illumination conditions to obtain a current image set and a reference image set wherein K and H are the number of input images; Step 2, using the input images, the current image set and the reference image set, to restore the surface structure map N of the target scene at the current time point cur , reflectivity map R cur , and the surface structure map N of the reference time point ref , reflectivity map R ref ; Step 3, the surface structure map and the reflectivity map of the two groups of surface structure maps and reflectivity maps obtained are respectively aligned in pixel and scale to obtain the surface structure map of the aligned reference time point and reflectivity map The specific process of the alignment processing is that: Step 3.1, input N cur , N ref and R cur , R ref to the optical flow estimation module respectively to obtain optical flow maps F N and F R ; Step 3.2, inputting the N cur , R ref , R cur , R ref and the optical flow map F N and F R into an encoder-decoder, the encoder-decoder obtaining a final optical flow map F according to the input information; wherein the encoder and the decoder of the encoder-decoder are both composed of 5 convolutional layers, and a skip connection layer is used to connect between the 4th and 7th layers and the 2nd and 9th layers; according to the optical flow map F, aligning the pixel pairs of the reference normal map and the reflectance map N ref , R ref to the current scene to obtain the pixel-aligned N The formula is as follows: Wherein, Warp is a coordinate mapping operation; The aligning the normal orientations of the normal maps to obtain the aligned normal maps The formula is as follows: wherein, with respect to the global rotation matrix corresponding to the normal map is calculated by the following equation: N represents a scene normal map, and p represents an index of a pixel in an image; exist Based on this, the overall color tone of the reflectance map is corrected to obtain the aligned reflectance map. The formula is as follows: wherein, respectively represent a conversion matrix and a conversion vector with respect to the color difference of the reflectance map, and R represents a computed scene reflectance map; wherein and calculated from the following equation: Step 4: Align the... and N cur R cur The input is fed into a multi-scale change detection model to obtain the detection results of minute changes at different scales. Detection results of minute changes at different scales The result is fused to obtain the final small change detection result C; the formula is as follows: wherein is an element-wise multiplication. 2.The micro-change monitoring method based on multi-task neural network according to claim 1, wherein, In the step 1, the specific acquisition process of the current image set and the reference image set is as follows: At a reference time point, H photos of the target scene are acquired under H different lighting conditions respectively, to obtain a reference image set At the current time point, K photos of the target scene are acquired under K different illumination conditions respectively, to obtain a current image set 3.The micro-change monitoring method based on multi-task neural network according to claim 2, wherein, The step 2 specifically comprises the following processing: Step 2.1, input the current image set and the reference image set into the photometric stereo network model respectively, to obtain the current surface structure map and the illumination condition N cur , and the reference surface structure map and the illumination condition N ref , Step 2.

2. Calculate the current reflectance map R cur , and the reference reflectance map R ref , using the Lambertian rendering model, respectively, based on the surface structure map and the lighting conditions N cur at the current time point and the reference surface structure map and the lighting conditions N ref , respectively. The calculation formula of the reflectance map is as follows: wherein represents element-wise division. 4.The micro-change monitoring method based on multi-task neural network according to claim 1, wherein, The feature extraction part of the multi-scale change detection model is composed of two pyramid networks sharing weights, and the two pyramid networks respectively perform multi-scale feature extraction on and N cur , R cur , and the scales of the corresponding depths are respectively input into the attention module to regress the change detection results of different scales

Images