Intelligent vehicle image completion method based on image cutting and tensor interconnection completion

Abstract

The application belongs to the technical field of image processing, and provides an intelligent carrier image completion method based on image cutting and tensor interconnection completion, which comprises the following steps: storing an RGB format picture shot by a camera in the form of a tensor as a target tensor; according to the size and gray value of a current frame tensor, pre-processing a prior tensor, and simultaneously cutting the pre-processed prior tensor to generate a reference tensor by using an AIP algorithm, and stacking the reference tensor and the target tensor into a synthetic tensor; processing the synthetic tensor by using an ITSC algorithm to obtain a processed tensor after interconnection completion; extracting a tensor of a current image corresponding region from the processed tensor to obtain a completion tensor, and completing image completion. The application is superior to the current main intelligent flight carrier completion method in stability and effectiveness of image completion, and application of the method can greatly improve the stability and effectiveness of image completion of the intelligent flight carrier in a bad weather and poor communication scene.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to an intelligent carrier image completion method based on image cutting and tensor interconnection completion. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute prior art.

[0003] In bad weather such as rain, snow and sand, and in poor communication conditions, the images taken by intelligent flying carriers often appear to be blocked, image missing, transmission data loss or noise interference, etc. These interferences seriously affect the quality, stability and reliability of the images taken by intelligent flying carriers.

[0004] At present, the image completion based on intelligent flying carriers and other aircraft mainly extracts features from single frame pictures taken by intelligent flying carriers, analyzes the contours of target objects, and performs tensor completion inside the target objects. This method can effectively handle local occlusion and a small amount of noise points on the picture, but has limited processing capacity for large area damage caused by bad weather and poor communication conditions, and easily loses some details of the target object, which cannot achieve good completion effect. SUMMARY

[0005] In order to solve the problems in the background art, the present application provides an intelligent carrier image completion method based on image cutting and tensor interconnection completion.

[0006] The present application provides an intelligent carrier image completion method based on image cutting and tensor interconnection completion, which comprises the following steps:

[0007] Step 1: store the RGB format picture taken by the camera in the form of tensor as a target tensor;

[0008] Step 2: pre-process the prior tensor according to the size and gray value of the current frame tensor, and cut the pre-processed prior tensor to generate a reference tensor using AIP algorithm, and stack the reference tensor and the target tensor into a synthetic tensor;

[0009] Step 3: process the synthetic tensor using ITSC algorithm to obtain a processed tensor after interconnection completion;

[0010] Step 4: extract the tensor of the corresponding region of the current image in the processed tensor to obtain a completion tensor, and complete the image completion.

[0011] Preferably, in the step 1, the R, G and B pixel values corresponding to each pixel point in the RGB format image are extracted, the obtained pixel values are filled into the corresponding layers of the tensor respectively in R, G and B values, and the tensor is obtained .

[0012] Preferably, the specific method of preprocessing the prior tensor in step 2 is as follows:

[0013] Step 2.1: Geometric transformation of the prior tensor according to the size of the target tensor, for making the prior tensor consistent with the size of the target tensor;

[0014] Step 2.2: Gray scale adjustment of the prior tensor according to the size of the target tensor, for making the overall brightness of the prior image consistent with that of the target image.

[0015] Preferably, the specific method of generating the reference tensor using the AIP algorithm in step 2 is as follows:

[0016] Step 3.1: Obtain the main motion parameters and image parameters of the intelligent flying vehicle;

[0017] Step 3.2: Determine the number of pixels cut from each side of the image according to the obtained parameters, and determine a new rectangular region. Pre-cut the prior tensor image to obtain the pre-cut region ;

[0018] Step 3.3: Perform semantic segmentation on the pre-cut region and the target tensor, and divide the image into different features , i is the feature number, and generate two feature sets;

[0019] Step 3.4: Generate feature matrices and from the position information of a pair of corresponding features in the target tensor and corresponding feature set, respectively;

[0020] Step 3.5: Calculate the affine matrix Γ from the two feature matrices and ;

[0021] Step 3.6: Process the feature matrix in according to the affine matrix Γ, and obtain a new feature matrix. Perform affine transformation on the features in according to the new feature matrix;

[0022] Step 3.7: Continue to take a pair of corresponding features that have not been processed, and repeat steps 3.4 to 3.6 until all features in the target tensor are processed;

[0023] Step 3.8: Fill in the pixel points with no information in the tensor after affine transformation with the surrounding pixel points to obtain the reference tensor processed by the AIP algorithm .

[0024] Preferably, the motion parameters include speed, height, pitch angle, roll angle, climb rate, acceleration, and rotation angle; and the image parameters include image resolution, horizontal view angle, vertical view angle, distance between the intelligent flying vehicle and the photographed object, and frame interval of the intelligent flying vehicle.

[0025] Preferably, in step 3.3, the specific method for performing semantic segmentation on the pre-cut region and the target tensor is as follows:

[0026] Step 4.1: Each pixel point in the image corresponds to an array at the corresponding x-axis and y-axis position in the tensor, so an array in the tensor determined by the x-axis and y-axis is used to represent a point on the graph, and the point mentioned in the following steps is an array in the tensor.

[0027] Step 4.2: Set appropriate growth threshold and termination division threshold.

[0028] Step 4.3: Select the center point of the tensor as the initial growth point, create a point set and add the initial growth point to the point set, then repeatedly check the adjacent points of the point set, and add the points meeting the growth threshold condition to the point set until no point can be added, and this point set is referred to as a feature.

[0029] Step 4.4: Select the largest region on the graph that is not divided into a feature, if the region is larger than the termination division threshold, select the center of the region as a new initial growth point, divide a new feature, and repeat the process until the semantic segmentation is completed, and all features are collectively referred to as a feature set of the image.

[0030] Preferably, in step 4, the specific method for processing the synthesized tensor by using the ITSC algorithm to obtain the processed tensor after the interconnection completion is as follows:

[0031] Step 5.1: Decompose the tensor into a product of a series of tensor factors, so as to facilitate subsequent optimization of the tensor by optimizing the tensor factors.

[0032] Step 5.2: Establish an error function and an optimization objective function model to measure the error size between the tensor decomposition result and the original tensor and iteratively update the tensor factors;

[0033] Step 5.3: Iteratively update the tensor factors based on the optimization objective function to minimize the difference between the upper and lower parts of the synthesized tensor.

[0034] Step 5.4: Combine the optimized tensor factors to obtain the processed tensor, and the size of the processed tensor is N×M×6.

[0035] Compared with the prior art, the application has the following beneficial effects: ​

[0036] The intelligent flight vehicle is stable and controllable, and the motion parameters such as flight speed are easy to obtain. According to the motion parameters, the corresponding region image of the previous frame is cut out as a prior image. The current frame image is used as a new prior image, and the key information of the missing part of the image is retained, realizing the memory image completion. The method is superior to the current main intelligent flight vehicle completion method in stability and effectiveness of image completion. The application of the method can greatly improve the stability and effectiveness of image completion of intelligent flight vehicles in bad weather and poor communication scenes. BRIEF DESCRIPTION OF DRAWINGS

[0037] The accompanying drawings, which form a part of the specification, are included to provide a further understanding of the application and are incorporated herein in conjunction with the description of the application. The embodiments of the present application, and their description, are used to explain the present application and are not intended to limit the present application.

[0038] Figure 1 An intelligent flight vehicle image adaptive cutting and completion flowchart with time memory,

[0039] Figure 2 A tensor completion model with space-time correction,

[0040] Figure 3 An interconnected tensor decomposition module,

[0041] Figure 4 An iterative module and an update module, DETAILED DESCRIPTION

[0042] The present application will be further described below in conjunction with the drawings and embodiments.

[0043] It should be noted that the terms used herein are only intended to describe specific embodiments and are not intended to limit the exemplary embodiments according to the present disclosure. As used herein, the singular form is intended to include the plural form unless the context clearly indicates otherwise, and it should also be understood that when the terms "comprise" and / or "include" are used in the specification, they indicate the presence of the features, steps, operations, devices, components and / or combinations thereof.

[0044] In the present disclosure, the terms such as "upper", "lower", "left", "right", "front", "back", "vertical", "horizontal", "side", "bottom" and the like indicate the orientation or positional relationship shown in the drawings, which is only a relationship word determined for the convenience of describing the structural relationship of the components or elements of the present disclosure, and cannot be understood as a limitation of the present disclosure.

[0045] Example 1

[0046] As Figures 1 to 4 shown, the present application provides an intelligent carrier image completion method based on image cutting and tensor interconnection completion, specifically as follows:

[0047] In the first aspect, the present application provides an adaptive image cutting method, which receives intelligent carrier motion parameters including flight speed, pitch angle, roll angle, climb rate, acceleration, and turning angle, etc. The motion parameters of the intelligent carrier are used to calculate the moving distance of the image captured by the unmanned aerial vehicle camera. This distance is converted into the number of pixels in the corresponding image. The corresponding area of the previous frame image is continuously cut during the movement of the unmanned aerial vehicle. The cut image can be used as a basis for image completion after processing, and adaptive image cutting is completed.

[0048] In the second aspect, the present application proposes an image completion method with memory. The previous frame image of the intelligent carrier is stored as a prior tensor (size ), which is cut into a reference tensor (size ) through adaptive cutting. The current frame image is converted into a target tensor . The reference tensor and the target tensor are stacked into a synthetic tensor (size ).

[0049] The synthetic tensor is completed using the tensor interconnection completion algorithm. The region corresponding to the completed target tensor is extracted, and the completion of the target tensor is completed. At the same time, the tensor is used as a new prior tensor, and the process is repeated continuously. Each time the image completion is affected by multiple previous images, so that the algorithm has memory.

[0050] In the third aspect, the present application proposes an interconnection tensor completion algorithm with space-time factor correction (i.e. ITSC algorithm), which completes the synthetic tensor to obtain the completed image tensor.

[0051] The specific implementation process is as follows:

[0052] The current frame image captured by the intelligent carrier is stored as a target tensor. The target tensor and the prior tensor are preprocessed. The gray value of each point is calculated according to the formula . Based on the gray value, the brightness histogram corresponding to the tensor is generated. The target tensor and the prior tensor are matched in brightness and histogram equalization, balancing the brightness distribution of the target tensor and the prior tensor.

[0053] When capturing the current frame image, the current real-time motion parameters of the intelligent carrier are received. The speed, pitch angle, roll angle, climb rate, acceleration, and turning angle parameters are input into the AIP algorithm for effective area identification, cutting, and alignment.

[0054] Specifically, the basic movements of the intelligent flying vehicle are divided into the following four kinds, and the movement of the intelligent flying vehicle is considered as the four basic movements or their combinations:

[0055] The vertical direction is taken as the z-axis, the camera shooting direction is taken as the x-axis, the direction perpendicular to the x-axis and the z-axis is taken as the y-axis, and the basic movement of the intelligent flying vehicle is defined as: moving along the z-axis at a speed upward movement, moving along the x-axis at a speed forward movement, moving along the y-axis at a speed right movement, horizontal angular velocity clockwise rotation.

[0056] The image is pre-cut according to the priori, the purpose is to roughly determine the effective area, determine the area for subsequent feature recognition, feature matching and affine transformation, reduce the interference of invalid area, the steps of image pre-cutting are as follows:

[0057] First, determine the motion state of the intelligent flying vehicle, and determine the effective area in the priori tensor according to the motion state of the intelligent flying vehicle and the following parameters, wherein the main parameters of the intelligent flying vehicle include the pixel number HxW of the camera equipment, the horizontal view angle , the vertical view angle , the distance d between the intelligent flying vehicle and the photographed object, and the frame interval time of the intelligent flying vehicle;

[0058] Record the pixel number of the effective area to the boundary of the priori tensor as (left side), (right side), (top side), (bottom side), and calculate according to the following steps:

[0059] Process the movement along the z-axis:

[0060] Process the movement along the y-axis:

[0061] Process the movement along the x-axis: (i=u, d, r, l)

[0062] Process the horizontal rotation with the clockwise direction as the positive direction:

[0063] A rectangular area with a distance of pixels from each side of the priori tensor is determined, the area is cut out, and the cut image is generated, and the image pre-cutting is completed.

[0064] For And the target tensor is characterized according to the following steps:

[0065] S1: select the region of the largest unclassified feature set in the image, and the center of the selected region is the growth point ;

[0066] S2: divide the growth point into the feature set (the feature set is a continuous point set);

[0067] S3: for the feature set , the outermost point , detect its adjacent point not in any feature set , respectively RGB three color channel corresponding value, η is the growth threshold, if there is an adjacent point satisfying the formula

[0068]

[0069] , add to the feature set ;

[0070] S4: after completing step S3, if there are new points added in the last round, then these points are used as the repeat step S3 until no point is added to the feature set in a round of detection ;

[0071] S5: check the region of the largest unclassified feature set in the image, if the area S of the region satisfies , wherein is the minimum division threshold set, then the feature recognition of the image is completed, otherwise, return to step 1 to determine a new feature set until the region of the largest unclassified feature set is smaller than the minimum division threshold .

[0072] For semantic segmentation of the prior tensor and the target image, first, adjust the number of the feature set of the prior tensor using the RGB data of the feature set of the two, and make the feature sets in the prior tensor and the target tensor one-to-one, for the pixel points in the prior tensor and the target tensor that are not classified into the feature set, divide them into the nearest feature set, and complete the semantic segmentation of the two images.

[0073] For each feature set containing n points , find the center point position, and mark the center point of the prior tensor feature , and the center point of the target tensor feature

[0074]

[0075] For each feature set , take all points in it to generate a feature matrix, denoted as , the feature matrix corresponding to the prior tensor, and , the feature matrix corresponding to the target tensor;

[0076] Determine the affine matrix according to the center point and the four corner points of the feature set , denoted as , the four corner points of the feature set in the prior tensor (k = 1, 2, 3, 4), and , the four corner points of the feature set in the target tensor (k = 1, 2, 3, 4), to obtain the following affine matrix:

[0077] Translation matrix T = Scaling matrix S =

[0078] , to obtain the affine matrix Γ = TS, process the prior using the affine matrix, and calculate = ,

[0079] Obtain , and project the RGB data of the points in the feature set to the points , )

[0080] Then traverse the points in the region corresponding to the target tensor, and if there is a pixel point that does not have an affine transformation corresponding to the data set in the prior tensor, use the average value of the neighboring RGB information pixel points to complete the RGB value of the pixel point, and finally generate a complete image .

[0081] Convert the target tensor and into tensor format, specifically:

[0082] Establish a three-order tensor, with the length of the x-axis and y-axis being the number of pixels in each column and each row of the target tensor, and the length of the z-axis being 3;

[0083] Fill the pixel values of red (R), green (G), and blue (B) corresponding to each pixel point of the image into the corresponding positions of the tensor to generate a three-order tensor that retains the RGB data of the image;

[0084] According to the AIP algorithm, perform adaptive cutting on the prior image to obtain the processed reference tensor.

[0085] The cut reference tensor is stretched and aligned with the target tensor, and the two have the same format, image size, pixel quality, overall gray scale and color channel.

[0086] The reference tensor and the target tensor are stacked into a three-order tensor with a z-axis length of 6, and the tensor is denoted as X and is input into the ITSC model for processing. The specific calculation process is as follows:

[0087] The ITSC algorithm model is initialized, the maximum value of the iteration number is set to = 500, and the initial input tensor is the synthesized tensor , that is, The initial ITSC decomposition rank is defined as follows:

[0088]

[0089] The initial value of the auxiliary matrix is as follows

[0090]

[0091] As shown in Figure 1 and Figure 3 , the tensor is decomposed into a series of tensor factors by using the ITSC interconnection decomposition module:

[0092]

[0093]

[0094] The error of the tensor decomposition is calculated by using the ITSC decomposition model, and the objective function is optimized

[0095]

[0096] A regularization term is added to filter the data , wherein the set is defined as

[0097]

[0098] A spatial correction term is added to ensure the correlation in space when the tensors of the same layer are completed

[0099]

[0100] As shown in Figure 3 , the prior image is used to correct the damaged part of the target tensor to minimize the difference between the reference tensor and the target tensor

[0101]

[0102] When the iteration number i < N, if the formula (9) is not satisfied, the model is iteratively corrected once, and the specific process is as follows: Update R = R + ΔR

[0103] ,

[0104] If the formula (10) is satisfied Expand ,

[0105]

[0106] Update by formula (11) ,

[0107]

[0108] Wherein,

[0109]

[0110] Update by formula (13) ,

[0111]

[0112] Repeat the iteration process until the formula (9) is satisfied, and output the completed tensor .

[0113] As shown in Figure 1 , the obtained completed tensor , that is, the completed intelligent flight vehicle image, is stored as a new prior tensor to participate in the next round of tensor completion.

[0114] As shown in Figures 2 to 4 , the core part of the interconnection tensor completion algorithm includes:

[0115] Decomposition module: decompose the tensor into a series of tensor factors, and calculate the local combined tensor set

[0116] Iteration module: used for correcting the decomposed tensor factor set in iterative calculation, optimizing the correlation between the tensor decomposition factors by using the interconnection model, and ensuring the correlation between the reference tensor and the target tensor by using the space-time correction factor;

[0117] Update module: update , R, in the iteration process.

[0118] Wherein, Figure 3 ​​Schematic diagram of interconnection tensor decomposition module, Figure 4 Schematic diagram of the action of the iteration module and the update module.

[0119] In summary, in the present application, the memory completion of the image of the intelligent flight vehicle is realized, and the implementation method of the present application retains the information of the previous image in the prior tensor, processes the target tensor corresponding to the current image using the prior tensor frame by frame, so that the information of the past multiple images can be obtained to complete the damaged or occluded part in the current image.

[0120] The application of the image adaptive cutting (AIP) algorithm ensures that the corresponding features in the prior tensor and the target tensor corresponding to the current image can be effectively matched while retaining the image features, thereby improving the generality and effectiveness of image completion.

[0121] The present image completion method is particularly suitable for completing the damaged points in the continuous image in cooperation with the motion data of the intelligent flight vehicle, and through the retained information of multiple images, the damaged or occluded part in each image can be restored, and the image completion effect is good for the short-term image damage and image occlusion.

[0122] The tensor interconnection completion algorithm (ITSC algorithm) is another important innovation of the present application, which can realize the iteration and update of the tensor itself by splitting the tensor factor, continuously optimize, minimize the difference between the benchmark tensor and the target tensor, and maximize the application of the stored memory image information to the current completion image.

[0123] The above is only a preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

[0124] The above describes the specific embodiments of the present application in combination with the accompanying drawings, but is not intended to limit the protection scope of the present application. Those skilled in the art should understand that various modifications or changes made on the basis of the technical solutions of the present application without creative labor are still within the protection scope of the present application.

Claims

1. An intelligent vehicle image inpainting method based on image cut and tensor interconnection completion, characterized in that, Comprising the following steps: Step 1: The RGB format picture taken by the camera is stored in the form of a tensor as a target tensor; Step 2: Preprocess the prior tensor according to the size and gray value of the current frame tensor, and cut the preprocessed prior tensor to generate a reference tensor using the AIP algorithm, and stack the reference tensor and the target tensor into a composite tensor; Step 3: Process the composite tensor using the ITSC algorithm to obtain a processed tensor after interconnection completion; Step 4: Extract the tensor corresponding to the current image region in the processed tensor to obtain a completed tensor, and complete the image completion; In step 2, the specific method for generating the reference tensor using the AIP algorithm is: Step 3.1: Obtain the main motion parameters and image parameters of the intelligent flying vehicle; Step 3.2: According to the obtained parameters, the number of pixels cut off from each side of the image distance is determined, a new rectangular region is determined, the prior tensor is pre-cut in the image, and the pre-cut region is obtained ; Step 3.3: On the pre-cut region and semantic segmentation of the target tensor, the image is segmented into different features i is the feature number, and two feature sets are generated; Step 3.4: Reshape the target tensor and Generating a feature matrix from the location information of a pair of corresponding features in the corresponding feature set and ; Step 3.5: Compute affine matrix Γ from the two feature matrices and Compute affine matrix Γ; Step 3.6: Pair according to the affine matrix Γ The characteristic matrix in The process is performed to obtain a new feature matrix, and then the new feature matrix is ​​used to... Perform affine transformations on the features in the data; Step 3.7: Continue to take a pair of corresponding features that have not been processed, and repeat steps 3.4 to 3.6 until all features in the target tensor are processed; Step 3.8: The pixels without information in the tensor after affine transformation are completed by using the surrounding pixels to obtain the reference tensor processed by the AIP algorithm ; In step 4, the specific method for processing the composite tensor using the ITSC algorithm to obtain a processed tensor after interconnection completion is: Step 5.1: Decompose the tensor into a series of tensor factors, which facilitates subsequent optimization of the tensor by optimizing the tensor factors; Step 5.2: Establish an error function and an optimization objective function model to measure the error between the tensor decomposition result and the original tensor and iteratively update the tensor factors; Step 5.3: Iteratively update the tensor factors based on the optimization objective function to minimize the difference between the upper and lower parts of the composite tensor; Step 5.4: Combine the optimized tensor factors into a processed tensor, which has a size of N x M x 6.

2. The intelligent vehicle image completion method based on image cutting and tensor interconnection completion according to claim 1, characterized in that: In the step 1, the R, G and B pixel values corresponding to each pixel point in the RGB format image are extracted, and the obtained pixel values are filled into the corresponding layers of the tensor respectively in R, G and B values to obtain the tensor .

3. The intelligent vehicle image completion method based on image cutting and tensor interconnection completion according to claim 1, characterized in that: In step 2, the specific method for preprocessing the prior tensor is: Adjust the gray scale of the prior tensor according to the size of the target tensor to make the overall brightness of the prior image consistent with that of the target image.

4. The intelligent vehicle image completion method based on image cutting and tensor interconnection completion according to claim 1, characterized in that: The motion parameters include speed, height, pitch angle, roll angle, climb rate, acceleration, and turning angle; The image parameters include image resolution, horizontal viewing angle, vertical viewing angle, distance between the intelligent vehicle and the photographed object, and frame interval time of the intelligent flying vehicle.

5. The intelligent vehicle image completion method based on image cutting and tensor interconnection completion according to claim 1, characterized in that: The step 3.3, after the pre-cutting of the region The specific method of semantic segmentation of the target tensor is as follows: Step 4.1: Each pixel in the image corresponds to an array in the tensor at the corresponding x-axis and y-axis position, so one array in the tensor determined by the x-axis and y-axis represents one point on the image, and the point mentioned in the following steps is an array in the tensor; Step 4.2: Set appropriate growth threshold and termination division threshold; Step 4.3: Select the center point of the tensor as the initial growth point, create a point set and add the initial growth point to the point set, then repeatedly check the adjacent points of the point set and add the points that meet the growth threshold condition to the point set until no point can be added, and this point set is called a feature. Step 4.4: Select the largest region on the graph that is not included in any feature. If this region is larger than the termination threshold, select the center of this region as the new initial seed point, divide a new feature, and repeat the process until the semantic segmentation is complete. All features are combined into a feature set for the image.

Images