Intelligent carrier image completion method based on image cutting and tensor interconnection completion

Abstract

The invention belongs to the technical field of image processing, and provides an intelligent carrier image completion method based on image cutting and tensor interconnection completion, and the method comprises the following steps: storing an RGB format image shot by a camera in a tensor form, and taking the image as a target tensor; preprocessing the prior tensor according to the size and the gray value of the current frame tensor, cutting the preprocessed prior tensor by using an AIP algorithm to generate a reference tensor, and stacking the reference tensor and a target tensor into a synthetic tensor; processing the synthetic tensor by using an ITSC algorithm to obtain a processed tensor after interconnection completion; and extracting the tensor of the corresponding region of the current image from the processed tensor to obtain a complemented tensor, and completing image complementation. According to the method, the stability and the effectiveness of image completion are superior to those of a current main intelligent flight vehicle completion method, and the stability and the effectiveness of image completion of the intelligent flight vehicle under severe weather and poor communication scenes can be greatly improved by applying the method.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and particularly to an intelligent vehicle image completion method based on image cutting and tensor interconnection completion. Background Technique

[0002] The statements in this part merely provide background technical information related to this application and do not necessarily constitute prior art.

[0003] In bad weather such as rain, snow, sand and dust, and in poor communication conditions, the images captured by intelligent flying vehicles often have problems such as being blocked, image loss, transmission data loss or noise interference. These interferences seriously affect the quality, stability and reliability of the images captured by intelligent flying vehicles.

[0004] Currently, image completion based on flying vehicles such as intelligent flying vehicles mainly extracts features from a single frame of picture captured by the intelligent flying vehicle, analyzes to obtain the contour of the target object, and performs tensor completion inside the target object. This method can effectively handle local occlusion and a small amount of noise on the picture, but has limited processing ability for large-area defects generated under bad weather and poor communication conditions, and is prone to losing some details of the target object, and cannot achieve a good completion effect. Summary of the Invention

[0005] To solve the problems in the background technique, this application proposes an intelligent vehicle image completion method based on image cutting and tensor interconnection completion.

[0006] This application provides an intelligent vehicle image completion method based on image cutting and tensor interconnection completion, which includes the following steps: Step 1: Store the RGB format picture captured by the camera in the form of a tensor as the target tensor. Step 2: Preprocess the prior tensor according to the size and gray value of the current frame tensor, and at the same time use the AIP algorithm to cut the preprocessed prior tensor to generate a reference tensor, and stack the reference tensor and the target tensor into a composite tensor; Step 3: Use the ITSC algorithm to process the composite tensor to obtain the processed tensor after interconnection completion; Step 4: Extract the tensor of the corresponding area of the current image from the processed tensor to obtain the completed tensor, and complete the image completion.

[0007] Preferably, in the step 1, extract the R, G, B pixel values corresponding to each pixel point in the RGB format image, and fill the obtained pixel values into the corresponding layers of the tensor respectively according to R, G, B values to obtain the tensor .

[0008] Preferably, in the step 2, the specific method for preprocessing the prior tensor is as follows: Step 2.1: Perform a geometric transformation on the prior tensor according to the size of the target tensor to make the size of the prior tensor consistent with that of the target tensor; Step 2.2: Adjust the grayscale of the prior tensor according to the size of the target tensor to make the overall brightness of the prior image consistent with that of the target image.

[0009] Preferably, in the step 2, the specific method for generating the reference tensor using the AIP algorithm is as follows: Step 3.1: Obtain the main motion parameters and image parameters of the intelligent flying vehicle; Step 3.2: Determine the number of pixels to be cut from each side of the image according to the obtained parameters, determine a new rectangular area, and perform image pre-cutting on the prior tensor to obtain the pre-cut area ; Step 3.3: Perform semantic segmentation on the pre-cut area and the target tensor, and segment the image into different features , where i is the feature number, to generate two feature sets; Step 3.4: Generate a feature matrix respectively from the position information of a pair of corresponding features in the target tensor and and ; Step 3.5: Calculate the affine matrix Γ according to the two feature matrices and ; Step 3.6: Process the feature matrix in according to the affine matrix Γ to obtain a new feature matrix, and perform an affine transformation on the features in according to the new feature matrix; Step 3.7: Continue to take an unprocessed pair of corresponding features and repeat steps 3.4 to 3.6 until all features in the target tensor are processed; Step 3.8: Complement the pixel points without information in the tensor after the affine transformation with the surrounding pixel points to obtain the reference tensor processed by the AIP algorithm .

[0010] Preferably, the motion parameters include speed, altitude, pitch angle, roll angle, climb rate, acceleration, and turning angle; the image parameters include image resolution, horizontal viewing angle, vertical viewing angle, distance between the intelligent vehicle and the object being photographed, and time interval between frames of the intelligent flying vehicle for photographing.

[0011] Preferably, in the step 3.3, for the pre-cut area The specific method for semantic segmentation with the target tensor is: Step 4.1: Each pixel in the image corresponds to an array of the corresponding x-axis and y-axis positions in the tensor. Therefore, an array determined by the x-axis and y-axis in the tensor is used to represent a point on the image. The point mentioned in the following steps is an array in the tensor. Step 4.2: Set appropriate growth threshold and termination threshold; Step 4.3: Select the center point of the tensor as the initial growth point, create a point set and add the initial growth point to the point set, then repeatedly check the adjacent points of the point set, add the points that meet the growth threshold conditions to the point set, until no points can be added, and this point set is called a feature; Step 4.4: Select the largest area on the graph that is not included in the feature. If this area is larger than the termination segmentation threshold, select the center of this area as the new initial growth point and divide the new features. Repeat this process until the semantic segmentation is completed. All features are collectively called the feature set of the image.

[0012] Preferably, in step 4, the specific method of processing the synthesized tensor using the ITSC algorithm to obtain the processed tensor after interconnection and completion is: Step 5.1: Decompose the tensor into the product of a series of tensor factors, so that the tensor can be optimized by optimizing the tensor factors later; Step 5.2: Establish an error function and an optimization objective function model to measure the error between the tensor decomposition result and the original tensor and iteratively update the tensor factor; Step 5.3: Iteratively update the tensor factors based on the optimization objective function to minimize the difference between the upper and lower parts of the composite tensor; Step 5.4: Combine the optimized tensor factors into a processed tensor with a size of N×M×6.

[0013] Compared with the prior art, the beneficial effects of this application are: The operation of the intelligent flying vehicle itself is stable and controllable, and it is easy to obtain motion parameters such as flight speed. According to these motion parameters, the corresponding area image of the previous frame is cut out as a priori image, and the priori image is used to complete the current frame image. The current frame image is then used as a new priori image, thereby retaining the key information of the missing part of the image and realizing memory-based image completion. This method is superior to the current main intelligent flying vehicle completion methods in terms of stability and effectiveness of image completion. The application of this method can greatly improve the stability and effectiveness of image completion of intelligent flying vehicles in severe weather and poor communication scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying drawings forming a part of this application are used to provide a further understanding of this application. The schematic embodiments of this application and their descriptions are used to explain this application and do not constitute an improper limitation to this application.

[0015] Figure 1 It is a flowchart of image adaptive cutting and complementing for an intelligent flying vehicle with time memory. Figure 2 It is a schematic diagram of a tensor completion model with spatio-temporal correction. Figure 3 It is a schematic diagram of an interconnected tensor decomposition module. Figure 4 It is a schematic diagram of the functions of an iterative module and an update module. Detailed implementation manners

[0016] The following further describes this application in conjunction with the accompanying drawings and embodiments.

[0017] It should be noted that the terms used herein are only for describing specific implementation manners and are not intended to limit the exemplary embodiments according to the present disclosure. As used herein, unless otherwise clearly specified in the context, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0018] In the present disclosure, terms such as "upper", "lower", "left", "right", "front", "rear", "vertical", "horizontal", "side", "bottom", etc. indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only relationship terms determined for facilitating the description of the structural relationship of each component or element of the present disclosure and do not specifically refer to any component or element in the present disclosure and should not be construed as a limitation to the present disclosure.

[0019] Embodiment 1 As Figures 1 to 4 shown, this application provides an intelligent vehicle image completion method based on image cutting and tensor interconnection completion, which is specifically as follows: In the first aspect, this application provides an adaptive image cutting method. It receives the motion parameters of the intelligent vehicle, including flight speed, pitch angle, roll angle, climb rate, acceleration, turning angle, etc. It calculates the moving distance of the image captured by the unmanned aerial vehicle camera using the motion parameters of the intelligent vehicle, converts this distance into the corresponding number of pixels in the image, and continuously cuts out the corresponding area of the previous frame image during the movement of the unmanned aerial vehicle. The cut image can be used as the basis for image completion after being processed, thus completing the adaptive image cutting.

[0020] In a second aspect, the present application proposes an image completion method with memory. The previous frame image of the intelligent vehicle is stored as a prior tensor (with a size of ), which is adaptively cut into a reference tensor (with a size of ). The image of the current frame is transformed into a target tensor , and the reference tensor and the target tensor are stacked into a composite tensor (with a size of ).

[0021] The composite tensor is completed using the tensor interconnection completion algorithm, and the area corresponding to the completed target tensor is extracted to complete the completion of the target tensor. At the same time, this tensor is used as a new prior tensor. After continuously repeating this process, each image completion is affected by multiple previous images, making the algorithm have memory.

[0022] In a third aspect, the present application proposes an interconnection tensor completion algorithm with spatio-temporal factor correction (i.e., the ITSC algorithm), which performs interconnection completion on the composite tensor to obtain a completed image tensor.

[0023] The specific implementation process is as follows: The current frame image captured by the intelligent vehicle is stored as a target tensor. The target tensor and the prior tensor are preprocessed, and the gray value of each point is calculated according to the formula . Based on the gray value, a brightness histogram corresponding to the tensor is generated, and brightness matching and histogram equalization are performed on the target tensor and the prior tensor to balance the brightness distributions of the target tensor and the prior tensor.

[0024] When capturing the current frame image, the current real-time motion parameters of the intelligent vehicle are received, and parameters such as speed, pitch angle, roll angle, climb rate, acceleration, and rotation angle are input into the AIP algorithm, and the AIP algorithm performs identification, cutting, and alignment of the effective area.

[0025] Specifically, the basic motion of the intelligent flying vehicle is divided into the following four types, and it is considered that the motion of the intelligent flying vehicle is these four basic motions or their combinations: Taking the vertical direction as the z-axis, the camera shooting direction is denoted as the x-axis, and the direction perpendicular to the x-axis and the z-axis is the y-axis. The basic motion of the intelligent flying vehicle is defined as: upward motion along the z-axis at a speed of , forward motion along the x-axis at a speed of , rightward motion along the y-axis at a speed of , and clockwise rotation at an angular velocity of in the horizontal direction.

[0026] Pre-cut the prior image. The purpose is to roughly determine the effective area, determine the area for subsequent feature recognition, feature matching, and affine transformation, and reduce the interference of the invalid area. The steps of the image pre-cut are as follows: First, determine the motion state of the intelligent flying vehicle. Based on the motion state of the intelligent flying vehicle and the following parameters, determine the effective region in the prior tensor. The main parameters of the intelligent flying vehicle include the number of pixels H×W of the camera device, the horizontal viewing angle , the vertical viewing angle , the distance d between the intelligent flying vehicle and the object to be photographed, and the time interval between frames captured by the intelligent flying vehicle ; Denote the number of pixels from the effective region to the boundary of the prior tensor as (left side), (right side), (upper side), (lower side), and calculate according to the following steps: Process the motion along the z-axis: Process the motion along the y-axis: Process the motion along the x-axis: (i = u, d, r, l) Process the horizontal rotation with the clockwise direction as positive: Determine a rectangular area with a distance of pixels from each side of the prior tensor, cut out this area, and generate the cut image , completing the pre-cutting of the image.

[0027] For and the target tensor, perform feature recognition according to the following steps: S1: Select the region with the largest unpartitioned feature set in the image, and the center of the selected region is the growth point ; S2: Incorporate the growth point into the feature set (the feature set is a continuous set of points); S3: For the outermost points of the feature set , detect their adjacent points that are not in any feature set, and denote as the corresponding values of the RGB three-color channels of the points respectively. η is the growth threshold. If there are adjacent points that satisfy the formula then incorporate into the feature set ; S4: After completing step S3, if there are newly added points in the previous round, use these points as the ones described in step S3 Repeat step S3 until no points are added to the feature set in one round of detection ; S5: Check the area of the largest unpartitioned feature set region in the image. If the area S of this region satisfies , where is the set minimum partitioning threshold, then the feature recognition of the image is completed. Otherwise, go back to step 1 to determine a new feature set until the area of the largest unpartitioned feature set region is less than the minimum partitioning threshold .

[0028] Perform semantic segmentation on the prior tensor and the target image. First, adjust the numbering of the feature set of the prior tensor using the RGB data of the feature sets of both. Make the feature sets in the prior tensor and the target tensor correspond one by one. For the pixel points in the prior tensor and the target tensor that are not included in the feature set, divide them into the nearest feature set to complete the semantic segmentation of the two images

[0029] For each feature set containing n points , calculate the position of its center point. Denote the center point of the prior tensor feature as , and the center point of the target tensor feature as For each feature set , take all the points to generate a feature matrix. Denote the feature matrix corresponding to the prior tensor as , and the feature matrix corresponding to the target tensor as ; Determine the affine matrix according to the center point and the four corner vertices of the feature set . Denote the four vertices of the feature set in the prior tensor as ([[]] )(k = 1, 2, 3, 4), and the four vertices of the feature set in the target tensor as ([[]] )(k = 1, 2, 3, 4) to obtain the following affine matrix Translation matrix T = Scaling matrix S = Obtain the affine matrix Γ = TS, and use the affine matrix to process the prior, calculate = , Obtain , and project the RGB data of the points in the feature set to the point ([[]] , )([[]] After traversing the points in the corresponding area of the target tensor, if there is a pixel point that does not correspond to the affine transformation of the dataset in the prior tensor, the RGB value of this pixel point is filled with the average value of the adjacent pixel points with existing RGB information, and finally a complete image is generated. 。

[0030] Convert the target tensor and into tensor format. The specific method is as follows: Create a third-order tensor with the lengths of the x-axis and y-axis being the number of pixels in each column and row of the target tensor respectively, and the length of the z-axis being 3. Fill the pixel values of the three colors, red (R), green (G), and blue (B), corresponding to each pixel point of the image into the corresponding positions of the tensor to generate a third-order tensor retaining the RGB data of the image. According to the AIP algorithm, adaptively cut the prior image to obtain the processed reference tensor.

[0031] After stretching, the cut reference tensor is aligned with the target tensor, and they have the same format, image size, similar pixel quality, overall gray level, and color channels.

[0032] Stack the reference tensor and the target tensor into a third-order tensor with the length of the z-axis being 6, and denote this tensor as X. Then bring it into the ITSC model for processing. The specific calculation process is as follows: Initialize the ITSC algorithm model, and set the maximum number of iterations to = 500, and the initial input tensor is the composite tensor , that is . Define the initial ITSC decomposition rank as follows: The initial values of the auxiliary matrices are as follows As Figure 1 and Figure 3 shown, decompose the tensor into the product of a series of tensor factors using the ITSC interconnection decomposition module: Calculate the error of the tensor decomposition using the ITSC decomposition model and optimize the objective function expressed as Add a regularization term to screen the data. , where the set is defined as Add a spatial correction term To ensure the spatial correlation during the completion of tensors at the same layer As Figure 3 shown, the prior image is used to correct the damaged part of the target tensor to minimize the difference between the reference tensor and the target tensor When the iteration number i < , if Equation (9) is not satisfied, the model is iteratively corrected once, and the specific process is as follows: Update R = , If Equation (10) is satisfied Expand , Update through formula (11) , where Update through Equation (13) , Repeat the iterative process until Equation (9) is satisfied, and output the completed tensor .

[0033] As Figure 1 shown, the obtained completed tensor is the completed intelligent flying vehicle image. Store it as a new prior tensor to participate in the next round of tensor completion.

[0034] As Figures 2 to 4 shown, the core part of the interconnected tensor completion algorithm includes: Decomposition module: Decompose the tensor into a series of tensor factors and calculate the set of local combined tensors; Iteration module: Used to correct the set of decomposed tensor factors during iterative calculation, optimize the correlation between tensor decomposition factors using the interconnected model, and ensure the correlation between the reference tensor and the target tensor using the spatio-temporal correction factor; Update module: Update , R, .

[0035] where Figure 3 is the schematic diagram of the interconnected tensor decomposition module, Figure 4 is the schematic diagram of the functions of the iteration module and the update module.

[0036] In summary, in this application, the memory-based completion of the images of intelligent flying vehicles is achieved. The implementation method of this application retains the information of previous images in the prior tensor, and processes the target tensor corresponding to the current image frame by frame using the prior tensor, enabling the damaged or occluded parts in the current image to be complemented with information obtained from multiple past images.

[0037] The application of the Adaptive Image Partitioning (AIP) algorithm ensures that while retaining the image features, it can effectively match the corresponding features in the prior tensor and the target tensor corresponding to the current image, improving the versatility and effectiveness of image completion.

[0038] This image completion method is particularly suitable for complementing damaged points in consecutive images in combination with the motion data of intelligent flying vehicles. By retaining the information of multiple frames of images, the damaged or occluded parts in each frame of the image can be restored, and it has a good effect on temporarily occurring image damage and image occlusion.

[0039] The Iterative Tensor Self-Completion (ITSC) algorithm is another important innovation of this application. By splitting the tensor factors, it enables the tensor itself to be iteratively updated and continuously optimized, minimizing the difference between the reference tensor and the target tensor, and applying the stored memory-based image information to the current completed image to the greatest extent.

[0040] The above are only the preferred embodiments of this application and are not used to limit this application. For those skilled in the art, various changes and modifications can be made to this application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this application shall be included within the protection scope of this application.

[0041] Although the specific implementation manners of this application are described above in conjunction with the accompanying drawings, it is not a limitation on the protection scope of this application. Those skilled in the art should understand that based on the technical solutions of this application, various modifications or deformations that can be made without creative efforts by those skilled in the art are still within the protection scope of this application.

Claims

1. An intelligent vehicle image completion method based on image segmentation and tensor interconnection completion, characterized in that It includes the following steps: Step 1: Store the RGB format image captured by the camera in the form of a tensor as the target tensor; Step 2: According to the size and grayscale value of the current frame tensor, preprocess the prior tensor. Meanwhile, use the AIP algorithm to cut the preprocessed prior tensor to generate a reference tensor, and stack the reference tensor and the target tensor into a synthetic tensor; Step 3: Use the ITSC algorithm to process the synthetic tensor to obtain a processed tensor after interconnection and completion; Step 4: Extract the tensor of the corresponding area of the current image from the processed tensor to obtain a completed tensor, and complete the image completion.

2. The intelligent vehicle image completion method based on image cutting and tensor interconnection and completion according to claim 1, characterized in that: In the step 1, the R, G, and B pixel values corresponding to each pixel point in the RGB format image are extracted, and the obtained pixel values are filled into the corresponding layers of the tensor according to the R, G, and B values respectively to obtain the tensor .

3. The intelligent vehicle image completion method based on image cutting and tensor interconnection and completion according to claim 1, characterized in that: In the said Step 2, the specific method for preprocessing the prior tensor is: Adjust the grayscale of the prior tensor according to the size of the target tensor to make the overall brightness of the prior image consistent with that of the target image.

4. The intelligent vehicle image completion method based on image cutting and tensor interconnection and completion according to claim 1, characterized in that: In the said Step 2, the specific method for generating the reference tensor by using the AIP algorithm is: Step 3.1: Obtain the main motion parameters and image parameters of the intelligent flying vehicle; Step 3.2: Determine the number of pixels for image cutting from each side according to the obtained parameters, determine a new rectangular area, and perform image pre-cutting on the prior tensor to obtain the pre-cut area ; Step 3.3: For the pre-cut area and the target tensor, perform semantic segmentation to divide the image into different features , where i is the feature number, and generate two feature sets; Step 3.4: Respectively generate a feature matrix from the position information of a pair of corresponding features in the target tensor and the corresponding feature set and ; Step 3.5: Calculate the affine matrix Γ based on the two feature matrices and ; Step 3.6: According to the affine matrix Γ, in the feature matrix is processed to obtain a new feature matrix, and the features in are subjected to an affine transformation according to the new feature matrix; Step 3.7: Continue to take an unprocessed pair of corresponding features and repeat Steps 3.4 to 3.6 until all features in the target tensor are processed; Step 3.8: Complete the pixel points without information in the tensor after the affine transformation using the surrounding pixel points to obtain the reference tensor processed by the AIP algorithm .

5. The intelligent vehicle image completion method based on image cutting and tensor interconnection and completion according to claim 4, characterized in that: The said motion parameters include speed, altitude, pitch angle, roll angle, climb rate, acceleration, turning angle; The said image parameters include image resolution, horizontal view angle, vertical view angle, distance between the intelligent vehicle and the object being photographed, time interval between frames taken by the intelligent flying vehicle.

6. The intelligent vehicle image completion method based on image cutting and tensor interconnection and completion according to claim 4, characterized in that: In step 3.3, for the pre-cut area and the specific method for performing semantic segmentation on the target tensor is as follows: Step 4.1: Each pixel point in the image corresponds to an array at the corresponding x-axis and y-axis positions in the tensor. Therefore, an array determined by the x-axis and y-axis in the tensor is used to represent a point on the graph, and the points mentioned in the following steps are arrays in the tensor; Step 4.2: Set appropriate growth threshold and termination division threshold; Step 4.3: Select the center point of the tensor as the initial growth point, create a point set and add the initial growth point to the point set. Then repeatedly check the points adjacent to the point set, and add the points that meet the growth threshold condition to the point set until no point can be added. This point set is called a feature; Step 4.4: Select the largest area on the graph that has not been divided into features. If this area is larger than the termination division threshold, select the center of this area as the new initial growth point and divide a new feature. Repeat this process until semantic segmentation is completed, and all features are collectively called the feature set of the image.

7. The intelligent vehicle image completion method based on image cutting and tensor interconnection and completion according to claim 1, characterized in that: In the said step 4, the specific method for processing the synthesized tensor by using the ITSC algorithm to obtain the processed tensor after interconnection completion is as follows: Step 5.1: Decompose the tensor into the product of a series of tensor factors to facilitate subsequent optimization of the tensor by optimizing the tensor factors. Step 5.2: Establish an error function and an optimization objective function model to measure the error between the tensor decomposition result and the original tensor and iteratively update the tensor factors. Step 5.3: Based on the optimization objective function, iteratively update the tensor factors to minimize the difference between the upper and lower parts of the synthesized tensor. Step 5.4: Synthesize the optimized tensor factors into the processed tensor, and its size is N×M×6.

Images