Image rectangularization method and device based on deformable grid and width residual network

Abstract

The present invention proposes a method and device for rectangularizing an image based on a deformable grid and a width residual network, and relates to the field of computer image processing. The number of rigid structures of an irregular image is calculated by a convolutional neural prediction network. A suitable grid structure is selected according to the prediction and a predefined target grid is generated. The irregular image, the predefined target grid and the predicted initial grid obtained by using the prediction network model are input into the width residual neural network for rectangularization regression. The loss function includes local and global losses, which control the information within the grid and the global information respectively. The generated rectangularized image can not only reduce the information loss in the image deformation, but also can be adaptively adjusted according to different actual input image structures, thereby further improving the effect of image rectangularization. The processed image is natural and meets the needs of practical applications. Provide technical support for video splicing correction in industrial production scenes such as coal mine comprehensive mining working faces. It has high practicality.

Description

Technical Field

[0001] The invention relates to the field of computer image processing, in particular to an image rectangularization method and an image rectangularization device based on a deformable grid and a width residual network. Background Art

[0002] Image processing technology has become one of the core technologies in the fields of computer vision and artificial intelligence. However, when processing irregularly shaped images, traditional image processing methods often face many difficulties. For example, traditional methods such as direct cropping or image completion usually cause a large amount of image information loss, making it impossible to accurately reflect the information of the real scene, resulting in unsatisfactory image rectangularization effect. These challenges make it difficult to rectangularize irregular images in practical applications, affecting the comprehensive application of image processing technology in human production and life.

[0003] At present, the rise of deep learning technology has brought new possibilities to the field of image processing. However, there are still some technical bottlenecks for the rectangularization of irregular shaped images. On the one hand, traditional deep learning methods often ignore the rigid structure inside the image, which makes it difficult to accurately capture the internal features of images with variable shapes. On the other hand, some past methods, while pursuing the rectangularization of images, are prone to cause the loss of global information, resulting in unnatural processed images that are difficult to meet the needs of practical applications. Summary of the invention

[0004] In view of the above problems, the present invention proposes an image rectangularization method and an image rectangularization device based on a deformable grid and a width residual network.

[0005] An embodiment of the present invention provides an image rectangularization method based on a deformable grid and a width residual network, and the image rectangularization method includes:

[0006] Predicting the number and direction of rigid structures of irregular-shaped images through a convolutional neural prediction network, selecting a suitable grid structure in a deformable grid structure according to the prediction result and generating a predefined target grid, wherein the deformable grid structure includes: a triangular grid, a rectangular grid or a regular hexagonal grid;

[0007] The prediction network model generates a prediction initialization grid according to the irregular shape image and its mask matrix, wherein the prediction network model is trained based on an irregular spliced ​​image dataset;

[0008] The irregular shape image, the predicted initialization grid and the predefined target grid are input into the width residual neural network to perform image rectangularization iterative regression training, wherein the loss function of the width residual network in the image rectangularization iterative regression training process includes: a local loss function and a global loss function, the local loss function is used to control the deformation loss of the target in the grid, and the global correlation loss function is used to avoid the global information loss caused in the deformation process;

[0009] When the loss function is iteratively reduced to a minimum, the training is stopped and the current result is output as the final result of the image rectangularization, so that the irregular shape image after image splicing can be rectangularized and corrected.

[0010] Optionally, the convolutional neural prediction network includes: YOLO or DarkNet.

[0011] Optionally, selecting a suitable grid structure in the deformable grid structure according to the prediction result includes:

[0012] Selection is made according to the direction and quantity of the rigid structures;

[0013] If there are more horizontal or vertical structures in the irregular shape image, the rectangular grid is selected;

[0014] If there are many irregular-shaped structures in the irregular-shaped image, the triangular mesh is selected;

[0015] If the horizontal or vertical structure in the irregular shape image is in a similar proportion to the irregular shape structure, the regular hexagonal grid is selected.

[0016] Optionally, the prediction network model is obtained by training the irregular splicing image dataset using a width residual neural network, and the irregular splicing image dataset includes: a large number of spliced ​​images of irregular shapes;

[0017] The trained prediction network model generates the prediction initialization grid by content-awareness of the irregular shape image and its mask matrix.

[0018] Optionally, the irregular shape image, the predicted initialization grid and the predefined target grid are input into a width residual neural network to perform image rectangularization iterative regression training, including:

[0019] Configuring weights for the local loss function and the global loss function respectively to adjust the influence of the two loss functions on the iterative regression training of image rectangularization;

[0020] If the weight of the local loss function is greater than the weight of the global loss function, the image rectangularization iterative regression training is performed with the main goal of controlling the deformation loss of the object in the grid to be smaller;

[0021] If the weight of the local loss function is less than the weight of the global loss function, the image rectangularization iterative regression training is performed with the main goal of avoiding the global information loss caused during the deformation process;

[0022] If the weight of the local loss function is equal to the weight of the global loss function, the deformation loss and the global information loss are balanced, and image rectangularization iterative regression training is performed.

[0023] An embodiment of the present invention provides an image rectangularization device based on a deformable grid and a width residual network, the image rectangularization device comprising:

[0024] A prediction and selection module is used to predict the number and direction of rigid structures of irregular-shaped images through a convolutional neural prediction network, select a suitable grid structure from a deformable grid structure according to the prediction result, and generate a predefined target grid. The deformable grid structure includes: a triangular grid, a rectangular grid, or a regular hexagonal grid;

[0025] An initialization module, used to generate a prediction initialization grid according to the irregular shape image and its mask matrix using a prediction network model, wherein the prediction network model is trained based on an irregular spliced ​​image data set;

[0026] A rectangularization iterative regression training module is used to input the irregular shape image, the predicted initialization grid and the predefined target grid into a width residual neural network to perform image rectangularization iterative regression training, wherein the loss function of the width residual network in the image rectangularization iterative regression training process includes: a local loss function and a global loss function, the local loss function is used to control the deformation loss of the target in the grid, and the global correlation loss function is used to avoid global information loss caused during the deformation process;

[0027] The output module is used to stop training when the loss function is iteratively reduced to a minimum, and output the current result as the final result of image rectangularization, so that the irregular shape image after image splicing can be rectangularized and corrected.

[0028] Optionally, the convolutional neural prediction network used by the prediction and selection module includes: YOLO or DarkNet.

[0029] Optionally, the prediction and selection module is specifically used for:

[0030] Selection is made according to the direction and quantity of the rigid structures;

[0031] If there are more horizontal or vertical structures in the irregular shape image, the rectangular grid is selected;

[0032] If there are many irregular-shaped structures in the irregular-shaped image, the triangular mesh is selected;

[0033] If the horizontal or vertical structure in the irregular shape image is in a similar proportion to the irregular shape structure, the regular hexagonal grid is selected.

[0034] Optionally, the prediction network model in the initialization module is obtained by training the irregular splicing image dataset using a width residual neural network, and the irregular splicing image dataset includes: a massive amount of spliced ​​images of irregular shapes;

[0035] The initialization module generates the prediction initialization grid by using the trained prediction network model to perceive the content of the irregular shape image and its mask matrix.

[0036] Optionally, the rectangular iterative regression training module includes:

[0037] A weight submodule, used to configure weights for the local loss function and the global loss function respectively, so as to adjust the influence of the two loss functions on the iterative regression training of image rectangularization;

[0038] A regression training submodule, for performing iterative regression training of image rectangularization with the main goal of controlling the deformation loss of the object in the grid to be smaller if the weight of the local loss function is greater than the weight of the global loss function;

[0039] The regression training submodule is further used for performing iterative regression training of image rectangularization with the main goal of avoiding global information loss caused during the deformation process if the weight of the local loss function is less than the weight of the global loss function;

[0040] The regression training submodule is also used to balance the deformation loss and the global information loss if the weight of the local loss function is equal to the weight of the global loss function, and perform iterative regression training for image rectangularization.

[0041] An embodiment of the present invention provides an electronic device, the electronic device comprising:

[0042] one or more processors; and

[0043] One or more machine-readable media having instructions stored thereon, when executed by the one or more processors, enable the electronic device to perform the image rectangularization method based on a deformable grid and a width residual network as described in any of the above items.

[0044] The image rectangularization method based on a deformable grid and a width residual network provided by the present invention first predicts the number and direction of rigid structures of an irregular shape image through a convolutional neural prediction network, selects a suitable grid structure in the deformable grid structure according to the prediction result, and generates a predefined target grid. Then, a prediction network model is used to generate a prediction initialization grid according to the irregular shape image and its mask matrix, and then the irregular shape image, the prediction initialization grid and the predefined target grid are input into the width residual neural network together to perform iterative regression training for image rectangularization. Finally, when the loss function iteration is reduced to the minimum, the training is stopped, and the current result is output as the final result of the image rectangularization, so that the irregular shape image after image splicing can be rectangularized and corrected.

[0045] The image rectangularization method based on deformable grid and width residual network proposed in the present invention introduces a method combining convolutional neural prediction network and width residual neural network. The convolutional neural prediction network predicts the number and direction of rigid structures of the image, selects the most suitable grid structure for the current image for rectangularization regression, and provides an effective basis for subsequent rectangularization processing.

[0046] At the same time, the width residual neural network is used to perform content-aware training on irregular spliced ​​images, which improves the adaptability to irregular shapes. In the process of rectangularization, the deformation of the target in the grid is controlled by the local loss function, and the loss of global information is avoided by the global correlation loss function. After the rectangularization process, the image maintains the rigidity of the internal structure and retains the global information, achieving a more ideal rectangularization effect. It provides new technical support for the development of rectangularization of irregular images and has strong practicality. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0048] Figure 1 It is a flow chart of a method for rectangularizing an image based on a deformable grid and a width residual network according to an embodiment of the present invention;

[0049] Figure 2 is an exemplary schematic diagram of a main process of inputting an irregular image and generating an output rectangular image in an embodiment of the present invention;

[0050] Figure 3 It is a block diagram of an image rectangularization device based on a deformable grid and a width residual network in an embodiment of the present invention. DETAILED DESCRIPTION

[0051] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention, are only part of the embodiments of the present invention, not all of the embodiments, and are not used to limit the present invention.

[0052] An image rectangularization method based on a deformable grid and a width residual network according to an embodiment of the present invention, referring to Figure 1 As shown in the flowchart, the image rectangularization method includes:

[0053] Step 101: predict the number and direction of rigid structures of the irregular shape image through a convolutional neural prediction network, select a suitable grid structure from a deformable grid structure according to the prediction result and generate a predefined target grid. The deformable grid structure includes: a triangular grid, a rectangular grid or a regular hexagonal grid.

[0054] The image rectangularization method based on deformable grid and width residual network proposed in the present invention first predicts the number and direction of rigid structures of irregular shape images through convolutional neural prediction network, selects appropriate grid structure in deformable grid structure and generates predefined target grid according to the prediction results of the number and direction of rigid structures. Among them, the deformable grid structure includes: triangular grid, rectangular grid or regular hexagonal grid, etc.

[0055] The convolutional neural prediction network may adopt a known convolutional neural prediction network, such as YOLO or DarkNet.

[0056] A better method for selecting a suitable grid structure in a deformable grid structure according to the prediction results includes:

[0057] The selection is made based on the direction and number of rigid structures; if there are more horizontal or vertical structures in the irregular shape image, a rectangular grid is selected; if there are more irregular shape structures in the irregular shape image, a triangular grid is selected; if the ratio of horizontal or vertical structures to irregular shape structures in the irregular shape image is equal, a regular hexagonal grid is selected.

[0058] Step 102: The prediction network model generates a prediction initialization grid according to the irregular shape image and its mask matrix, wherein the prediction network model is trained based on an irregular spliced ​​image dataset.

[0059] After the predefined target grid is selected, a prediction network model is used to generate a prediction initialization grid according to the irregular shape image and its mask matrix, wherein the prediction network model is trained based on an irregular spliced ​​image dataset.

[0060] Preferably, the prediction network model can be obtained by training an irregular splicing image dataset using a width residual neural network, wherein the irregular splicing image dataset includes: a massive amount of splicing images of irregular shapes. This can ensure the accuracy of the prediction network model in generating the prediction initialization grid.

[0061] After the prediction network model is trained, the prediction initialization grid can be generated by using the trained prediction network model to perceive the content of the irregular shape image and its mask matrix.

[0062] Step 103: The irregular shape image, the predicted initialization grid and the predefined target grid are input into the width residual neural network to perform iterative regression training for image rectangularization, wherein the loss function of the width residual network in the iterative regression training process for image rectangularization includes: a local loss function and a global loss function, the local loss function is used to control the deformation loss of the target in the grid, and the global correlation loss function is used to avoid the global information loss caused by the deformation process.

[0063] After the predefined target grid and the predicted initialization grid are obtained in steps 101 to 102, the irregular shape image, the predicted initialization grid and the predefined target grid are input into the width residual neural network to perform iterative regression training for image rectangularization.

[0064] Among them, the width residual network needs to perform regression training based on the loss function in the iterative regression training process of image rectangularization. The loss function includes: local loss function and global loss function. The local loss function is used to control the deformation loss of the target in the grid, and the global correlation loss function is used to avoid the global information loss caused by the deformation process.

[0065] That is to say, through the iterative regression training of image rectangularization, the rigidity of the internal structure of the irregular shaped image is maintained, and its global information is retained, thus achieving a more ideal rectangularization effect.

[0066] A better method for iterative regression training of image rectangularization includes:

[0067] First, weights are configured for the local loss function and the global loss function respectively. The purpose of the weights is to adjust the impact of the two loss functions on the iterative regression training of image rectangularization.

[0068] If the weight of the local loss function is greater than the weight of the global loss function, the image rectangularization iterative regression training is performed with the main goal of controlling the deformation loss of the target in the grid smaller; if the weight of the local loss function is less than the weight of the global loss function, the image rectangularization iterative regression training is performed with the main goal of avoiding the global information loss caused by the deformation process smaller; if the weight of the local loss function is equal to the weight of the global loss function, the deformation loss and the global information loss are balanced, and the image rectangularization iterative regression training is performed. The specific weight setting can be determined according to actual needs. If you want the deformation loss of the final output image to be smaller, set the weight of the local loss function greater than the weight of the global loss function. If you want the global information loss of the final output image to be smaller, set the weight of the local loss function less than the weight of the global loss function. If you want to balance the deformation loss and the global information loss of the final output image, set the weight of the local loss function equal to the weight of the global loss function.

[0069] Step 104: When the loss function iteration is reduced to the minimum, the training is stopped and the current result is output as the final result of the image rectangularization, so that the irregular shape image after image splicing can be rectangularized and corrected.

[0070] During the iterative regression training of image rectangularization, when the loss function iteration is reduced to the minimum, that is, when the iteration of both the local loss function and the global loss function is reduced to the minimum, the training is stopped, and the current result is output as the final result of image rectangularization. The final output image is the image obtained by splicing irregular shaped images and performing rectangularization correction, achieving a more ideal rectangularization effect. The processed image is natural and meets the needs of practical applications.

[0071] In order to better explain and illustrate the above-mentioned image rectangularization method based on deformable grid and width residual network, refer to Figure 2 The figure shows an exemplary schematic diagram of the main process of generating an output rectangular image from an input irregular image.

[0072] Figure 2 201 represents an input irregular image (i.e., an irregular shape image); 202 represents selecting a suitable grid structure in a deformable grid structure, and 202 exemplarily shows a triangular grid, a rectangular grid, and a regular hexagonal grid. 203 represents a generated predefined target grid, and it can be known that a regular hexagonal grid is selected as the predefined target grid according to the prediction result.

[0073] 205 indicates that it is also necessary to use the prediction network model to generate a prediction initialization grid according to the irregular shape image and its mask matrix. It can be seen that the hexagon of the prediction initialization grid has undergone irregular changes relative to the predefined target grid. It is no longer a regular hexagon and fits the irregular shape image more accurately.

[0074] 204 represents a width residual neural network, which receives the irregular shape image of 201, the predicted initialization grid of 205 and the predefined target grid of 203 as common inputs, and performs iterative regression training for image rectangularization. 204 exemplarily represents a method for performing iterative regression training for image rectangularization using a width residual neural network.

[0075] 206 represents the output result after the image rectangularization iterative regression training of the 204 width residual neural network. The output result is the result of the image rectangularization regression. It can be seen that the irregular shape image 201 after image splicing is rectangularized and corrected, and a more ideal rectangularization effect is obtained. The processed image is natural and meets the needs of practical applications.

[0076] Based on the above-mentioned image rectangularization method based on a deformable grid and a width residual network, an embodiment of the present invention further provides an image rectangularization device based on a deformable grid and a width residual network, referring to Figure 3 As shown, the image rectangularization device includes:

[0077] A prediction and selection module 310 is used to predict the number and direction of rigid structures of irregular-shaped images through a convolutional neural prediction network, select a suitable grid structure from a deformable grid structure according to the prediction result, and generate a predefined target grid, wherein the deformable grid structure includes: a triangular grid, a rectangular grid, or a regular hexagonal grid;

[0078] An initialization module 320, configured to generate a prediction initialization grid according to the irregular shape image and its mask matrix using a prediction network model, wherein the prediction network model is trained based on an irregular spliced ​​image dataset;

[0079] A rectangularization iterative regression training module 330 is used to input the irregular shape image, the predicted initialization grid and the predefined target grid into a width residual neural network to perform image rectangularization iterative regression training, wherein the loss function of the width residual network during the image rectangularization iterative regression training includes: a local loss function and a global loss function, wherein the local loss function is used to control the deformation loss of the target in the grid, and the global correlation loss function is used to avoid global information loss caused during the deformation process;

[0080] The output module 340 is used to stop training when the loss function is iteratively reduced to a minimum, and output the current result as the final result of the image rectangularization, so that the irregular shape image after image splicing can be rectangularized and corrected.

[0081] Optionally, the convolutional neural prediction network used by the prediction and selection module 310 includes: YOLO or DarkNet.

[0082] Optionally, the prediction selection module 310 is specifically used for:

[0083] Selection is made according to the direction and quantity of the rigid structures;

[0084] If there are more horizontal or vertical structures in the irregular shape image, the rectangular grid is selected;

[0085] If there are many irregular-shaped structures in the irregular-shaped image, the triangular mesh is selected;

[0086] If the horizontal or vertical structure in the irregular shape image is in a similar proportion to the irregular shape structure, the regular hexagonal grid is selected.

[0087] Optionally, the prediction network model in the initialization module 320 is obtained by training the irregular spliced ​​image dataset using a width residual neural network, and the irregular spliced ​​image dataset includes: a massive amount of spliced ​​images of irregular shapes;

[0088] The initialization module 320 generates the prediction initialization grid by using the trained prediction network model to perceive the content of the irregular shape image and its mask matrix.

[0089] Optionally, the rectangularization iterative regression training module 330 includes:

[0090] A weight submodule, used to configure weights for the local loss function and the global loss function respectively, so as to adjust the influence of the two loss functions on the iterative regression training of image rectangularization;

[0091] A regression training submodule, for performing iterative regression training of image rectangularization with the main goal of controlling the deformation loss of the object in the grid to be smaller if the weight of the local loss function is greater than the weight of the global loss function;

[0092] The regression training submodule is further used for performing iterative regression training of image rectangularization with the main goal of avoiding global information loss caused during the deformation process if the weight of the local loss function is less than the weight of the global loss function;

[0093] The regression training submodule is also used to balance the deformation loss and the global information loss if the weight of the local loss function is equal to the weight of the global loss function, and perform iterative regression training for image rectangularization.

[0094] Based on the above-mentioned image rectangularization method based on a deformable grid and a width residual network, an embodiment of the present invention further provides an electronic device, the electronic device comprising:

[0095] one or more processors; and

[0096] One or more machine-readable media having instructions stored thereon, when executed by the one or more processors, enable the electronic device to perform the image rectangularization method based on the deformable grid and the width residual network as described in the above steps 101 to 104.

[0097] Through the above description, the image rectangularization method based on the deformable grid and the width residual network provided by the present invention first predicts the number and direction of rigid structures of the irregular shape image through the convolutional neural prediction network, selects a suitable grid structure in the deformable grid structure according to the prediction result and generates a predefined target grid. After that, the prediction network model is used to generate a prediction initialization grid according to the irregular shape image and its mask matrix, and then the irregular shape image, the prediction initialization grid and the predefined target grid are input into the width residual neural network together to perform iterative regression training for image rectangularization. Finally, when the loss function iteration is reduced to the minimum, the training is stopped, and the current result is output as the final result of the image rectangularization, so that the irregular shape image after image splicing can be rectangularized and corrected.

[0098] The image rectangularization method based on deformable grid and width residual network proposed in the present invention introduces a method combining convolutional neural prediction network and width residual neural network. The convolutional neural prediction network predicts the number and direction of rigid structures of the image, selects the most suitable grid structure for the current image for rectangularization regression, and provides an effective basis for subsequent rectangularization processing.

[0099] At the same time, the width residual neural network is used to perform content-aware training on irregular spliced ​​images, which improves the adaptability to irregular shapes. In the process of rectangularization, the deformation of the target in the grid is controlled by the local loss function, and the loss of global information is avoided by the global correlation loss function. After the rectangularization, the image maintains the rigidity of the internal structure and the global information, achieving a more ideal rectangularization effect. The processed image is natural and meets the needs of practical applications. It provides new technical support for the development of rectangularization of irregular images and has strong practicality.

[0100] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0101] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0102] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation modes, which are merely illustrative rather than restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are within the protection of the present invention.

Claims

1. A method for rectangularizing an image based on a deformable grid and a width residual network, characterized in that: The image rectangularization method comprises: Predicting the number and direction of rigid structures of irregular-shaped images through a convolutional neural prediction network, selecting a suitable grid structure in a deformable grid structure according to the prediction result and generating a predefined target grid, wherein the deformable grid structure includes: a triangular grid, a rectangular grid or a regular hexagonal grid; The prediction network model generates a prediction initialization grid according to the irregular shape image and its mask matrix, wherein the prediction network model is trained based on an irregular spliced ​​image dataset; The irregular shape image, the predicted initialization grid and the predefined target grid are input into the width residual neural network to perform image rectangularization iterative regression training, wherein the loss function of the width residual network in the image rectangularization iterative regression training process includes: a local loss function and a global loss function, the local loss function is used to control the deformation loss of the target in the grid, and the global loss function is used to avoid the global information loss caused in the deformation process; When the loss function is iteratively reduced to a minimum, the training is stopped and the current result is output as the final result of the image rectangularization, so that the irregular shape image after image splicing can be rectangularized and corrected.

2. The image rectangularization method according to claim 1, characterized in that: The convolutional neural prediction network includes: YOLO or DarkNet.

3. The image rectangularization method according to claim 1, characterized in that: According to the prediction results, a suitable grid structure is selected from the deformable grid structure, including: Selection is made according to the direction and quantity of the rigid structures; If the irregular shape image has more horizontal or vertical structures than irregular shape structures, then selecting the rectangular grid; If the irregular shape image has more irregular shape structures than horizontal or vertical structures, then selecting the triangular grid; If the horizontal or vertical structure in the irregular shape image is in a similar proportion to the irregular shape structure, the regular hexagonal grid is selected.

4. The image rectangularization method according to claim 1, characterized in that: The prediction network model is obtained by training the irregular splicing image data set using a width residual neural network, and the irregular splicing image data set includes: a splicing image of a large number of irregularly shaped images; The trained prediction network model generates the prediction initialization grid by content-awareness of the irregular shape image and its mask matrix.

5. The image rectangularization method according to claim 1, characterized in that: The irregular shape image, the predicted initialization grid and the predefined target grid are input into a width residual neural network to perform image rectangularization iterative regression training, including: Configuring weights for the local loss function and the global loss function respectively to adjust the influence of the two loss functions on the iterative regression training of image rectangularization; If the weight of the local loss function is greater than the weight of the global loss function, the image rectangularization iterative regression training is performed with the main goal of controlling the deformation loss of the object in the grid to be smaller; If the weight of the local loss function is less than the weight of the global loss function, the image rectangularization iterative regression training is performed with the main goal of avoiding the global information loss caused during the deformation process; If the weight of the local loss function is equal to the weight of the global loss function, the deformation loss and the global information loss are balanced, and image rectangularization iterative regression training is performed.

6. An image rectangularization device based on a deformable grid and a width residual network, characterized in that: The image rectangularization device comprises: A prediction and selection module is used to predict the number and direction of rigid structures of irregular-shaped images through a convolutional neural prediction network, select a suitable grid structure from a deformable grid structure according to the prediction result, and generate a predefined target grid. The deformable grid structure includes: a triangular grid, a rectangular grid, or a regular hexagonal grid; An initialization module, used to generate a prediction initialization grid according to the irregular shape image and its mask matrix using a prediction network model, wherein the prediction network model is trained based on an irregular spliced ​​image data set; A rectangularization iterative regression training module is used to input the irregular shape image, the predicted initialization grid and the predefined target grid into a width residual neural network to perform image rectangularization iterative regression training, wherein the loss function of the width residual network in the image rectangularization iterative regression training process includes: a local loss function and a global loss function, the local loss function is used to control the deformation loss of the target in the grid, and the global loss function is used to avoid global information loss caused during the deformation process; The output module is used to stop training when the loss function is iteratively reduced to a minimum, and output the current result as the final result of image rectangularization, so that the irregular shape image after image splicing can be rectangularized and corrected.

7. The image rectangularization device according to claim 6, characterized in that: The convolutional neural prediction network used by the prediction and selection module includes: YOLO or DarkNet.

8. The image rectangularization device according to claim 6, characterized in that: The prediction and selection module is specifically used for: Selection is made according to the direction and quantity of the rigid structures; If the irregular shape image has more horizontal or vertical structures than irregular shape structures, then selecting the rectangular grid; If the irregular shape image has more irregular shape structures than horizontal or vertical structures, then selecting the triangular grid; If the horizontal or vertical structure in the irregular shape image is in a similar proportion to the irregular shape structure, the regular hexagonal grid is selected.

9. The image rectangularization device according to claim 6, characterized in that: The prediction network model in the initialization module is obtained by training the irregular splicing image data set using a width residual neural network, and the irregular splicing image data set includes: a massive amount of spliced ​​images of irregular shapes; The initialization module generates the prediction initialization grid by using the trained prediction network model to perceive the content of the irregular shape image and its mask matrix.

10. An electronic device, comprising: one or more processors; and One or more machine-readable media having instructions stored thereon, when the instructions are executed by the one or more processors, the electronic device executes the image rectangularization method based on a deformable grid and a width residual network as described in any one of claims 1 to 5.

Images