Cabinet image correction method and device, computer equipment, readable storage medium and program product

Abstract

The invention relates to a cabinet image correction method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: inputting a cabinet image into a trained semantic segmentation model to obtain a segmentation prediction result output by the semantic segmentation model; extracting a cabinet contour point set from the cabinet image according to a segmentation prediction result; performing contour simplification processing on the cabinet contour point set to obtain cabinet vertex coordinates of the cabinet image; and performing perspective transformation on the cabinet image according to the cabinet vertex coordinates to obtain a corrected cabinet image. By adopting the method, the cabinet image can be corrected more accurately.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a cabinet image correction method, device, computer equipment, computer-readable storage medium and computer program product. Background Art

[0002] Traditional equipment management in computer rooms usually relies on manual equipment inventory, which is cumbersome and time-consuming. With the development of digital twin technology, image processing technology can be combined to simulate and model the actual computer room environment using images as a medium, so as to achieve more efficient management and maintenance of computer room equipment.

[0003] However, in actual production, when photographing a cabinet in a narrow computer room environment, it is often necessary to photograph the cabinet from overhead, side, and other angles to obtain a global view of the cabinet, which often causes the captured cabinet image to have obvious deformation. Summary of the invention

[0004] Based on this, it is necessary to provide a cabinet image correction method, device, computer equipment, computer readable storage medium and computer program product to address the above technical problems.

[0005] In a first aspect, the present application provides a cabinet image correction method, comprising:

[0006] Inputting the cabinet image into a trained semantic segmentation model to obtain a segmentation prediction result output by the semantic segmentation model;

[0007] Extracting a cabinet contour point set in the cabinet image according to the segmentation prediction result;

[0008] Performing contour simplification processing on the cabinet contour point set to obtain cabinet vertex coordinates of the cabinet image;

[0009] The cabinet image is perspective transformed according to the cabinet vertex coordinates to obtain a corrected cabinet image.

[0010] In one of the embodiments, extracting a cabinet contour point set from the cabinet image based on the segmentation prediction result includes: binarizing the cabinet image based on the segmentation prediction result; performing edge detection on the binarized cabinet image to obtain an edge marked image; and performing contour detection on the edge marked image to obtain the cabinet contour point set.

[0011] In one of the embodiments, the cabinet contour point set is subjected to contour simplification processing to obtain cabinet vertex coordinates of the cabinet image, including: using a convex hull algorithm to calculate the minimum convex polygon of the cabinet contour point set, and obtaining a first cabinet contour based on the minimum convex polygon; using a polygon approximation algorithm to process the first cabinet contour to obtain a second cabinet contour of the cabinet contour; and obtaining the cabinet vertex coordinates based on the vertices of the second cabinet contour.

[0012] In one of the embodiments, the perspective transformation of the cabinet image according to the cabinet vertex coordinates to obtain a corrected cabinet image includes: obtaining target vertex coordinates corresponding to each cabinet vertex coordinate in a target plane; obtaining a transformation matrix between the cabinet image and the target plane according to each cabinet vertex coordinate and the corresponding target vertex coordinate; and using the transformation matrix to transform each pixel of the cabinet image to the target plane to obtain the corrected cabinet image.

[0013] In one of the embodiments, the method further comprises: if the corrected cabinet image includes a vacant pixel, obtaining the pixel content of the vacant pixel by using an interpolation method.

[0014] In one of the embodiments, the trained semantic segmentation model is trained using a labeled cabinet image dataset.

[0015] In one embodiment, the trained semantic segmentation model includes an encoder and a decoder; the encoder is used to extract image features of the cabinet image through a convolutional neural network, and perform atrous spatial pyramid pooling on the image features to obtain encoding features; the decoder is used to upsample the encoding features, fuse the processed encoding features with the shallow features that have been convolved to obtain fused features, and perform convolution and upsampling on the fused features to obtain the segmentation prediction result.

[0016] In a second aspect, the present application also provides a cabinet image correction device, comprising:

[0017] A segmentation module, used to input the cabinet image into a trained semantic segmentation model to obtain a segmentation prediction result output by the semantic segmentation model;

[0018] An extraction module, used to extract a cabinet contour point set in the cabinet image according to the segmentation prediction result;

[0019] A simplification module, used for performing contour simplification processing on the cabinet contour point set to obtain cabinet vertex coordinates of the cabinet image;

[0020] The transformation module is used to perform perspective transformation on the cabinet image according to the cabinet vertex coordinates to obtain a corrected cabinet image.

[0021] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0022] Inputting the cabinet image into a trained semantic segmentation model to obtain a segmentation prediction result output by the semantic segmentation model;

[0023] Extracting a cabinet contour point set in the cabinet image according to the segmentation prediction result;

[0024] Performing contour simplification processing on the cabinet contour point set to obtain cabinet vertex coordinates of the cabinet image;

[0025] The cabinet image is perspective transformed according to the cabinet vertex coordinates to obtain a corrected cabinet image.

[0026] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0027] Inputting the cabinet image into a trained semantic segmentation model to obtain a segmentation prediction result output by the semantic segmentation model;

[0028] Extracting a cabinet contour point set in the cabinet image according to the segmentation prediction result;

[0029] Performing contour simplification processing on the cabinet contour point set to obtain cabinet vertex coordinates of the cabinet image;

[0030] The cabinet image is perspective transformed according to the cabinet vertex coordinates to obtain a corrected cabinet image.

[0031] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0032] Inputting the cabinet image into a trained semantic segmentation model to obtain a segmentation prediction result output by the semantic segmentation model;

[0033] Extracting a cabinet contour point set in the cabinet image according to the segmentation prediction result;

[0034] Performing contour simplification processing on the cabinet contour point set to obtain cabinet vertex coordinates of the cabinet image;

[0035] The cabinet image is perspective transformed according to the cabinet vertex coordinates to obtain a corrected cabinet image.

[0036] The above-mentioned cabinet image correction method, device, computer equipment, computer-readable storage medium and computer program product first input the cabinet image into a trained semantic segmentation model to obtain the segmentation prediction result output by the semantic segmentation model, then extract the cabinet contour point set in the cabinet image according to the segmentation prediction result, then perform contour simplification processing on the cabinet contour point set to obtain the cabinet vertex coordinates of the cabinet image, and then perform perspective transformation on the cabinet image according to the cabinet vertex coordinates to obtain the corrected cabinet image. This method, by using a trained semantic segmentation model to perform semantic segmentation on the cabinet image, can accurately mark the category to which each pixel in the cabinet image belongs, so as to better realize the recognition and segmentation of cabinet images collected in a complex computer room environment. By extracting the cabinet contour in the cabinet image according to the segmentation prediction result and performing contour simplification processing on it, it is possible to accurately simplify the set of cabinet pixels in the image into cabinet vertices that can be used for positioning in subsequent image correction without manual participation and without the need for additional template images. Moreover, this method is based on the cabinet vertex coordinates and corrects the cabinet image through perspective transformation, so that the image of the cabinet and the internal equipment can be restored more accurately, laying the foundation for subsequent image processing processes such as equipment model recognition, text recognition, and digital modeling of the computer room. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0038] Figure 1 is a schematic diagram of a process of a cabinet image correction method in one embodiment;

[0039] Figure 2 A schematic diagram of a process of extracting a cabinet contour point set in one embodiment;

[0040] Figure 3 A schematic diagram of a process for obtaining the vertex coordinates of a cabinet in one embodiment;

[0041] Figure 4 A schematic diagram of a process for obtaining a corrected cabinet image in one embodiment;

[0042] Figure 5 is a schematic diagram of perspective transformation in one embodiment;

[0043] Figure 6 is a schematic diagram of the structure of a trained semantic segmentation model in one embodiment;

[0044] Figure 7 A schematic diagram of a process of a cabinet image correction method in another embodiment;

[0045] Figure 8 is a structural block diagram of a cabinet image correction device in one embodiment;

[0046] Fig. 9 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0048] In one embodiment, Figure 1 As shown, a cabinet image correction method is provided. This embodiment uses the method applied to a server as an example for illustration. It can be understood that the method can also be applied to a terminal, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0049] Step S101: input the cabinet image into a trained semantic segmentation model to obtain a segmentation prediction result output by the semantic segmentation model.

[0050] Among them, semantic segmentation is a computer vision task whose goal is to classify each pixel in an image into a specific category. Unlike image classification tasks, semantic segmentation not only determines which categories exist in the image, but also accurately locates the locations of these categories in the image. Therefore, semantic segmentation can annotate each pixel in the image so that each pixel corresponds to a specific category label.

[0051] In this step, the cabinet image can be input into the trained semantic segmentation model, and the cabinet image can be processed by the semantic segmentation model to obtain the segmentation prediction result output by the model.

[0052] For example, the cabinet image may be a cabinet photo taken by a handheld device or the like in a computer room environment.

[0053] Exemplarily, in some embodiments, the trained semantic segmentation model may be trained using a labeled cabinet image dataset. Exemplarily, the cabinet image dataset may include multiple cabinet image samples and a category label corresponding to each pixel in each cabinet image sample. The category label may include a category label corresponding to the cabinet contour range.

[0054] Based on this, after the cabinet image is input into the trained semantic segmentation model in this step, the semantic segmentation model can predict the category to which each pixel in the cabinet image belongs and output the segmentation prediction result corresponding to the image. RGB image of size (i.e., cabinet image) as input, the segmentation prediction result of the output can be a two-dimensional array with the same size as the input image , where the value of each element in the two-dimensional array corresponds to a category. It can be understood that in the segmentation prediction result output by the model, the pixels in the cabinet image corresponding to the area inside the cabinet can be labeled as the category label corresponding to the cabinet contour range. Exemplarily, the two-dimensional array can be regarded as a mask of the cabinet image, where the pixel value of the pixel within the cabinet contour range can be 1 (i.e., the category label corresponding to the cabinet contour range), and the pixel value of the remaining area can be 0.

[0055] Step S102: extracting a cabinet contour point set from the cabinet image according to the segmentation prediction result.

[0056] Exemplarily, in this step, the outer edge pixel points can be identified in the image area marked as corresponding to the cabinet contour range by contour detection and other methods, and the cabinet contour point set is obtained therefrom. It can be understood that based on the cabinet contour point set, the cabinet contour in the cabinet image can be determined accordingly.

[0057] Step S103, performing contour simplification processing on the cabinet contour point set to obtain cabinet vertex coordinates of the cabinet image.

[0058] In this step, the cabinet outline point set can be processed by an outline simplification algorithm. This process can simplify the points in the cabinet outline point set to obtain a curve with fewer points and a shape similar to the cabinet outline determined by the cabinet outline point set, that is, a simplified outline corresponding to the cabinet outline can be obtained. Then, the vertices of the simplified outline can be used as cabinet vertices in the cabinet image, and then the corresponding cabinet vertex coordinates can be obtained according to the position of each cabinet vertex in the cabinet image.

[0059] Step S104 , performing perspective transformation on the cabinet image according to the cabinet vertex coordinates to obtain a corrected cabinet image.

[0060] In this step, the cabinet image can be corrected by perspective transformation according to the previously determined cabinet vertex coordinates.

[0061] For example, in some embodiments, in order to facilitate the application of the cabinet image in subsequent image processing such as equipment model recognition, text recognition, and digital modeling of computer rooms, the target viewing angle that the cabinet image needs to meet can be pre-set. Then, in this step, according to the perspective relationship between the acquisition viewing angle of the cabinet image and the target viewing angle, according to the cabinet vertex coordinates, the pixels in the cabinet image corresponding to the cabinet vertex and the cabinet contour range determined by the cabinet vertex are perspectively transformed to obtain a corrected cabinet image corresponding to the target viewing angle.

[0062] Exemplarily, in some embodiments, a target plane on which the corrected cabinet image is located may be pre-set, and the target vertex coordinates of each cabinet vertex on the target plane may be pre-determined based on the structural characteristics of the cabinet. Then, based on the transformation relationship between the cabinet vertex coordinates and the target vertex coordinates, the remaining pixels in the cabinet image may be perspective transformed to obtain a corrected cabinet image.

[0063] In the above cabinet image correction method, the cabinet image is first input into the trained semantic segmentation model to obtain the segmentation prediction result output by the semantic segmentation model, and then the cabinet contour is extracted from the cabinet image according to the segmentation prediction result, and then the cabinet contour is simplified to obtain the cabinet vertex coordinates of the cabinet image, and then the cabinet image is perspective transformed according to the cabinet vertex coordinates to obtain the corrected cabinet image. This method, by using the trained semantic segmentation model to perform semantic segmentation on the cabinet image, can accurately label the category to which each pixel in the cabinet image belongs, so as to better realize the recognition and segmentation of cabinet images collected in a complex computer room environment. By extracting the cabinet contour from the cabinet image according to the segmentation prediction result and performing contour simplification on it, the set of cabinet pixels in the image can be accurately simplified into cabinet vertices that can be used for positioning in subsequent image correction without manual intervention and without the need for additional template images. Moreover, this method is based on the cabinet vertex coordinates and corrects the cabinet image through perspective transformation, so that the image of the cabinet and the internal equipment can be restored more accurately, laying the foundation for subsequent image processing processes such as equipment model recognition, text recognition, and digital modeling of the computer room.

[0064] In an exemplary embodiment, Figure 2 As shown, according to the segmentation prediction result, the cabinet contour point set is extracted from the cabinet image, which may include:

[0065] Step S201: binarize the cabinet image according to the segmentation prediction result.

[0066] According to the segmentation prediction result output by the model, the pixel range corresponding to the cabinet outline range can be determined in the cabinet image. In this step, the cabinet image can be binarized to distinguish the pixels corresponding to the cabinet outline range from the remaining pixels in the cabinet image. Exemplarily, the binarization process can be to set the pixel values ​​of the pixels corresponding to the cabinet outline range to 255, and to set the pixel values ​​of the remaining pixels in the image to 0.

[0067] Step S202: performing edge detection on the binarized cabinet image to obtain an edge marked image.

[0068] In this step, an edge detection algorithm can be used to perform edge detection on the binarized cabinet image. Among them, the edge detection algorithm can identify and locate the area of ​​significant brightness change in the image through filtering, gradient calculation, etc., so as to extract the contour and boundary features of the object. Based on this, by performing edge detection on the binarized cabinet image, the edge pixels of the cabinet contour edge can be located in the binarized cabinet image, and these pixels can be marked in the image, so that the edge marked image corresponding to the cabinet image can be obtained. Exemplarily, in this step, the canny edge detection algorithm can be used to perform edge detection on the binarized cabinet image.

[0069] Step S203: Perform contour detection on the edge-marked image to obtain a cabinet contour point set.

[0070] In this step, a contour detection algorithm can be used to perform contour detection on the edge-marked image. The contour detection algorithm can find the outermost contour for the pixel points marked as edge pixels in the edge-marked image, and determine the cabinet contour point set based on the pixel points on the outermost contour. The cabinet contour point set can include all the pixel points on the outermost contour, or only some of the pixel points that can be used to determine the outermost contour. Exemplarily, in this step, the findContours method of the open source computer vision library (OpenCV) can be used to perform contour detection on the edge-marked image and output the cabinet contour point set.

[0071] In this embodiment, according to the segmentation prediction result, binarization processing, edge detection and contour detection are performed on the cabinet image in sequence, which can effectively reduce the noise in the image that affects contour extraction and obtain a more accurate contour extraction result.

[0072] In an exemplary embodiment, Figure 3 As shown, the cabinet contour point set is simplified to obtain the cabinet vertex coordinates of the cabinet image, which may include:

[0073] Step S301: Calculate the minimum convex polygon of the cabinet outline point set using a convex hull algorithm, and obtain a first cabinet outline according to the minimum convex polygon.

[0074] Among them, the convex hull algorithm can be used to find the minimum convex closed boundary of a set of points in a plane or higher-dimensional space. Its goal is to determine the "convex hull" of a given set of points, that is, the set of vertices that can surround all these points and form a convex polygon with the lowest capacity.

[0075] In this step, by using the convex hull algorithm to process the cabinet outline point set, a minimum convex polygon that can surround all points in the cabinet outline point set can be obtained, thereby simplifying the cabinet outline. Based on this, the minimum convex polygon can be used as the first cabinet outline obtained after the cabinet outline is simplified.

[0076] Step S302: Process the first cabinet outline using a polygonal approximation algorithm to obtain a second cabinet outline.

[0077] Among them, the polygonal approximation algorithm can perform polygonal approximation on the contour and delete redundant points in the contour, so that the contour can be approximated to a polygon with a specified accuracy, thereby simplifying the contour.

[0078] In this step, the first cabinet outline is processed by using a polygonal approximation algorithm, and the first cabinet outline can be polygonally approximated to delete redundant points therein, thereby obtaining a simplified second cabinet outline. Exemplarily, the Douglas-Peucker algorithm can be used to process the first cabinet outline in this step.

[0079] Step S303: Obtain cabinet vertex coordinates according to the vertices of the second cabinet outline.

[0080] Exemplarily, in this step, the vertices of the second cabinet outline may be used as cabinet vertices, so that the cabinet vertex coordinates may be obtained accordingly according to the coordinates of each vertex of the second cabinet outline in the cabinet image.

[0081] In this embodiment, the cabinet outline point set is processed by using the convex hull algorithm and the polygon approximation algorithm in sequence, which can realize the gradual simplification of the cabinet outline, so that the cabinet outline can be simplified as much as possible while maintaining accuracy, and the cabinet vertex coordinates that can be applied to subsequent perspective transformation are obtained. This process can effectively reduce the amount of cabinet outline data, thereby saving storage space and improving processing efficiency.

[0082] In an exemplary embodiment, Figure 4 As shown, performing perspective transformation on the cabinet image according to the cabinet vertex coordinates to obtain a corrected cabinet image may include:

[0083] Step S401, obtaining the target vertex coordinates corresponding to the vertex coordinates of each cabinet on the target plane.

[0084] Among them, Figure 5 As shown, image correction based on perspective transformation can project an image from one viewing angle of the projection center to another viewing angle. In this embodiment, the plane where the corrected cabinet image is located can be set as the target plane in advance, and then the cabinet image can be mapped from the image plane to the target plane through perspective transformation.

[0085] In this step, the target vertex coordinates corresponding to each cabinet vertex can be determined on the target plane in combination with the structural characteristics of the cabinet. For example, the target vertex coordinates corresponding to a cabinet vertex can be set on the target plane first, and then the relative positions of other cabinet vertices and the cabinet vertex can be determined according to the structural characteristics of the cabinet, so that the target vertex coordinates of these cabinet vertices can be determined on the target plane.

[0086] For example, assuming that the cabinet vertex coordinates include , then the corresponding target vertex coordinates can be determined on the target plane as In some embodiments, the target vertex coordinates may be set to [(0, 0), (width, 0), (width, height), (0, height)], where width may correspond to the maximum value of the lengths of the two short sides of the cabinet outline in the uncorrected cabinet image, and height may be calculated based on a fixed ratio of the width and height of the cabinet and the value of width.

[0087] Step S402: Obtain a transformation matrix between the cabinet image and the target plane according to the vertex coordinates of each cabinet and the corresponding target vertex coordinates.

[0088] In this embodiment, a transformation matrix can be used to perform perspective transformation on the cabinet image. and the corresponding coordinate points on the target plane , the perspective transformation can be expressed as:

[0089] = ,

[0090] In the formula, the matrix The distance between the cabinet image and the target plane Transformation matrix.

[0091] Among them, the cabinet vertex coordinates include , the target vertex coordinates include For example, by substituting the cabinet vertex coordinates and target vertex coordinates corresponding to each cabinet vertex into the above expression, the transformation matrix can be realized. The solution.

[0092] Step S403: transform each pixel of the cabinet image to the target plane using a transformation matrix to obtain a corrected cabinet image.

[0093] In this step, each pixel in the cabinet image can be mapped to a corresponding position on the target plane using the transformation matrix obtained above, so that a corrected cabinet image can be obtained.

[0094] In this embodiment, the transformation matrix between the cabinet image and the target plane is calculated based on the cabinet vertex coordinates and the target vertex coordinates, and the transformation matrix is ​​then used to map the pixels in the cabinet image to the target plane. This allows perspective transformation of each pixel of the cabinet image to obtain a more accurate correction result.

[0095] In an exemplary embodiment, the method may further include: if the corrected cabinet image contains missing pixels, obtaining the pixel content of the missing pixels by using an interpolation method.

[0096] Since the size of the corrected cabinet image may be different from that of the original cabinet image, the corrected cabinet image may contain missing pixels. In this embodiment, when it is detected that the corrected cabinet image contains missing pixels, the pixel content of the missing pixels can be determined by interpolation, so that the missing parts in the corrected cabinet image can be supplemented.

[0097] In this embodiment, the interpolation method is used to supplement the missing pixels in the corrected cabinet image, which can effectively improve the integrity of the corrected cabinet image.

[0098] In an exemplary embodiment, the trained semantic segmentation model includes an encoder and a decoder. The encoder is used to extract image features of the cabinet image through a convolutional neural network, and perform a hollow space pyramid pooling process on the image features to obtain encoding features. The decoder is used to upsample the encoding features, fuse the processed encoding features with the shallow features processed by convolution to obtain fused features, and perform convolution and upsampling on the fused features to obtain segmentation prediction results.

[0099] Among them, the trained semantic segmentation model used in this embodiment can be as follows Figure 6 As shown, it may include an encoder and a decoder.

[0100] The encoder of the model can receive an input cabinet image and use a pre-trained convolutional neural network as a feature extraction network to process the cabinet image, and extract high-level features (i.e., image features) of the cabinet image through multi-layer convolution and pooling operations. For example, the pre-trained convolutional neural network can be a residual neural network (ResidualNetwork, ResNet), or an Xception network, etc. Then, the encoder can perform a hollow spatial pyramid pooling process on the image features. For example, the encoder can receive RGB image of size (i.e., cabinet image) as input, use convolutional neural network to process the image, extract image features, and then connect the image features through parallel processing of hole convolution with different sampling rates, and then process them through 1×1 convolution to obtain encoding features. For example, Figure 6 As shown, the parallel dilated convolution processing can include 1×1 convolution processing, 3×3 convolution processing with a dilation rate of 6, 3×3 convolution processing with a dilation rate of 12, 3×3 convolution processing with a dilation rate of 18, global average pooling processing, etc.

[0101] The decoder of the model can upsample the encoded features output by the encoder and fuse the obtained results with the shallow features processed by 1×1 convolution to obtain fused features. Then, the fused features can be convoluted and upsampled. For example, Figure 6 As shown in the figure, the multiple of the two upsampling processes of the decoder can be 4, and the convolution process performed by the decoder on the fused features can be a 3×3 convolution process. Then, a 1x1 convolution layer can be used through the classification layer (not shown) to map the upsampled feature map to the dimension of the number of categories, that is, predict a category label for each pixel. Then, a two-dimensional array with the same size as the input cabinet image can be returned. as a predicted result.

[0102] like Figure 6 As shown, by using the trained semantic segmentation model of this embodiment, a segmentation prediction result capable of marking the cabinet contour range can be obtained based on a cabinet image including the cabinet and its internal equipment, cabinet doors, and adjacent cabinets.

[0103] In this embodiment, the semantic segmentation model applies atrous spatial pyramid pooling in the encoding module and fuses the encoded features with shallow features in the decoder, so as to achieve accurate positioning and segmentation of the target and capture and fusion of multi-scale information. The model is then applied to semantic segmentation of cabinet images, which can better adapt to the complex environment of the existing network and help improve the robustness of semantic segmentation in conditions such as cluttered wiring and dim light.

[0104] In order to further illustrate the cabinet image correction method of the present application, the following is a detailed description of the method through an embodiment:

[0105] For example, the process of the cabinet image correction method in this embodiment can be as follows: Figure 7 shown.

[0106] In this embodiment, a plurality of cabinet image samples may be first obtained, and data annotation may be performed on each cabinet image sample to obtain an annotated cabinet image dataset. A trained semantic segmentation model may be obtained by training a semantic segmentation model using the annotated cabinet image dataset.

[0107] For example, the structure of the trained semantic segmentation model in this embodiment can be as follows: Figure 6 As shown, it includes an encoder and a decoder. The encoder is used to extract the image features of the cabinet image through a convolutional neural network, and perform a hollow space pyramid pooling process on the image features to obtain a coding feature. The decoder can be used to upsample the coding feature, fuse the processed coding feature with the shallow feature processed by convolution to obtain a fusion feature, and perform convolution and upsampling on the fusion feature to obtain a segmentation prediction result.

[0108] When correcting the cabinet image, the cabinet image can be input into a trained semantic segmentation model to obtain a segmentation prediction result output by the model. Exemplarily, the segmentation prediction result can be a two-dimensional array of the same size as the input image, and the value of each element in the two-dimensional array can correspond to a category. Exemplarily, the two-dimensional array can be regarded as a mask of the cabinet image, where the pixel value of the pixel within the cabinet outline is 1, and the pixel value of the remaining area is 0.

[0109] According to the segmentation prediction result, in this embodiment, the cabinet image may be subjected to binarization processing, edge detection, and contour detection in sequence to extract the cabinet contour point set.

[0110] Exemplarily, in the binarization process, the pixel value of each pixel in the cabinet image can be binarized according to the value of each element in the two-dimensional array, the pixel value corresponding to the pixel within the cabinet contour range is set to 255, and the pixel value of the remaining pixels in the image is set to 0.

[0111] Exemplarily, in edge detection, an edge detection algorithm can be used to process the binarized cabinet image, where edge pixels of the cabinet outline edge are located, and these pixels are marked in the image, so that an edge marked image corresponding to the cabinet image can be obtained. Exemplarily, a canny edge detection algorithm can be used to perform edge detection on the binarized cabinet image.

[0112] Exemplarily, in contour detection, a contour detection algorithm can be used to perform contour detection processing on the edge marker image, which can find the outermost contour of the edge pixel points, and determine the cabinet contour point set based on the pixel points on the outermost contour. Exemplarily, the findContours method of the open source computer vision library (OpenCV) can be used to perform contour detection on the edge marker image and output the cabinet contour point set.

[0113] Among them, Figure 7 As shown, after the aforementioned binarization processing, edge detection and contour detection, the contour extraction result corresponding to the cabinet image can be obtained. Among them, according to the cabinet contour point set obtained by contour detection, the cabinet contour can be determined in the cabinet image. Based on this, in this embodiment, the cabinet contour corresponding to the cabinet contour point set can be simplified by the convex hull algorithm and the polygon approximation algorithm, and the cabinet vertices can be extracted according to the simplified cabinet contour.

[0114] Exemplarily, the cabinet outline point set is processed by a convex hull algorithm to obtain a minimum convex polygon that can surround all points in the cabinet outline point set, and then the minimum convex polygon can be used as the first cabinet outline obtained after the cabinet outline is simplified.

[0115] Exemplarily, by processing the first cabinet outline using a polygonal approximation algorithm, the first cabinet outline can be polygonally approximated, and redundant points therein can be deleted, so as to obtain a simplified second cabinet outline. Exemplarily, the first cabinet outline can be processed using a Douglas-Peucker algorithm. Optionally, the accuracy of the polygonal approximation algorithm can be set to 0.01 times the length of the first cabinet outline.

[0116] like Figure 7 As shown, after the cabinet contour point set is processed using the convex hull algorithm and the polygon approximation algorithm in sequence, a simplified cabinet contour can be obtained, and then the contour vertices of the contour can be used as cabinet vertices, so that the cabinet vertex coordinates can be obtained according to the vertex coordinates of each contour vertex.

[0117] In this embodiment, the cabinet image can be corrected by perspective transformation, and then the pixel contents of the missing pixels in the corrected cabinet image can be supplemented by interpolation.

[0118] Exemplarily, in this embodiment, the plane where the corrected cabinet image is located can be set as the target plane in advance, and the target vertex coordinates corresponding to the vertex coordinates of each cabinet in the target plane can be obtained. Then, based on the vertex coordinates of each cabinet and the corresponding target vertex coordinates, the transformation matrix between the image plane where the cabinet image is located and the target plane can be calculated. Then, the transformation matrix can be used to map each pixel point in the cabinet image to the target plane respectively to obtain the corrected cabinet image. Subsequently, in the case where the corrected cabinet image contains vacant pixels, the interpolation method can also be used to obtain the pixel content of the vacant pixels to supplement the vacant pixels.

[0119] like Figure 7 As shown, after perspective transformation and interpolation, the correction result of the cabinet image can be obtained.

[0120] In this embodiment, the cabinet image is processed using a semantic segmentation model to address the large deformation, irregular image, cluttered acquisition environment, poor lighting, and other situations that are prone to occur in the cabinet image, and accurate positioning and classification at the pixel level can be achieved. Compared with the rough positioning method based on target detection for clean, high-contrast images in the prior art, this embodiment can better handle the cabinet images with cluttered routing, dim light, and severe occlusion in the existing network environment by applying a semantic segmentation model, and achieve accurate recognition of the cabinet contour range. After obtaining the segmentation prediction results, the contour extraction and simplification technology based on edge detection, contour detection, convex hull algorithm, and polygon approximation algorithm can accurately simplify the set of cabinet pixels into the positioning points required for perspective transformation. Unlike the correction method on the market that requires manual point selection, the method of this embodiment realizes fully automatic image correction without manual participation and without the need for additional template images. In addition, in this embodiment, the perspective distortion is corrected by the perspective transformation, a nonlinear transformation, so that the cabinet image can be corrected more accurately and a more accurate correction result can be obtained. Therefore, the image correction method of this embodiment can achieve accurate detection and correction in complex environments, and has a high degree of automation without the need for human intervention. It can effectively reduce manual intervention in digital twin projects such as computer room digital modeling, thereby saving human resources, and can serve as a pre-processing for other subsequent image processing, effectively improving the accuracy of image processing.

[0121] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0122] Based on the same inventive concept, the embodiment of the present application also provides a cabinet image correction device for implementing the cabinet image correction method involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in one or more cabinet image correction device embodiments provided below can refer to the limitations of the cabinet image correction method above, and will not be repeated here.

[0123] In an exemplary embodiment, Figure 8 As shown, a cabinet image correction device 800 is provided, comprising:

[0124] The segmentation module 801 is used to input the cabinet image into the trained semantic segmentation model to obtain the segmentation prediction result output by the semantic segmentation model.

[0125] The extraction module 802 is used to extract a cabinet contour point set in the cabinet image according to the segmentation prediction result.

[0126] The simplification module 803 is used to perform contour simplification processing on the cabinet contour point set to obtain the cabinet vertex coordinates of the cabinet image.

[0127] The transformation module 804 is used to perform perspective transformation on the cabinet image according to the cabinet vertex coordinates to obtain a corrected cabinet image.

[0128] In an exemplary embodiment, the extraction module 802 is further used to: binarize the cabinet image according to the segmentation prediction result; perform edge detection on the binarized cabinet image to obtain an edge marked image; perform contour detection on the edge marked image to obtain a cabinet contour point set.

[0129] In an exemplary embodiment, the simplification module 803 is also used to: use a convex hull algorithm to calculate the minimum convex polygon of the cabinet contour point set, and obtain a first cabinet contour based on the minimum convex polygon; use a polygon approximation algorithm to process the first cabinet contour to obtain a second cabinet contour of the cabinet contour; and obtain cabinet vertex coordinates based on the vertices of the second cabinet contour.

[0130] In an exemplary embodiment, the transformation module 804 is also used to: obtain the target vertex coordinates corresponding to each cabinet vertex coordinate in the target plane; obtain the transformation matrix between the cabinet image and the target plane based on each cabinet vertex coordinate and the corresponding target vertex coordinate; and use the transformation matrix to transform each pixel of the cabinet image to the target plane to obtain a corrected cabinet image.

[0131] In an exemplary embodiment, the device further includes: an interpolation module, which is used to obtain pixel content of the vacant pixels by using an interpolation method if the corrected cabinet image contains vacant pixels.

[0132] In an exemplary embodiment, the trained semantic segmentation model is trained using a labeled cabinet image dataset.

[0133] In an exemplary embodiment, the trained semantic segmentation model includes an encoder and a decoder; the encoder is used to extract image features of a cabinet image through a convolutional neural network, and perform atrous spatial pyramid pooling on the image features to obtain encoding features; the decoder is used to upsample the encoding features, fuse the processed encoding features with the shallow features that have been convolved to obtain fused features, and perform convolution and upsampling on the fused features to obtain segmentation prediction results.

[0134] Each module in the above cabinet image correction device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module.

[0135] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Fig. 9As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data such as cabinet images. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a cabinet image correction method is implemented.

[0136] Those skilled in the art will understand that Fig. 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0137] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0138] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0139] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0140] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0141] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a data processing logic based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0142] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0143] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A cabinet image correction method, characterized in that: The method comprises: Inputting the cabinet image into a trained semantic segmentation model to obtain a segmentation prediction result output by the semantic segmentation model; Extracting a cabinet contour point set in the cabinet image according to the segmentation prediction result; Performing contour simplification processing on the cabinet contour point set to obtain cabinet vertex coordinates of the cabinet image; The cabinet image is perspective transformed according to the cabinet vertex coordinates to obtain a corrected cabinet image.

2. The method according to claim 1, characterized in that Extracting a cabinet contour point set from the cabinet image according to the segmentation prediction result includes: According to the segmentation prediction result, binarization processing is performed on the cabinet image; Performing edge detection on the cabinet image that has been binarized to obtain an edge marked image; Perform contour detection on the edge-marked image to obtain the cabinet contour point set.

3. The method according to claim 1, characterized in that The step of performing contour simplification processing on the cabinet contour point set to obtain cabinet vertex coordinates of the cabinet image includes: Calculate the minimum convex polygon of the cabinet outline point set using a convex hull algorithm, and obtain a first cabinet outline according to the minimum convex polygon; Processing the first cabinet outline using a polygonal approximation algorithm to obtain a second cabinet outline of the cabinet outline; The cabinet vertex coordinates are obtained according to the vertices of the second cabinet outline.

4. The method according to claim 1, characterized in that The performing perspective transformation on the cabinet image according to the cabinet vertex coordinates to obtain a corrected cabinet image includes: Obtaining target vertex coordinates corresponding to the vertex coordinates of each cabinet on the target plane; Obtaining a transformation matrix between the cabinet image and the target plane according to the cabinet vertex coordinates and the corresponding target vertex coordinates; Each pixel of the cabinet image is transformed to the target plane using the transformation matrix to obtain the corrected cabinet image.

5. The method according to claim 1, characterized in that The method further comprises: If the corrected cabinet image contains missing pixels, the pixel content of the missing pixels is obtained by using an interpolation method.

6. The method according to claim 1, characterized in that The trained semantic segmentation model is trained using a labeled cabinet image dataset.

7. The method according to any one of claims 1 to 6, characterized in that: The trained semantic segmentation model includes an encoder and a decoder; The encoder is used to extract image features of the cabinet image through a convolutional neural network, and perform a hollow space pyramid pooling process on the image features to obtain encoding features; The decoder is used to upsample the coding features, fuse the processed coding features with the shallow features that have been convolved to obtain fused features, and perform convolution and upsampling on the fused features to obtain the segmentation prediction result.

8. A cabinet image correction device, characterized in that: The device comprises: A segmentation module, used to input the cabinet image into a trained semantic segmentation model to obtain a segmentation prediction result output by the semantic segmentation model; An extraction module, used to extract a cabinet contour point set in the cabinet image according to the segmentation prediction result; A simplification module, used for performing contour simplification processing on the cabinet contour point set to obtain cabinet vertex coordinates of the cabinet image; The transformation module is used to perform perspective transformation on the cabinet image according to the cabinet vertex coordinates to obtain a corrected cabinet image.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images