Equipment checking method and device and related equipment

Abstract

The invention provides an equipment checking method and device and related equipment, and belongs to the technical field of image recognition, and the equipment checking method provided by the embodiment of the invention comprises the steps: obtaining an original image of to-be-checked equipment; identifying a region of interest (ROI) of the original image by using a U-shaped network (U-Net) network model, wherein the ROI comprises a visual coding identifier of the to-be-checked device; monitoring the visual coding identifier contained in the ROI to obtain visual coding identifier information; and performing checking operation on the to-be-checked equipment according to the visual code identification information. The equipment checking method can improve the checking efficiency.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method, apparatus and related equipment for inventorying devices. Background Technology

[0002] With the rapid development of computer and internet technologies, the effective management of data center equipment is becoming increasingly important for enterprises. Equipment inventory is a crucial task in data center equipment management. Currently, equipment inventory involves manual recording of equipment information by inventory personnel, followed by comparison and updating with previous records. When dealing with a large variety of equipment and multiple inventory checks, this results in a significant amount of repetitive and high-frequency work for inventory personnel, leading to low inventory efficiency. Summary of the Invention

[0003] This application provides a method, apparatus, and related equipment for inventory counting, which can solve the technical problem of low inventory counting efficiency.

[0004] In a first aspect, embodiments of this application provide a method for inventorying equipment, the method comprising:

[0005] Acquire the original images of the equipment to be inventoried;

[0006] The U-Net network model is used to identify the region of interest (ROI) in the original image, and the ROI contains the visual coded identifier of the device to be inventoried.

[0007] Monitor the visually encoded identifiers contained in the ROI to obtain visually encoded identifier information;

[0008] The inventory operation is performed on the equipment to be inventoried based on the visual coding identification information.

[0009] Optionally, the step of identifying the ROI of the original image using the U-Net network model includes:

[0010] The original image is subjected to multi-level depthwise separable convolution and downsampling operations using the encoder of the U-Net network model to generate the first intermediate feature map corresponding to each level;

[0011] Using the decoder of the U-Net network model, the first intermediate feature map and the second intermediate feature map corresponding to each layer are fused to obtain the fused feature map of each layer. Then, convolution and upsampling operations are performed on the fused feature map of each layer to obtain the first intermediate image.

[0012] The second intermediate feature map is obtained by upsampling the first intermediate feature map of the last level and / or the fused feature maps of each level; the first intermediate image includes a target region and a background region outside the target region, and the position of the target region in the first intermediate image matches the position of the ROI in the original image;

[0013] Based on the target region of the first intermediate image and the original image, the ROI of the original image is identified.

[0014] Optionally, the step of fusing the first intermediate feature map and the second intermediate feature map corresponding to each level to obtain the fused feature map of each level includes:

[0015] The first intermediate feature map, the second intermediate feature map, and the pixel weight vector corresponding to each level are weighted and fused to obtain the fused feature map of each level.

[0016] The pixel weight vector of each level includes the first pixel weight of the first intermediate feature map corresponding to each level and / or the first pixel weight of the second intermediate feature map corresponding to each level.

[0017] Optionally, the encoder of the U-Net network model includes multiple depthwise separable convolutional layers connected along the downsampling path;

[0018] Each of the depthwise separable convolutional layers comprises multiple cascaded depthwise separable convolutional modules; the separable convolutional module includes depthwise convolutional blocks and pointwise convolutional blocks;

[0019] The depthwise separable convolution module is used for depthwise separable convolution operations; the depthwise convolution block is used for depthwise convolution operations; and the pointwise convolution block is used for pointwise convolution operations.

[0020] Optionally, the depthwise separable convolution module includes multiple convolution kernels, each convolution kernel including multiple second pixel weights, and the method further includes:

[0021] Each convolutional kernel is pruned by removing a preset weight from the weights of the second pixel within the convolutional kernels to obtain multiple pruned convolutional kernels.

[0022] The data bit width of the second pixel weights of the multiple pruned convolutional kernels is adjusted to a preset data bit width to obtain the adjusted multiple convolutional kernels;

[0023] The original image and / or the first intermediate feature map are subjected to depthwise separable convolution operations using the adjusted multiple convolution kernels.

[0024] Optionally, the method further includes:

[0025] Using the target region of the first intermediate image as a mask, a masking operation is performed on the original image to obtain a second intermediate image. The second intermediate image contains the ROI of the original image and the background region of the first intermediate image.

[0026] Identify the ROI of the original image based on the second intermediate image.

[0027] Optionally, the target region of the first intermediate image is subjected to grayscale binarization to obtain a binarized first intermediate image, wherein the target region of the binarized first intermediate image is a white pixel region and the background region of the binarized first intermediate image is a black pixel region.

[0028] Contour extraction is performed on the target region of the binarized first intermediate image to obtain the contour information of the target region;

[0029] The target region of the first intermediate image after contour extraction is dilated using a preset structuring element to fill in the pixel gaps and / or breaks in the target region of the first intermediate image.

[0030] Optionally, the monitoring of the visually encoded identifiers contained in the ROI to obtain visually encoded identifier information includes:

[0031] A perspective transformation operation is performed on the ROI in the original image included in the second intermediate image to obtain the perspective-corrected second intermediate image;

[0032] Image monitoring and / or character monitoring are performed on the ROIs in the original image included in the perspective-corrected second intermediate image to obtain the visual coding identification information.

[0033] Secondly, embodiments of this application provide an equipment inventory device, the device comprising:

[0034] The acquisition unit is used to acquire the original images of the equipment to be inventoried.

[0035] The first processing unit is used to identify the region of interest (ROI) of the original image using a U-Net network model, wherein the ROI contains the visual encoding identifier of the original image;

[0036] A monitoring unit is used to monitor the video encoding identifier of the original image contained in the ROI and obtain visual encoding identifier information;

[0037] An execution unit is used to perform an inventory operation on the equipment to be inventoried based on the visual coding identification information.

[0038] Thirdly, embodiments of this application provide an electronic device, including: a processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, it implements the steps of the device inventory method as described in the first aspect.

[0039] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the device inventory method as described in the first aspect.

[0040] Fifthly, embodiments of this application provide a computer program product including computer instructions that, when executed by a processor, implement the steps of the device inventory method as described in the first aspect.

[0041] In this embodiment, the original image of the device to be inventoried is acquired; a U-Net network model is used to identify the Region of Interest (ROI) of the original image, where the ROI contains the visual coded identifier of the device to be inventoried; the visual coded identifier contained in the ROI is monitored to obtain visual coded identifier information; and an inventory operation is performed on the device to be inventoried based on the visual coded identifier information. Because the original image of the device to be inventoried is processed using the U-Net network model, the ROI containing the visual coded identifier can be quickly identified. Then, the inventory operation is performed on the device based on the monitored visual coded identifier information, which reduces manual operation and improves inventory efficiency. Attached Figure Description

[0042] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 This is a flowchart of a data processing method provided in an embodiment of this application;

[0044] Figure 2 This is a flowchart of another data processing method provided in an embodiment of this application;

[0045] Figure 3 This is a schematic diagram of a U-Net network model provided in an embodiment of this application;

[0046] Figure 4 This is a schematic diagram of another U-Net network model provided in an embodiment of this application;

[0047] Figure 5 This is a schematic diagram of a second image provided in an embodiment of this application;

[0048] Figure 6 This is a schematic diagram of a first intermediate image after binarization provided in an embodiment of this application;

[0049] Figure 7 This is a schematic diagram of a first intermediate image after contour extraction provided in an embodiment of this application;

[0050] Figure 8 This is a schematic diagram of the ROI in the original image included in a perspective-corrected second intermediate image provided in an embodiment of this application;

[0051] Figure 9 This is a schematic diagram of the ROI in the original image included in a preprocessed second intermediate image provided in an embodiment of this application;

[0052] Figure 10 This is a schematic diagram of the structure of an equipment inventory device provided in an embodiment of this application;

[0053] Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0054] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0055] The terms "first," "second," etc., used in this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of the same class, not limited in number; for example, the first object can be one or more. Furthermore, "and / or" in this application indicates at least one of the connected objects. For example, the scope of protection of "A and / or B" covers at least three scenarios: Scenario 1: including A but not B; Scenario 2: including B but not A; Scenario 3: including both A and B. Additionally, the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0056] See Figure 1 , Figure 1 This is a flowchart of an equipment inventory method provided in an embodiment of this application, such as... Figure 1 As shown, the method includes the following steps:

[0057] Step 101: Obtain the original image of the equipment to be inventoried;

[0058] The equipment to be inventoried can be various types of computer room equipment, or other equipment. For example, network cabinets, air conditioning and refrigeration equipment, or printers.

[0059] In some implementations, please refer to Figure 2 The Figure 2 This is a flowchart of another equipment inventory method provided in an embodiment of this application, such as... Figure 2 As shown:

[0060] The process of acquiring the original image of the equipment to be inventoried can be achieved by capturing a video stream of the equipment on a mobile device, processing the acquired video stream (e.g., extracting keyframes from the original image to obtain the original image of the equipment to be inventoried), and then performing image and / or text detection on the output image. The text detection can also be referred to as character detection.

[0061] The mobile device can be a mobile phone, tablet, or camera, or other portable recording device. The mobile device can be handheld by an operator or mounted on various mobile vehicles, mobile robots, or drones; there are no restrictions on this.

[0062] In this step, the original images of the equipment to be inventoried are acquired, providing data support for identifying the visual encoding identifiers of the original images of the equipment to be inventoried.

[0063] Step 102: Use the U-Net network model to identify the Region of Interest (ROI) of the original image. The ROI contains the visual encoding identifier of the device to be inventoried.

[0064] The U-Net (U-shaped network) model can be an end-to-end neural network model for image segmentation. Its network structure is U-shaped, hence the name U-Net. For details, please refer to [link to relevant documentation]. Figure 3 , Figure 3 This is a schematic diagram of a U-net network model provided in this application, such as... Figure 3 As shown:

[0065] The U-Net network model can include an encoder (also known as a compressed channel) on one side and a decoder (also known as an expanded channel) on the other side. The encoder can extract features through convolution operations and downsampling (e.g., adding pooling layers and using a 2×2 max pooling window to halve the size of the input image (e.g., the original image and / or the first feature map described later); the decoder can gradually recover image details through upsampling, feature fusion, and convolution to obtain the output image. In step 102, the U-Net network model can be used to identify the ROI of the original image.

[0066] The ROI can refer to a specific local area delineated from the original image or intermediate feature map in computer vision, U-Net network model image processing, or other image processing technologies, which requires focused analysis, segmentation, detection, or feature extraction. For example, in medical image scenarios, the ROI can be a suspected lesion area, a tumor boundary area, or an organ parenchyma area; in industrial application scenarios, the ROI can be a product defect candidate area or a key area in the monitoring screen; in this application, the ROI can be the area containing the visually encoded identifier of the equipment to be inventoried.

[0067] The visual coding identifier refers to a coding form that has visual recognizability and is used to uniquely identify a target object (such as equipment or product). It can be read through image acquisition, character recognition, or graphic decoding to realize functions such as object traceability, inventory, and verification.

[0068] For example, the visual encoding identifier can be a serial number (SN code), a part number (PN code), or a QR code, or other types, such as a barcode, asset number, or equipment unique identifier.

[0069] In this step, the U-Net network model is used to process the acquired raw images, identifying the Regions of Interest (ROIs) containing visually coded identifiers of the equipment to be inventoried. This significantly reduces the time and error associated with manual screening and location. Compared to traditional manual searching methods, the U-Net model can process a large number of raw images of equipment to be inventoried in batches, automatically and accurately, effectively improving the overall efficiency and automation level of equipment inventory. Moreover, with the passage of time and the accumulation of experience, the U-Net model can be continuously trained and optimized to further improve the accuracy of identification.

[0070] Step 103: Monitor the visual encoding identifier contained in the ROI to obtain visual encoding identifier information;

[0071] For example, when the visual encoding is represented as a barcode or QR code, the visual encoding identifier within the ROI can be decoded through barcode / QR code recognition operations to obtain the visual encoding identifier information.

[0072] In this step, by monitoring the visual coding marks in the ROI, we can obtain the visual coding mark information of the device, which significantly reduces human intervention and human error, and improves recognition speed and processing efficiency.

[0073] Step 104: Perform an inventory operation on the equipment to be inventoried based on the visual coding identification information.

[0074] The step of performing the inventory operation on the equipment to be inventoried based on the visual coding identification information can be: comparing the acquired visual coding identification information with the registration information in the equipment asset management system to achieve automatic identification, inventory status update and recording of the equipment to be inventoried, thereby improving the intelligence and automation level of asset inventory.

[0075] In this step, the inventory operation of the equipment to be inventoried is performed according to the visual coding identification information, which can achieve efficient inventory of the equipment to be inventoried.

[0076] In this embodiment, since the original image of the device to be inventoried is processed using the U-Net network model, the region of interest containing the visual coded identifier can be quickly identified. Then, the inventory operation of the device to be inventoried is performed based on the monitored visual coded identifier information, which can reduce manual operation and improve inventory efficiency.

[0077] In some implementations, identifying the ROI of the original image using the U-Net network model includes:

[0078] The original image is subjected to multi-level depthwise separable convolution and downsampling operations using the encoder of the U-Net network model to generate the first intermediate feature map corresponding to each level;

[0079] Using the decoder of the U-Net network model, the first intermediate feature map and the second intermediate feature map corresponding to each layer are fused to obtain the fused feature map of each layer. Then, convolution and upsampling operations are performed on the fused feature map of each layer to obtain the first intermediate image.

[0080] The second intermediate feature map is obtained by upsampling the first intermediate feature map of the last layer and / or the fused feature maps of each layer; the first intermediate image includes the target region (see [reference needed]). Figure 4 The target region and the background region outside the target region, wherein the position of the target region in the first intermediate image matches the position of the ROI in the original image;

[0081] Based on the target region of the first intermediate image and the original image, the ROI of the original image is identified.

[0082] The depthwise separable convolution operation can be an efficient convolution operation, including depthwise convolution and pointwise convolution. Depthwise convolution only considers the spatial relationships within each channel of the feature map (e.g., the original image and / or the first feature map), while pointwise convolution only considers the channel relationships at each location. Therefore, the depthwise separable convolution operation can learn spatial and channel features with fewer parameters and less computation, thereby reducing computational complexity and improving the efficiency of the U-Net network model in recognizing the ROI of the original image.

[0083] For example, when performing depthwise separable convolution on the input feature maps (e.g., the original image and the first feature map mentioned above), depthwise convolution is the first step. In this step, each input channel uses a separate convolution kernel. If there are M channels, there will be M convolution kernels, each performing convolution only on its corresponding channel. Pointwise convolution is the second step. In this step, a convolution kernel with a preset stride (e.g., 1x1 stride) is used to convolve the output of the depthwise convolution. The purpose is to combine the features from different channels, thereby changing the number of channels. This step is equivalent to performing a fully connected operation along the channel dimension, which can increase the expressive power of the model and reduce the computational complexity.

[0084] In this embodiment, the U-Net network model significantly reduces computational load and shortens feature extraction time by using multi-level depthwise separable convolution and downsampling operations in the encoder, thereby improving the efficiency of obtaining the first feature map. The decoder fuses the first intermediate feature map with the second intermediate feature map generated by upsampling, and performs convolution and upsampling operations on the fused feature maps at each level to obtain the first intermediate image (see [reference]). Figure 4 The first intermediate image includes a target region and a background region outside the target region. The position of the target region in the first intermediate image matches the position of the ROI in the original image, eliminating the need for manual calibration. Based on the target region of the first intermediate image and the original image, the ROI of the original image is identified, reducing manual reliance and improving inventory efficiency.

[0085] The position of the target region in the first intermediate image matches the position of the ROI in the original image. This can be either the same as the position of the ROI in the first intermediate image, or the position of the target region in the first intermediate image is similar to the position of the ROI in the original image.

[0086] In some implementations, the step of fusing the first intermediate feature map and the second intermediate feature map corresponding to each level to obtain the fused feature map for each level includes:

[0087] The first intermediate feature map, the second intermediate feature map, and the pixel weight vector corresponding to each level are weighted and fused to obtain the fused feature map of each level.

[0088] The pixel weight vector of each level includes the first pixel weight of the first intermediate feature map corresponding to each level and / or the first pixel weight of the second intermediate feature map corresponding to each level.

[0089] The pixel weight vectors at each level can be pre-defined by those skilled in the art, or they can be adaptive pixel weight vectors generated based on a channel attention mechanism, dynamically assigning weights to features from different channels. These pixel weight vectors at each level can be used in the decoder section of the U-Net network to improve multi-scale feature fusion.

[0090] In this embodiment, the first intermediate feature map, the second intermediate feature map, and the pixel weight vector corresponding to each level are weighted and fused to obtain the fused feature map of each level. By using the pixel weight vector of each level, the influence of invalid features caused by interference factors is reduced, and the fused feature map focuses on the core discrimination information of ROI, which improves the efficiency and accuracy of ROI recognition, thereby improving the efficiency of equipment inventory.

[0091] In some implementation methods, please participate Figure 4 , Figure 4 This is a schematic diagram of another U-Net network model provided in an embodiment of this application, such as... Figure 4 As shown:

[0092] The encoder of the U-Net network model includes multiple depthwise separable convolutional layers connected along the downsampling path;

[0093] Each of the depthwise separable convolutional layers comprises multiple cascaded depthwise separable convolutional modules (i.e., Figure 4 The bottleneck module in the middle); the separable convolution module includes depthwise convolutional blocks and pointwise convolutional blocks (in the middle). Figure 4 (not shown);

[0094] The depthwise separable convolution module is used for depthwise separable convolution operations; the depthwise convolution block is used for depthwise convolution operations; and the pointwise convolution block is used for pointwise convolution operations.

[0095] In this embodiment, the encoder of the U-Net network model employs multiple depthwise separable convolutional layers connected along the downsampling path. Each layer contains multiple cascaded depthwise separable convolutional modules, each module consisting of a depthwise convolutional block and a pointwise convolutional block, responsible for depthwise convolution and pointwise convolution operations respectively. The separable convolutional layers can accelerate the extraction speed of original image features and the first feature map, capture local image detail features and inter-channel correlation features, extract multi-scale features through multi-level progressive extraction, effectively condense information within the ROI, and filter background redundancy interference, providing support for identifying the ROI in the original image, thereby improving the inventory efficiency and accuracy of the device.

[0096] In some embodiments, the depth-separable convolution module includes multiple convolution kernels, each kernel including multiple second pixel weights; the method further includes:

[0097] Each convolutional kernel is pruned by removing a preset weight from the weights of the second pixel within the convolutional kernels to obtain multiple pruned convolutional kernels.

[0098] The data bit width of the second pixel weights of the multiple pruned convolutional kernels is adjusted to a preset data bit width to obtain the adjusted multiple convolutional kernels;

[0099] The original image and / or the first intermediate feature map are subjected to depthwise separable convolution operations using the adjusted multiple convolution kernels.

[0100] The convolution kernel can be a core operator in the U-Net network model used to extract image features; for example, the convolution kernel can be a multidimensional matrix (or tensor) of fixed size, whose element values ​​are preset or optimized weight parameters (such as the second pixel weight). In depthwise separable convolution operations, the convolution kernel extracts local features (such as edges, textures, color blocks, etc.) by performing sliding window-like product summation operations on local regions of the image.

[0101] For example, adjusting the data bit width of the second pixel weights of the plurality of pruned convolution kernels to a preset data bit width can be achieved by converting the data bit width of the second pixel weights from a 32-bit floating-point number to an 8-bit integer representation, thereby significantly reducing the model's storage space and computational overhead.

[0102] In this embodiment, a pruning algorithm is used to remove preset weights from the second pixel weights within the convolution kernel, eliminating invalid or low-contribution preset weights. This reduces the number of parameters and computational load of the convolution kernel while preserving its core feature extraction capability. The data bit width of the second pixel weights of the multiple pruned convolution kernels is adjusted to a preset data bit width. The adjusted multiple convolution kernels are then used to perform depthwise separable convolution operations on the original image and / or the first intermediate feature map. This further compresses the storage volume and data transmission overhead of the U-Net model, improves the ROI recognition efficiency of the original image, and thus improves the efficiency of the device inventory method.

[0103] In some implementations, please refer to the specific examples. Figure 4 and Figure 5 , Figure 5 This is a schematic diagram of a second image provided in an embodiment of this application, such as... Figure 5 As shown, the method further includes:

[0104] Using the target region of the first intermediate image as a mask, a masking operation is performed on the original image to obtain a second intermediate image. The second intermediate image contains the ROI of the original image and the background region of the first intermediate image.

[0105] Identify the ROI of the original image based on the second intermediate image.

[0106] In this embodiment, the target region of the first intermediate image is used as a mask to perform a masking operation on the original image, which can accurately preserve the region in the original image corresponding to the ROI; at the same time, redundant background information is masked to generate a second intermediate image containing the ROI of the original image and the background region of the first intermediate image, thereby achieving efficient separation of the ROI from the background, improving the efficiency of identifying the ROI of the original image, and also ensuring the accuracy of ROI positioning.

[0107] For example, the masking operation can be a pixel-level operation between a mask image (e.g., a first intermediate image) and the corresponding region of the original image (e.g., the ROI), thereby achieving processing or filtering of a specified region. Specifically, a mask image (e.g., the first intermediate image) of the same size as the original image needs to be created, where the pixel value represents the selection of a region or pixel. Typically, the mask value is binary (0 or 1), where 0 indicates no selection and 1 indicates selection. By operating the mask image (e.g., the first intermediate image) on the corresponding region of the original image, and by operating on the selected pixels in the mask with the pixels of the corresponding region in the original image, a new image (e.g., the mask region changes) can be obtained. Figure 5 (The second intermediate image shown).

[0108] In some implementations, please refer to Figure 4 , Figure 6 and Figure 7 , Figure 6 This is a schematic diagram of a first intermediate image after binarization, provided in an embodiment of this application. Figure 7 This is a schematic diagram of a first intermediate image after contour extraction provided in an embodiment of this application, as shown below. Figure 4 , Figure 6 and Figure 7 As shown:

[0109] The background region of the first intermediate image is a black pixel region, and the method further includes:

[0110] The target region of the first intermediate image is subjected to grayscale binarization to obtain the binarized first intermediate image. The target region of the binarized first intermediate image is a white pixel region, and the background region of the binarized first intermediate image is a black pixel region.

[0111] Contour extraction is performed on the target region of the binarized first intermediate image to obtain the contour information of the target region;

[0112] The target region of the first intermediate image after contour extraction is dilated using a preset structuring element to fill in the pixel gaps and / or breaks in the target region of the first intermediate image.

[0113] In this embodiment, by performing grayscale binarization, contour extraction, and dilation operations on the target area of ​​the first intermediate image, the distinction between the target area and the black background is enhanced, the target contour is accurately obtained, and the pixel gaps and broken parts are filled in, thereby reducing recognition interference, improving the accuracy of ROI positioning, and thus improving the efficiency and accuracy of ROI recognition, and consequently improving the efficiency and accuracy of the inventory equipment.

[0114] For example, the grayscale binarization of the target region of the first intermediate image can be performed as follows: First, a weighted average method is used to perform grayscale processing on the target region of the first intermediate image. The values ​​of the red, green, and blue channels are used, and the grayscale value is calculated by combining the channel weights with the human eye's sensitivity to each color. An appropriate threshold is selected to divide the pixels in the grayscale image into two categories. Pixels below the threshold are set to black, while pixels above or equal to the threshold are set to white.

[0115] For example, the contour extraction operation may include: determining a connected region (i.e., the target region) with white pixel values ​​in the binarized first intermediate image; extracting the corresponding contour of the target region according to a set contour retrieval mode; and processing the points of the contour according to a contour approximation method to obtain a valid contour. After extracting the valid contour, determining the principal direction feature vector of the point set within the valid contour (and possibly the covariance matrix and / or eigenvalues); determining the minimum bounding rectangle angle of the target region based on the principal direction feature vector; rotating the point set within the valid contour according to the minimum bounding rectangle angle of the target region; then determining the point set enclosing the rotated valid contour; and finally determining the first intermediate image after contour extraction.

[0116] In this embodiment, by extracting, approximating, rotating, and enclosing the target region contour of the first intermediate image with a minimum bounding rectangle, the first intermediate image after contour extraction is determined, making the target region more consistent with the actual shape of the ROI in the original image, avoiding positioning deviations caused by the target region being tilted or having irregular edges; at the same time, the boundary of the target region is effectively regularized, improving the efficiency and accuracy of subsequent ROI recognition in the original image.

[0117] For example, the dilation operation can be performed using a sliding structuring element, which is a small image region, typically a small rectangle or circle. For each pixel, the structuring element is placed at that pixel's position, and the overlapping area between the structuring element and the image is checked. If any pixel in the structuring element matches a corresponding pixel in the image (i.e., both are white or bright), that pixel is set to white. Therefore, the dilation operation can cause the white area to expand, thereby filling in pixel gaps and / or breaks present in the target region of the first intermediate image. The dilation operation can be described by the following formula:

[0118]

[0119] Where C represents the output image; A represents the input image; "⊕" represents the dilation operation; and B represents the structuring element. Specifically, the basic idea of ​​the dilation operation is to slide the structuring element on the input image. If the structuring element intersects with the input image in an overlapping area, the corresponding position in the output image C is set to white, thereby filling in the pixel gaps and / or breaks in the corresponding area of ​​the output image C.

[0120] In some implementations, please refer to Figure 8 , Figure 8 This is a schematic diagram of the Region of Interest (ROI) in the original image included in a perspective-corrected second intermediate image provided in an embodiment of this application, as shown below. Figure 8 As shown:

[0121] The monitoring of the visually encoded identifiers contained in the ROI yields visually encoded identifier information, including:

[0122] A perspective transformation operation is performed on the ROI in the original image included in the second intermediate image to obtain the perspective-corrected second intermediate image;

[0123] Image monitoring and / or character monitoring are performed on the ROIs in the original image included in the perspective-corrected second intermediate image to obtain the visual coding identification information.

[0124] In this embodiment, by performing perspective transformation on the ROIs in the original image included in the second intermediate image, perspective correction is performed to correct the distortion of visually encoded identifiers caused by factors such as shooting angle and tilt, so that the ROI visually encoded identifiers are presented in a standardized and positive form. Image monitoring and / or character monitoring are then performed on the ROIs in the original image included in the perspective-corrected second intermediate image, which can significantly improve the detection accuracy, recognition robustness, and recognition efficiency of visually encoded identifier information.

[0125] Specifically, the perspective transformation operation performed on the ROI in the original image, which includes the second intermediate image, to obtain the perspective-corrected second intermediate image can be a perspective transformation based on key point selection, including:

[0126] The corner coordinates of the Regions of Interest (ROIs) in the original image included in the second intermediate image are sorted. The four corner coordinates are then used to determine a perspective transformation matrix. This matrix is ​​then used to map the ROIs in the original image included in the second intermediate image before perspective transformation to the ROIs in the original image included in the second intermediate image after perspective correction, thus achieving perspective correction. In other words, the perspective correction operation uses a perspective transformation matrix to map points in the source image to new positions in the target image.

[0127] For example, the perspective operation can be described by the following formula:

[0128]

[0129] in, =[ x′,y′,1] T These can be the homogeneous coordinates of a point in the target image, P=[x,y,1] T H can be the homogeneous coordinates of points in the source image, and H can represent the perspective transformation matrix (the size of the perspective transformation matrix can be arbitrarily set by those skilled in the art as needed, for example, H represents a 3x3 perspective transformation matrix). The perspective transformation matrix H can also be determined by the least squares method or other methods.

[0130] Specifically, the step of performing image detection and character detection on the ROI in the original image, which includes the perspective-corrected second intermediate image, to obtain the visually encoded identification information may include:

[0131] The ROIs in the original image included in the perspective-corrected second intermediate image are preprocessed, for example, by enhancing the brightness of the ROIs in the original image included in the perspective-corrected second intermediate image, to obtain the ROIs in the original image included in the preprocessed second intermediate image (see details). Figure 9 , Figure 9 This is a schematic diagram of the ROI in the original image included in a preprocessed second intermediate image provided in this application embodiment, in order to achieve higher image detection accuracy and character detection accuracy.

[0132] Specifically, the image monitoring can be performed by using a preset image library to determine the area where the visually encoded identifier is located, and then identifying the visually encoded identifier information shown.

[0133] Taking a barcode as an example of a visually encoded identifier, the image monitoring can involve using a multidimensional barcode recognition library to find possible barcode regions. For instance, a multidimensional barcode recognition library could be the Zxing library (Zebra Crossing Library). Zxing is an open-source multidimensional barcode recognition library. Its barcode image detection module searches for possible barcode regions in an image, using different detection algorithms for different types of barcodes to locate their positions and identifiers. Once the barcode region is located, Zxing analyzes the pixels in the image to find the various modules and information of the barcode. Different types of barcodes have different encoding rules, and Zxing parses the data in the image according to these rules. Zxing uses error-correcting codes and algorithms to repair damaged modules, allowing the target to be identified even if some modules in the image are damaged or blurred, thus improving the decoding success rate. After successful decoding, Zxing generates a result object containing the barcode type and decoded data.

[0134] Specifically, the character detection can be optical character recognition (OCR) monitoring.

[0135] More specifically, the principle of the OCR character detection can be as follows: the numbers in the image are identified. Since the barcode characters are closely arranged, the image is first segmented into individual characters or words. Then, features for recognition are extracted for each individual word. Finally, the template and features are repeatedly compared to output the read barcode information.

[0136] It should be noted that the above-described equipment inventory method can be performed by an electronic device, that is, all the steps included in the above method are performed by the electronic device, which can be an electronic device such as a server, computer or mobile phone.

[0137] Please see Figure 10 , Figure 10 This is a schematic diagram of the structure of an equipment inventory device provided in an embodiment of this application, as shown below. Figure 10 As shown, the equipment inventory device 1000 includes:

[0138] Acquisition unit 1001 is used to acquire the original image of the equipment to be inventoried;

[0139] The first processing unit 1002 is used to identify the region of interest (ROI) of the original image using a U-Net network model, wherein the ROI contains the visual encoding identifier of the original image;

[0140] The monitoring unit 1003 is used to monitor the video encoding identifier of the original image contained in the ROI and obtain visual encoding identifier information;

[0141] The execution unit 1004 is used to perform an inventory operation on the equipment to be inventoried according to the visual coding identification information.

[0142] Optionally, the step of identifying the ROI of the original image using the U-Net network model includes:

[0143] The original image is subjected to multi-level depthwise separable convolution and downsampling operations using the encoder of the U-Net network model to generate the first intermediate feature map corresponding to each level;

[0144] Using the decoder of the U-Net network model, the first intermediate feature map and the second intermediate feature map corresponding to each layer are fused to obtain the fused feature map of each layer. Then, convolution and upsampling operations are performed on the fused feature map of each layer to obtain the first intermediate image.

[0145] The second intermediate feature map is obtained by upsampling the first intermediate feature map of the last level and / or the fused feature maps of each level; the first intermediate image includes a target region and a background region outside the target region, and the position of the target region in the first intermediate image matches the position of the ROI in the original image;

[0146] Based on the target region of the first intermediate image and the original image, the ROI of the original image is identified.

[0147] Optionally, the step of fusing the first intermediate feature map and the second intermediate feature map corresponding to each level to obtain the fused feature map of each level includes:

[0148] The first intermediate feature map, the second intermediate feature map, and the pixel weight vector corresponding to each level are weighted and fused to obtain the fused feature map of each level.

[0149] The pixel weight vector of each level includes the first pixel weight of the first intermediate feature map corresponding to each level and / or the first pixel weight of the second intermediate feature map corresponding to each level.

[0150] Optionally, the encoder of the U-Net network model includes multiple depthwise separable convolutional layers connected along the downsampling path;

[0151] Each of the depthwise separable convolutional layers comprises multiple cascaded depthwise separable convolutional modules; the separable convolutional module includes depthwise convolutional blocks and pointwise convolutional blocks;

[0152] The depthwise separable convolution module is used for depthwise separable convolution operations; the depthwise convolution block is used for depthwise convolution operations; and the pointwise convolution block is used for pointwise convolution operations.

[0153] Optionally, the depthwise separable convolution module includes multiple convolution kernels, each convolution kernel including multiple second pixel weights, and the method further includes:

[0154] Each convolutional kernel is pruned by removing a preset weight from the weights of the second pixel within the convolutional kernels to obtain multiple pruned convolutional kernels.

[0155] The data bit width of the second pixel weights of the multiple pruned convolutional kernels is adjusted to a preset data bit width to obtain the adjusted multiple convolutional kernels;

[0156] The original image and / or the first intermediate feature map are subjected to depthwise separable convolution operations using the adjusted multiple convolution kernels.

[0157] Optionally, the equipment inventory device 1000 further includes: a second processing unit 1005;

[0158] The second processing unit 1005 is used to perform a masking operation on the original image using the target region of the first intermediate image as a mask to obtain a second intermediate image. The second intermediate image includes the ROI of the original image and the background region of the first intermediate image.

[0159] Identify the ROI of the original image based on the second intermediate image.

[0160] Optionally, the background area of ​​the first intermediate image is a black pixel area, and the device inventory device 1000 further includes: a third processing unit 1006;

[0161] The third processing unit 1006 is used to perform grayscale binarization processing on the target area of ​​the first intermediate image to obtain a binarized first intermediate image. The target area of ​​the binarized first intermediate image is a white pixel area, and the background area of ​​the binarized first intermediate image is a black pixel area.

[0162] Contour extraction is performed on the target region of the binarized first intermediate image to obtain the contour information of the target region;

[0163] The target region of the first intermediate image after contour extraction is dilated using a preset structuring element to fill in the pixel gaps and / or breaks in the target region of the first intermediate image.

[0164] Optionally, the monitoring of the visually encoded identifiers contained in the ROI to obtain visually encoded identifier information includes:

[0165] A perspective transformation operation is performed on the ROI in the original image included in the second intermediate image to obtain the perspective-corrected second intermediate image;

[0166] Image monitoring and / or character monitoring are performed on the ROIs in the original image included in the perspective-corrected second intermediate image to obtain the visual coding identification information.

[0167] The equipment inventory device 1000 is capable of implementing each process of the above-described equipment inventory method, with one-to-one correspondence of technical features and achieving the same technical effect. To avoid repetition, it will not be described in detail here.

[0168] This application also provides an electronic device, including: a processor, a memory, and a program stored in the memory and executable on the processor. When the program is executed by the processor, it implements the various processes of the above-described device inventory method embodiments and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0169] For details, see Figure 11 This application also provides an electronic device, including a bus 1101, a transceiver 1102, an antenna 1103, a bus interface 1104, a processor 1105, and a memory 1106.

[0170] The transceiver 1102 is used to acquire the original image of the equipment to be inventoried;

[0171] The processor 1105 is used to identify the region of interest (ROI) of the original image using a U-Net network model, wherein the ROI contains the visual coded identifier of the device to be inventoried.

[0172] Monitor the visually encoded identifiers contained in the ROI to obtain visually encoded identifier information;

[0173] The inventory operation is performed on the equipment to be inventoried based on the visual coding identification information.

[0174] Optionally, the step of identifying the ROI of the original image using the U-Net network model includes:

[0175] The original image is subjected to multi-level depthwise separable convolution and downsampling operations using the encoder of the U-Net network model to generate the first intermediate feature map corresponding to each level;

[0176] Using the decoder of the U-Net network model, the first intermediate feature map and the second intermediate feature map corresponding to each layer are fused to obtain the fused feature map of each layer. Then, convolution and upsampling operations are performed on the fused feature map of each layer to obtain the first intermediate image.

[0177] The second intermediate feature map is obtained by upsampling the first intermediate feature map of the last level and / or the fused feature maps of each level; the first intermediate image includes a target region and a background region outside the target region, and the position of the target region in the first intermediate image matches the position of the ROI in the original image;

[0178] Based on the target region of the first intermediate image and the original image, the ROI of the original image is identified.

[0179] Optionally, the step of fusing the first intermediate feature map and the second intermediate feature map corresponding to each level to obtain the fused feature map of each level includes:

[0180] The first intermediate feature map, the second intermediate feature map, and the pixel weight vector corresponding to each level are weighted and fused to obtain the fused feature map of each level.

[0181] The pixel weight vector of each level includes the first pixel weight of the first intermediate feature map corresponding to each level and / or the first pixel weight of the second intermediate feature map corresponding to each level.

[0182] Optionally, the encoder of the U-Net network model includes multiple depthwise separable convolutional layers connected along the downsampling path;

[0183] Each of the depthwise separable convolutional layers comprises multiple cascaded depthwise separable convolutional modules; the separable convolutional module includes depthwise convolutional blocks and pointwise convolutional blocks;

[0184] The depthwise separable convolution module is used for depthwise separable convolution operations; the depthwise convolution block is used for depthwise convolution operations; and the pointwise convolution block is used for pointwise convolution operations.

[0185] Optionally, the depthwise separable convolution module includes multiple convolution kernels, each convolution kernel including multiple second pixel weights, and the method further includes:

[0186] Each convolutional kernel is pruned by removing a preset weight from the weights of the second pixel within the convolutional kernels to obtain multiple pruned convolutional kernels.

[0187] The data bit width of the second pixel weights of the multiple pruned convolutional kernels is adjusted to a preset data bit width to obtain the adjusted multiple convolutional kernels;

[0188] The original image and / or the first intermediate feature map are subjected to depthwise separable convolution operations using the adjusted multiple convolution kernels.

[0189] Optionally, the processor 1105 is further configured to perform a masking operation on the original image using the target region of the first intermediate image as a mask to obtain a second intermediate image, wherein the second intermediate image includes the ROI of the original image and the background region of the first intermediate image.

[0190] Identify the ROI of the original image based on the second intermediate image.

[0191] Optionally, the background area of ​​the first intermediate image is a black pixel area, and the processor 1105 is further configured to perform grayscale binarization processing on the target area of ​​the first intermediate image to obtain a binarized first intermediate image, wherein the target area of ​​the binarized first intermediate image is a white pixel area, and the background area of ​​the binarized first intermediate image is a black pixel area.

[0192] Contour extraction is performed on the target region of the binarized first intermediate image to obtain the contour information of the target region;

[0193] The target region of the first intermediate image after contour extraction is dilated using a preset structuring element to fill in the pixel gaps and / or breaks in the target region of the first intermediate image.

[0194] Optionally, the monitoring of the visually encoded identifiers contained in the ROI to obtain visually encoded identifier information includes:

[0195] A perspective transformation operation is performed on the ROI in the original image included in the second intermediate image to obtain the perspective-corrected second intermediate image;

[0196] Image monitoring and / or character monitoring are performed on the ROIs in the original image included in the perspective-corrected second intermediate image to obtain the visual coding identification information.

[0197] exist Figure 11 In this document, a bus architecture (represented by bus 1101) is used. Bus 1101 may include any number of interconnected buses and bridges, linking various circuits including one or more processors represented by processor 1105 and memory represented by memory 1106. Bus 1101 may also link various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. Bus interface 1104 provides an interface between bus 1101 and transceiver 1102. Transceiver 1102 may be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by processor 1105 is transmitted over a wireless medium via antenna 1103, which further receives data and transmits it to processor 1105.

[0198] Processor 1105 is responsible for managing bus 1101 and general processing, and can also provide various functions, including timing, peripheral interface, voltage regulation, power management, and other control functions. Memory 1106 can be used to store data used by processor 1105 during operation.

[0199] Optionally, the processor 1105 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or a complex programmable logic device (CPLD).

[0200] This application also provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the various processes of the above-described device inventory embodiments and achieves the same technical effects. To avoid repetition, it will not be described again here. The computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0201] This application also provides a computer program product, including computer instructions. When executed by a processor, these computer instructions implement the various processes of the above-described device inventory method embodiments and achieve the same technical effects. To avoid repetition, they will not be described again here.

[0202] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0203] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0204] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A method for inventorying equipment, characterized in that, The method includes: Acquire the original images of the equipment to be inventoried; The U-Net network model is used to identify the region of interest (ROI) in the original image, and the ROI contains the visual coded identifier of the device to be inventoried. Monitor the visually encoded identifiers contained in the ROI to obtain visually encoded identifier information; The inventory operation is performed on the equipment to be inventoried based on the visual coding identification information.

2. The method according to claim 1, characterized in that, The process of identifying the Region of Interest (ROI) of the original image using the U-Net network model includes: The original image is subjected to multi-level depthwise separable convolution and downsampling operations using the encoder of the U-Net network model to generate the first intermediate feature map corresponding to each level; Using the decoder of the U-Net network model, the first intermediate feature map and the second intermediate feature map corresponding to each layer are fused to obtain the fused feature map of each layer. Then, convolution and upsampling operations are performed on the fused feature map of each layer to obtain the first intermediate image. The second intermediate feature map is obtained by upsampling the first intermediate feature map of the last level and / or the fused feature maps of each level; the first intermediate image includes a target region and a background region outside the target region, and the position of the target region in the first intermediate image matches the position of the ROI in the original image; Based on the target region of the first intermediate image and the original image, the ROI of the original image is identified.

3. The method according to claim 2, characterized in that, The step of fusing the first intermediate feature map and the second intermediate feature map corresponding to each level to obtain the fused feature map of each level includes: The first intermediate feature map, the second intermediate feature map, and the pixel weight vector corresponding to each level are weighted and fused to obtain the fused feature map of each level. The pixel weight vector of each level includes the first pixel weight of the first intermediate feature map corresponding to each level and / or the first pixel weight of the second intermediate feature map corresponding to each level.

4. The method according to claim 3, characterized in that, The encoder of the U-Net network model includes multiple depthwise separable convolutional layers connected along the downsampling path; Each of the depthwise separable convolutional layers comprises multiple cascaded depthwise separable convolutional modules; the separable convolutional module includes depthwise convolutional blocks and pointwise convolutional blocks; The depthwise separable convolution module is used for depthwise separable convolution operations; the depthwise convolution block is used for depthwise convolution operations; and the pointwise convolution block is used for pointwise convolution operations.

5. The method according to claim 4, characterized in that, The depthwise separable convolution module includes multiple convolution kernels, each convolution kernel including multiple second pixel weights, and the method further includes: Each convolutional kernel is pruned by removing a preset weight from the weights of the second pixel within the convolutional kernels to obtain multiple pruned convolutional kernels. The data bit width of the second pixel weights of the multiple pruned convolutional kernels is adjusted to a preset data bit width to obtain the adjusted multiple convolutional kernels; The original image and / or the first intermediate feature map are subjected to depthwise separable convolution operations using the adjusted multiple convolution kernels.

6. The method according to any one of claims 2 to 5, characterized in that, The method further includes: Using the target region of the first intermediate image as a mask, a masking operation is performed on the original image to obtain a second intermediate image. The second intermediate image contains the ROI of the original image and the background region of the first intermediate image. Identify the ROI of the original image based on the second intermediate image.

7. The method according to claim 6, characterized in that, The background region of the first intermediate image is a black pixel region, and the method further includes: The target region of the first intermediate image is subjected to grayscale binarization to obtain the binarized first intermediate image. The target region of the binarized first intermediate image is a white pixel region, and the background region of the binarized first intermediate image is a black pixel region. Contour extraction is performed on the target region of the binarized first intermediate image to obtain the contour information of the target region; The target region of the first intermediate image after contour extraction is dilated using a preset structuring element to fill in the pixel gaps and / or breaks in the target region of the first intermediate image.

8. The method according to claim 6, characterized in that, The monitoring of the visually encoded identifiers contained in the ROI yields visually encoded identifier information, including: A perspective transformation operation is performed on the ROI in the original image included in the second intermediate image to obtain the perspective-corrected second intermediate image; Image monitoring and / or character monitoring are performed on the ROIs in the original image included in the perspective-corrected second intermediate image to obtain the visual coding identification information.

9. An equipment inventory device, characterized in that, The device includes: The acquisition unit is used to acquire the original images of the equipment to be inventoried. The first processing unit is used to identify the region of interest (ROI) of the original image using a U-Net network model, wherein the ROI contains the visual encoding identifier of the original image; A monitoring unit is used to monitor the video encoding identifier of the original image contained in the ROI and obtain visual encoding identifier information; An execution unit is used to perform an inventory operation on the equipment to be inventoried based on the visual coding identification information.

10. An electronic device, characterized in that, include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method as described in any one of claims 1 to 8.

12. A computer program product, characterized in that, Includes computer instructions that, when executed by a processor, implement the steps of the method as described in any one of claims 1 to 8.

Images