Port occupation detection method and device, equipment, storage medium and program product

Abstract

The invention relates to the technical field of artificial intelligence, and provides an occupied port detection method and device, equipment, a storage medium and a product. The method comprises the steps of obtaining a to-be-detected image; inputting the to-be-detected image into the target detection model to obtain a port position detection result; based on the port position detection result, determining a currently occupied port of the optical cable cross connecting cabinet to be subjected to quality inspection; the target detection model is obtained by pre-training an initial model constructed based on a YOLO-EF algorithm based on a preset loss function, a sample detection image and a sample port position detection result; the sample port position detection result is used for adjusting model parameters of the initial model, so that the preset loss function conforms to a preset threshold value. Through the above mode, a semantic segmentation model does not need to be adopted, the problems of low detection efficiency and low accuracy of the semantic segmentation model can be avoided, accurate detection of the ports occupied by the optical cable cross connecting cabinet is realized by using the deep learning model, and improvement of the quality inspection accuracy of the optical cable cross connecting cabinet is facilitated.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an occupied port detection method, device, equipment, storage medium and product. Background Art

[0002] Light Container Delivery is usually placed on the trunk optical cable for branching optical cables, which can realize jumper connection of optical cables and can also be used for testing and maintenance. There are multiple ports on the light container delivery. The ports that are not connected to cables are unoccupied ports, and the ports that are connected to cables are occupied ports. According to the actual occupancy of each port, it can be determined whether the optical cable resources after construction meet the quality inspection requirements.

[0003] In some related technologies, the method for detecting occupied ports of optical cross-boxes is to detect the port occupancy of the optical cross-boxes in real time through a semantic segmentation model based on the optical cross-box equipment images taken by operation and maintenance personnel, so as to determine whether the optical cable resources after construction meet the quality inspection requirements.

[0004] However, the semantic segmentation model is relatively complex and requires multiple conversions to identify the port area. The detection is time-consuming and has a high recognition error rate. Especially when there are multiple ports and multiple port labels in the optical cross-box image, the semantic segmentation model usually finds it difficult to accurately identify the port position and label position in the image, which easily affects the quality inspection accuracy.

[0005] Therefore, the existing occupied port detection method has low detection efficiency and low accuracy, resulting in low quality inspection accuracy of the optical cross-connect box. Summary of the invention

[0006] The embodiments of the present application provide an occupied port detection method, device, equipment, storage medium and product to solve the technical problem that the existing occupied port detection method has low detection efficiency and low accuracy, resulting in low quality inspection accuracy of optical cross-connect boxes.

[0007] In a first aspect, an embodiment of the present application provides an occupied port detection method, comprising: acquiring an image to be detected; the image to be detected is obtained by photographing an optical cross-box to be inspected; the image to be detected is input into a target detection model to obtain a port position detection result of the optical cross-box to be inspected output by the target detection model; based on the port position detection result, determining the current occupied port of the optical cross-box to be inspected; wherein the target detection model is obtained by pre-training an initial model constructed based on a YOLO-EF algorithm based on a preset loss function, a sample detection image and a sample port position detection result corresponding to the sample detection image; the sample port position detection result is used to adjust the model parameters of the initial model so that the preset loss function meets the preset threshold.

[0008] In one embodiment, before the image to be detected is input into the target detection model and the port position detection result of the optical cross box to be inspected is obtained by the output of the target detection model, it also includes: obtaining multiple initial images to be labeled; the initial image to be labeled is obtained by photographing the optical cross box; performing image angle correction processing on the multiple initial images to be labeled to obtain multiple images to be labeled; respectively annotating the port positions and port occupancy of ports at different angles in each image to be labeled, and annotating the label positions of labels at different angles in each image to be labeled to obtain multiple sample detection images and sample port position detection results corresponding to each sample detection image; based on the multiple sample detection images and the sample port position detection results corresponding to each sample detection image, pre-training the initial model constructed based on the YOLO-EF algorithm, and adjusting the model parameters of the initial model based on each sample port position detection result until the preset loss function meets the preset threshold, and obtaining the target detection model.

[0009] In one embodiment, a plurality of initial images to be labeled are obtained, including: obtaining a sample video sequence; the sample video sequence is obtained by shooting a light intersection box, and the sample video sequence includes a plurality of image frames; based on the plurality of image frames, a plurality of inter-frame similarities are determined; each inter-frame similarity is determined based on the distance between any two image frames in the sample video sequence; based on the plurality of inter-frame similarities, the plurality of image frames are clustered to obtain a first clustering result; the first clustering result includes a plurality of first clusters, each of which includes at least one image frame; a first clustering center of each first cluster is determined; based on each first cluster center, a membership of each image frame in each first cluster is determined; based on the membership of each image frame in each first cluster, the first clustering result is optimized to obtain a second clustering result; the second clustering result includes a plurality of second clusters, each of which includes at least one image frame; a target image frame of each second cluster is determined, and the plurality of target image frames are used as a plurality of initial images to be labeled; each target image frame is an image frame in each second cluster that is closest to the second cluster center of the second cluster.

[0010] In one embodiment, the preset loss function is determined based on the confidence loss and the position loss, and the target detection model is pre-trained with the goal of minimizing the confidence loss and maximizing the position loss.

[0011] In one embodiment, the optical cross-link box to be inspected includes at least one port and at least one label; based on the port position detection result, the currently occupied port of the optical cross-link box to be inspected is determined, including: obtaining the port number of each port; the port number of each row in the optical cross-link box to be inspected increases from right to left; based on the port position detection result and the image color feature of the image to be inspected, image extraction is performed on the image to be inspected to obtain the port image of the optical cross-link box to be inspected; the port image is vertically projected to obtain a pixel histogram corresponding to the port image; based on the pixel histogram, the port position of each port of the optical cross-link box to be inspected is determined; based on the image color feature, image extraction is performed on the image to be inspected to obtain the label image of the optical cross-link box to be inspected; based on the label image and the pixel histogram, the label position of each label of the optical cross-link box to be inspected and the label orientation of each label are determined; based on each port number, each port position, each label position and each label orientation, the currently occupied port of the optical cross-link box to be inspected is determined.

[0012] In one embodiment, obtaining an image to be inspected includes: photographing an optical cross-link box to be inspected to obtain a video sequence to be inspected; performing frame extraction processing on the video sequence to be inspected to obtain an initial image to be inspected; determining whether a port of the optical cross-link box to be inspected in the initial image to be inspected is above a label; if the port of the optical cross-link box to be inspected in the initial image to be inspected is not above the label, performing image angle correction processing on the initial image to be inspected until the port of the optical cross-link box to be inspected in the initial image to be inspected is above the label, thereby obtaining the image to be inspected.

[0013] In a second aspect, an embodiment of the present application provides an occupied port detection device, comprising: an acquisition module, used to acquire an image to be detected; the image to be detected is obtained by photographing an optical cross-box to be inspected; a target detection module, used to input the image to be detected into a target detection model, and obtain a port position detection result of the optical cross-box to be inspected output by the target detection model; an occupied port detection module, used to determine the current occupied port of the optical cross-box to be inspected based on the port position detection result; wherein the target detection model is obtained by pre-training an initial model constructed based on the YOLO-EF algorithm based on a preset loss function, a sample detection image and a sample port position detection result corresponding to the sample detection image; the sample port position detection result is used to adjust the model parameters of the initial model so that the preset loss function meets the preset threshold.

[0014] In a third aspect, an embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, any of the above-mentioned occupied port detection methods is implemented.

[0015] In a fourth aspect, an embodiment of the present application provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-mentioned occupied port detection methods.

[0016] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which implements any of the above-mentioned occupied port detection methods when executed by a processor.

[0017] The occupied port detection method, apparatus, equipment, storage medium and product provided in the embodiments of the present application introduce a deep learning algorithm in the port detection process, build a target detection model based on the YOLO-EF algorithm, perform target detection on the image of the optical cross-box to be inspected through the target detection model, obtain the port position detection result of the optical cross-box to be inspected, and determine the current occupied port of the optical cross-box to be inspected according to the port position detection result. There is no need to adopt a semantic segmentation model, which can avoid the problems of low detection efficiency and low accuracy of the semantic segmentation model, and uses the deep learning model to realize accurate detection of occupied ports of the optical cross-box, which is beneficial to improving the quality inspection accuracy of the optical cross-box. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present application or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 This is one of the flow charts of the occupied port detection method provided in the embodiment of the present application.

[0020] Figure 2 This is the second flow chart of the occupied port detection method provided in the embodiment of the present application.

[0021] Figure 3 It is a flowchart of the video clustering method provided in an embodiment of the present application.

[0022] Figure 4 It is a flowchart of the port position detection provided in the embodiment of the present application.

[0023] Figure 5 It is a flowchart of port position determination provided in an embodiment of the present application.

[0024] Figure 6 Schematic diagram of a white pixel histogram provided in an embodiment of the present application.

[0025] Figure 7It is a structural diagram of an occupied port detection device provided in an embodiment of the present application.

[0026] Figure 8 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0028] See also Figure 1 , Figure 1 This is one of the flow charts of the occupied port detection method provided in the embodiment of the present application. In the embodiment of the present application, the occupied port detection method includes steps S110 to S130, and each step is as follows: S110: Acquire an image to be detected.

[0029] The image to be inspected is obtained by photographing the optical inspection box to be inspected.

[0030] See also Figure 2 , Figure 2 This is the second flow chart of the occupied port detection method provided in the embodiment of the present application.

[0031] Specifically, Figure 2 As shown in the figure, the video materials of the optical cross box to be inspected are collected and processed: the optical cross box to be inspected is photographed through the APP (application) to complete the collection of on-site videos to ensure the authenticity and non-tamperability of the video; the video sequence to be inspected is obtained through the key frame extraction process based on video clustering, and the initial image to be inspected is extracted from the video sequence to be inspected for subsequent port position detection and occupied port confirmation; the initial image to be inspected is processed for image angle correction (i.e. Figure 2 ), and obtain the image to be detected.

[0032] S120: Input the image to be inspected into the target detection model to obtain the port position detection result of the optical cross-connect box to be inspected output by the target detection model.

[0033] S130: Based on the port position detection result, determine the currently occupied port of the optical cross-connect box to be inspected.

[0034] Among them, the target detection model is obtained by pre-training the initial model constructed based on the YOLO-EF algorithm based on the preset loss function, sample detection image and the sample port position detection results corresponding to the sample detection image; the sample port position detection results are used to adjust the model parameters of the initial model so that the preset loss function meets the preset threshold.

[0035] The occupied port detection method provided in the embodiment of the present application introduces a deep learning algorithm in the port detection process, builds a target detection model based on the YOLO-EF algorithm, performs target detection on the image of the optical cross-box to be inspected through the target detection model, obtains the port position detection result of the optical cross-box to be inspected, and determines the current occupied port of the optical cross-box to be inspected according to the port position detection result. There is no need to adopt a semantic segmentation model, which can avoid the problems of low detection efficiency and low accuracy of the semantic segmentation model, and uses the deep learning model to realize accurate detection of occupied ports of the optical cross-box, which is beneficial to improving the quality inspection accuracy of the optical cross-box.

[0036] In some embodiments, before the image to be detected is input into the target detection model and the port position detection result of the optical cross box to be inspected is obtained by the output of the target detection model, it also includes: obtaining multiple initial images to be labeled; the initial image to be labeled is obtained by photographing the optical cross box; performing image angle correction processing on the multiple initial images to be labeled to obtain multiple images to be labeled; respectively annotating the port positions and port occupancy of ports at different angles in each image to be labeled, and annotating the label positions of labels at different angles in each image to be labeled to obtain multiple sample detection images and sample port position detection results corresponding to each sample detection image; based on the multiple sample detection images and the sample port position detection results corresponding to each sample detection image, pre-training the initial model constructed based on the YOLO-EF algorithm, and adjusting the model parameters of the initial model based on each sample port position detection result until the preset loss function meets the preset threshold, and obtaining the target detection model.

[0037] It is understandable that before performing port position detection, a target detection model needs to be trained.

[0038] Specifically, the APP is used to record sample videos of the optical cross-box on site to ensure that the collected sample video sequence can reflect the actual on-site situation of the optical cross-box; since the sample video sequence includes multiple image frames, and there may be a large number of identical or similar image frames in the sample video sequence, the video clustering algorithm can be used to extract some key frames from the sample video sequence as the initial images to be labeled, and the identical or similar image frames are eliminated. This provides the required image data for subsequent steps, while reducing the statistics and collection of invalid samples and reducing the data processing amount of the model.

[0039] Furthermore, image angle correction processing is performed on the multiple initial images to be labeled to obtain multiple images to be labeled.

[0040] Specifically, since different operation and maintenance personnel may take photos of the optical cross-box at different angles, the initial image to be labeled needs to be corrected: if the port of the optical cross-box in the initial image to be labeled is not above the label, the initial image to be labeled is rotationally corrected (for example, rotated 90°, 180° or 270°), etc.; generally, when the port of the optical cross-box is on top and the label is on the bottom, it is the front angle of the optical cross-box device. At this time, the port numbers of each row on the optical cross-box increase from right to left, which also provides a prerequisite for finally determining the occupied port number; when the port position of the optical cross-box is not in a horizontal position, it may affect the accuracy of determining the position of each port. Therefore, the image needs to be corrected at a horizontal angle. For example, the Hough transform algorithm is used to find the average tilt angle of the horizontal line in the optical cross-box port position area, and then the image is rotationally corrected according to the average tilt angle to obtain multiple images to be labeled after image angle correction.

[0041] Furthermore, the port positions and port occupancy are annotated for ports at different angles in each image to be annotated, and the label positions are annotated for labels at different angles in each image to be annotated, so as to obtain multiple sample detection images and sample port position detection results corresponding to each sample detection image.

[0042] Specifically, for each image to be annotated, graphic tools are used to annotate the port area and label area of ​​the optical cross-box in the image to be annotated with port positions, port occupancy and label positions; after all the images to be annotated have completed port position annotation and label position annotation, multiple sample detection images and sample port position detection results corresponding to each sample detection image can be obtained.

[0043] Among them, in the sample annotation stage, at least 10,000 images are marked, and each port in each row of the optical intersection box in the image needs to be labeled with two standards: "occupied" and "not occupied", and the optical intersection boxes at different angles need to be labeled at the same time.

[0044] Furthermore, based on multiple sample detection images and the sample port position detection results corresponding to each sample detection image, the initial model constructed based on the YOLO-EF algorithm is pre-trained, and the model parameters of the initial model are adjusted based on each sample port position detection result until the preset loss function meets the preset threshold, and the target detection model is obtained.

[0045] In some embodiments, obtaining a plurality of initial images to be labeled includes: obtaining a sample video sequence; the sample video sequence is obtained by shooting a light intersection box, and the sample video sequence includes a plurality of image frames; based on the plurality of image frames, determining a plurality of similarities between frames; each similarity between frames is determined based on the distance between any two image frames in the sample video sequence; based on the plurality of similarities between frames, clustering the plurality of image frames to obtain a first clustering result; the first clustering result includes a plurality of first clusters, and each first cluster includes at least one image frame; determining a first cluster center of each first cluster; based on each first cluster center, determining a membership of each image frame in each first cluster; optimizing the first clustering result based on the membership of each image frame in each first cluster to obtain a second clustering result; the second clustering result includes a plurality of second clusters, and each second cluster includes at least one image frame; determining a target image frame of each second cluster, and using the plurality of target image frames as a plurality of initial images to be labeled; each target image frame is an image frame in each second cluster that is closest to a second cluster center of the second cluster.

[0046] See also Figure 3 , Figure 3 It is a flowchart of the video clustering method provided in an embodiment of the present application.

[0047] like Figure 3 As shown, after obtaining a sample video sequence including multiple image frames, it is necessary to calculate the similarity between frames to perform video clustering.

[0048] Specifically, the sample video sequence includes multiple image frames. For any two image frames in the sample video sequence, the Euclidean distance between the two image frames can be calculated. The Euclidean distance is used to represent the inter-frame similarity between the two image frames. The smaller the Euclidean distance, the higher the inter-frame similarity, and the larger the Euclidean distance, the lower the inter-frame similarity.

[0049] Among them, the Euclidean distance between two image frames is defined as: ; Wherein, A represents image frame A; B represents image frame B; Represents the inter-frame similarity between image frame A and image frame B; Indicates the number of rows of pixels in the image frame; Indicates the number of columns of pixels in the image frame; Indicates the first Line Column pixels; Indicates the image frame B Line Column of pixels.

[0050] In the above formula, Represents the square of the difference in position between two pixels in space; Represents the sum of squared position differences of all pixels; It represents the square root of the sum of the squares of the position differences of all pixels, i.e. the Euclidean distance. The summation range can be defined as the entire image frame as above, or it can be defined as an area of ​​the same size or a point in the image frame. The latter two represent the Euclidean distance under the local feature metric.

[0051] Similar frames can be determined by calculating the similarity values ​​of all pixels in the image frame. The threshold of the similarity value MSE can be set to 10. If it is greater than 10, it is considered dissimilar. If the similarity of the pixels in the image frame exceeds 90%, the two image frames are considered to be similar.

[0052] For each image frame in the sample video sequence, the inter-frame similarity between the image frame and other image frames can be calculated in the above manner, thereby obtaining multiple inter-frame similarities.

[0053] Furthermore, based on the similarities between the multiple frames, the multiple image frames are clustered by a hierarchical clustering algorithm to obtain a first clustering result, the first clustering result includes multiple first clusters, and each first cluster includes at least one image frame.

[0054] Specifically, an agglomerative hierarchical clustering algorithm is used to perform initial clustering on the image frames in the sample video sequence. The main idea is: first, each sample to be clustered is regarded as a class, and then, according to the size of the similarity between frames, similar classes are merged into one class until the termination condition is met.

[0055] To facilitate understanding of the video clustering method of this embodiment, 100 image frames are taken as an example: Assuming that there are 100 image frames in the sample video sequence, each image frame can be regarded as a sample to be clustered, then there are 100 samples to be clustered, and the basic steps of clustering are: (1) Determine the termination condition: Assume that the number of key frames to be extracted is 10, that is, the termination condition is that the number of clusters is 10.

[0056] In the clustering process, the mean of the feature vector is recorded as M, the variance of the feature vector is recorded as V, the key frame is one of the image frames among all similar image frames, and D={d i |i=1, 2, 3, ..., N} represents a distance vector. In this embodiment, the final value is equal to the inter-frame distance d(Fa, Fb)>M+2*the number of image frames.

[0057] (2) Initialization: Classify each sample to be clustered into a separate category, and calculate and obtain the inter-frame similarity between each two categories (i.e., two samples to be clustered).

[0058] Among them, the similarity between frames is expressed by Euclidean distance.

[0059] (3) Find the two samples to be clustered with the closest Euclidean distance (that is, find the two samples to be clustered with the highest inter-frame similarity), and classify these two samples to be clustered into one category.

[0060] (4) Recalculate the inter-frame similarity between the newly generated class and each old class.

[0061] (5) Repeat steps (3) and (4) until the number of clusters reaches 10, obtaining the first clustering result and ending clustering.

[0062] At this time, the first clustering result includes 10 first clusters, and each first cluster includes at least one image frame.

[0063] Furthermore, the clustering results are optimized using a weighted fuzzy K-Means clustering algorithm: the first cluster center of each first cluster is determined, and based on each first cluster center, the membership of each image frame in each first cluster is determined, and then based on the membership of each image frame in each first cluster, the first clustering result is optimized to obtain a second clustering result, the second clustering result includes multiple second clusters, and each second cluster includes at least one image frame.

[0064] Furthermore, a target image frame of each second cluster is determined, and the multiple target image frames are used as multiple initial images to be labeled.

[0065] Each target image frame is an image frame in each second cluster that is closest to the second cluster center of the second cluster.

[0066] Specifically, the results of hierarchical clustering are used as the input of the K-Means clustering algorithm, and the cluster center of each cluster generated by hierarchical clustering is used as the initial cluster center of the K-Means clustering algorithm. This can avoid the influence of randomly generated initial cluster centers on the clustering results of the K-Means clustering algorithm.

[0067] Here we continue to take the first clustering result containing 10 first clusters as an example, and the specific steps of cluster optimization are as follows: (1) Determine the 10 cluster centers generated by the hierarchical clustering algorithm, that is, the average value of each cluster object.

[0068] (2) Calculate the membership degree of each sample (i.e., image frame) in the 10 first clusters.

[0069] The calculation formula of membership degree is as follows: ; in, Representation sample Cluster Center The degree of membership; is the number of cluster centers; is the fuzziness coefficient, which is usually greater than 1 (for example, =2); It is a sample With cluster center The Euclidean distance between It is a sample With cluster center The Euclidean distance between .

[0070] (3) Combined with the local density of samples, the cluster center is updated using the weighted average method.

[0071] (4) Repeat steps (2) and (3) until the objects in each second cluster no longer change, obtain the second clustering result, and end the clustering.

[0072] At this time, the second clustering result also includes 10 second clusters (ie, 10 optimized clusters), and each second cluster includes at least one image frame.

[0073] (5) Calculate the cluster center of each second cluster. For each second cluster, output the image frame with the closest Euclidean distance to the cluster center of the second cluster as the key frame (i.e., the target image frame). Then, 10 target image frames can be obtained as 10 initial images to be labeled.

[0074] In some embodiments, the preset loss function is determined based on confidence loss and position loss, and the target detection model is pre-trained with the goal of minimizing confidence loss and maximizing position loss.

[0075] See also Figure 4 , Figure 4 It is a flowchart of the port position detection provided in the embodiment of the present application.

[0076] like Figure 4 As shown, after obtaining multiple images to be labeled, for each image to be labeled, the port area and label area of ​​the optical cross-box in the image to be labeled can be labeled with port positions and label positions using graphic tools; after all images to be labeled have completed port position labeling and label position labeling, multiple sample detection images and sample port position detection results corresponding to each sample detection image can be obtained.

[0077] Furthermore, the initial model built based on the YOLO-EF (YOLO with EnhancedFocal and Efficient IOU Loss) algorithm is pre-trained through labeled sample detection images, and the model parameters are adjusted according to each detection result, so that the trained model can recognize the optical cross-box ports and labels, and the preset loss function meets the preset threshold, and the characteristic value of the target area is obtained, thereby obtaining the target detection model.

[0078] In this embodiment, the existing YOLO algorithm is improved and an improved YOLO algorithm, namely, the YOLO-EF algorithm, is proposed.

[0079] Among them, the preset loss function of the target detection model built based on the YOLO-EF algorithm consists of two parts: confidence loss and position loss. These losses are used to measure the difference between the predicted box and the real box in the target detection task: (1) Confidence loss function: Generally, the confidence loss function can use the logistic regression loss function, such as the commonly used cross entropy loss function. Cross entropy is a common indicator for measuring the difference between two probability distributions and is often used as a loss function for classification tasks.

[0080] In this embodiment, the confidence loss function adopts the improved cross entropy loss function: by introducing a modulation factor to reduce the weight of easy-to-classify samples and increase the weight of difficult-to-classify samples, the focal loss (Focal Loss) of the improved cross entropy loss function can be obtained. For the binary classification problem, the focal loss It can be expressed as: ; in, is the predicted probability of the target detection model for the positive class; is the weight of the positive sample, which is used to balance the ratio of positive and negative samples; is a hyperparameter used to adjust the difficulty weight of the sample.

[0081] For multi-classification problems, focal loss Can be expanded to: ; in, is the predicted probability of the target detection model for category c, and M is the total number of categories.

[0082] Specifically, the confidence loss of this embodiment may adopt any one of the two focal loss functions mentioned above.

[0083] (2) Position loss function: The position loss function can be expressed using the EIOU loss function, which is a bounding box regression loss function for object detection. It is an extension of the traditional IOU loss and takes into account the differences in overlapping areas, center point distances, and aspect ratios. The calculation formula for the EIOU loss function is as follows: ; in, , the intersection area is the area of ​​the intersection of the predicted box and the true box, and the union area is the total area of ​​the predicted box and the true box minus the intersection area; It is the Euclidean distance between the center point of the predicted box and the center point of the real box; is the diagonal length of the minimum enclosing rectangle containing the predicted box and the true box; is the absolute value of the difference between the width and height of the prediction box; is the absolute value of the difference between the width and height of the real box; is the width of the minimum enclosing rectangle; is the height of the minimum enclosing rectangle.

[0084] The target detection model aims to minimize the confidence loss and maximize the position loss. By optimizing the model parameters, the EIOU value can be increased and the cross entropy loss value can be reduced to achieve the best optimization result.

[0085] Furthermore, the trained target detection model and feature values ​​can be used to perform port detection and recognition on the images of the optical cross-box to be inspected. Since the port position and label image features are obvious, the model can detect the target area with better performance and accuracy.

[0086] In some embodiments, the optical cross-border box to be inspected includes at least one port and at least one label; based on the port position detection result, the currently occupied port of the optical cross-border box to be inspected is determined, including: obtaining the port number of each port; the port number of each row in the optical cross-border box to be inspected increases from right to left; based on the port position detection result and the image color characteristics of the image to be inspected, image extraction is performed on the image to be inspected to obtain the port image of the optical cross-border box to be inspected; the port image is vertically projected to obtain a pixel histogram corresponding to the port image; based on the pixel histogram, the port position of each port of the optical cross-border box to be inspected is determined; based on the image color characteristics, image extraction is performed on the image to be inspected to obtain the label image of the optical cross-border box to be inspected; based on the label image and the pixel histogram, the label position of each label of the optical cross-border box to be inspected and the label orientation of each label are determined; based on each port number, each port position, each label position and each label orientation, the currently occupied port of the optical cross-border box to be inspected is determined.

[0087] See also Figure 5 and Figure 6 , Figure 5 is a schematic diagram of a flow chart of port position determination provided by an embodiment of the present application, Figure 6 Schematic diagram of a white pixel histogram provided in an embodiment of the present application.

[0088] like Figure 5 As shown, the port position detection result includes the port positions of all ports on the optical cross-connect box. After the port position detection result is obtained, the specific position of the port and the specific position of the label can be confirmed according to the port position detection result.

[0089] Specifically, the port number of each port in the optical cross-box to be inspected is determined, and based on the port position detection result and the image color characteristics of the image to be inspected, image extraction is performed on the image to be inspected to obtain the port image of the optical cross-box to be inspected, and the port image is vertically projected to obtain a pixel histogram corresponding to the port image, and finally, based on the pixel histogram, the port position of each port of the optical cross-box to be inspected is determined.

[0090] First, the process of determining the port position of each port on the optical delivery box to be inspected is as follows: Generally, the blue and green image parts in the image to be inspected are port images, and the white image part is the label image (because most labels are white); after all the ports of the optical delivery box to be inspected are identified in the image to be inspected of the optical delivery box to be inspected, the blue and green image parts of the optical delivery box ports can be extracted using the image color features, and the starting and ending positions of each port area on the optical delivery box to be inspected can be determined by the vertical projection method.

[0091] like Figure 6 As shown in the figure, since there is a certain gap between the ports of the optical cross-box to be inspected, after vertical projection of the port image, several independent white areas can be obtained ( Figure 6 The white part in the image), each white area is the projection position of a port; the number of white pixels in each column under vertical projection is plotted into a histogram, and the white pixel histogram corresponding to the port image can be obtained. Then, the pixels in each column of the white pixel histogram are calculated: if the current column pixel (i.e. Figure 6 The rightmost black part of the port does not contain white pixels, which is recorded as the starting position of the port; moving from right to left, if the current column does not contain white pixels, but the previous column (that is, the first column connected to the right of the current column) contains white pixels, it is recorded as the end position of the port (that is, Figure 6 the leftmost black part of the current column); if the current column contains white pixels, it means that the pixels in this column belong to the port area.

[0092] According to the starting and ending positions of each port area, the port position of each port on the optical delivery box to be inspected can be located. At the same time, since the port numbers of the ports in each row of the optical delivery box to be inspected start from 1 by default and increase from right to left, after obtaining the port number of each port, the port position of each port on the optical delivery box to be inspected and the port number corresponding to each port position can be confirmed.

[0093] Furthermore, based on the image color features, image extraction is performed on the image to be inspected to obtain the label image of the optical cross-link box to be inspected; based on the label image and the pixel histogram, the label position and label orientation of each label in the optical cross-link box to be inspected are determined.

[0094] Specifically, after confirming the port position of each port on the optical cross-box to be inspected and the port number corresponding to each port position, it is necessary to determine the label position of each label and the label orientation of each label: the occupied port is connected to the cable, and there is a label at the connection between the cable and the port. The white part representing the label area in the image to be inspected is extracted to obtain the label image of the optical cross-box to be inspected. Combined with the projected white pixel histogram, the number of pixels on the left and right sides of each label is calculated. For each label, the side with fewer pixels is the connection position between the label and the port. If the connection is on the right side of the label, the label orientation is the label facing left; if the connection is on the left side of the label, the label orientation is the label facing right. If the connection is on the right side of the label, the label faces left; if the connection is on the left side of the label, the label faces right.

[0095] Furthermore, based on each port number, each port position, each label position and each label orientation, the currently occupied port of the optical cross-connect box to be inspected is determined.

[0096] Specifically, according to each label position and each label orientation extracted by the above method, the occupied port position is determined: if the label is facing left, the port with the closest horizontal distance to the right side of the label connection is the occupied port; if the label is facing right, the port with the closest horizontal distance to the left side of the label connection is the occupied port; combined with the port number corresponding to each port position determined in the early stage, the port number of the currently occupied port of the optical delivery box to be inspected is determined, and the detection results are fed back to the operation and maintenance personnel in real time.

[0097] In some embodiments, obtaining an image to be inspected includes: photographing an optical cross-box to be inspected to obtain a video sequence to be inspected; performing frame extraction processing on the video sequence to be inspected to obtain an initial image to be inspected; determining whether a port of the optical cross-box to be inspected in the initial image to be inspected is above the label; if the port of the optical cross-box to be inspected in the initial image to be inspected is not above the label, performing image angle correction processing on the initial image to be inspected until the port of the optical cross-box to be inspected in the initial image to be inspected is above the label, thereby obtaining the image to be inspected.

[0098] Understandably, since different operation and maintenance personnel may take photos of the optical cross-box at different angles, the initial image to be detected needs to be corrected: if the port of the optical cross-box in the initial image to be detected is not above the label, the initial image to be detected is subjected to rotation correction (for example, rotated 90°, 180° or 270°), etc.; generally, when the port of the optical cross-box is on top and the label is on the bottom, it is the front angle of the optical cross-box device. At this time, the port numbers of each row on the optical cross-box increase from right to left, which also provides a prerequisite for finally determining the occupied port number; when the port position of the optical cross-box is not in a horizontal position, it may affect the accuracy of determining the position of each port, so the image needs to be corrected at a horizontal angle, such as using the Hough transform algorithm to find the average tilt angle of the horizontal line in the optical cross-box port position area, and then performing rotation correction on the image according to the average tilt angle to obtain the image to be detected after image angle correction.

[0099] In view of the slow speed, low accuracy and poor real-time performance of identifying the target area in the prior art, this embodiment proposes a solution based on a deep learning algorithm for detecting the occupied ports of the optical cross-box. The occupied port detection method provided in the embodiment of the present application uses the YOLO-EF target detection algorithm to capture the video into pictures, and combines the image processing method to quickly identify the optical cross-box port area and label area, divide the specific position of each port, and finally determine the specific position of the occupied port of the optical cross-box according to the vertical projection method, and return the detection result in real time. This solution decomposes complex problems into simple problems with multiple steps. The processing process of each step is simple and lightweight, and the deep learning algorithm has the advantages of fast and accurate identification of the target area, and has different degrees of improvement in computing speed and accuracy. Secondly, the operation and maintenance personnel can receive the pre-inspection results in real time on site, and make corresponding adjustments to the pre-inspection results to improve the construction quality; at the same time, for image quality problems such as insufficient brightness and blurred images, corresponding optimization processing can be made to improve the recognizability of the image, thereby reducing the workload of manual intervention.

[0100] The present application also provides an occupied port detection device. Figure 7 , Figure 7 7 is a schematic diagram of the structure of an occupied port detection device provided in an embodiment of the present application. In the embodiment of the present application, the occupied port detection device includes an acquisition module 710 , a target detection module 720 and an occupied port detection module 730 .

[0101] The acquisition module 710 is used to acquire the image to be detected.

[0102] The image to be inspected is obtained by photographing the optical inspection box to be inspected.

[0103] The target detection module 720 is used to input the image to be detected into the target detection model to obtain the port position detection result of the optical cross-link box to be inspected output by the target detection model.

[0104] The occupied port detection module 730 is used to determine the currently occupied port of the optical cross-connect box to be inspected based on the port position detection result.

[0105] Among them, the target detection model is obtained by pre-training the initial model constructed based on the YOLO-EF algorithm based on the preset loss function, sample detection image and the sample port position detection results corresponding to the sample detection image; the sample port position detection results are used to adjust the model parameters of the initial model so that the preset loss function meets the preset threshold.

[0106] In some embodiments, the target detection module 720 is used to obtain multiple initial images to be labeled; the initial images to be labeled are obtained by photographing the light intersection box; the multiple initial images to be labeled are subjected to image angle correction processing to obtain multiple images to be labeled; the ports at different angles in each image to be labeled are respectively labeled with port positions and port occupancy, and the labels at different angles in each image to be labeled are labeled with label positions to obtain multiple sample detection images and sample port position detection results corresponding to each sample detection image; based on the multiple sample detection images and the sample port position detection results corresponding to each sample detection image, an initial model constructed based on the YOLO-EF algorithm is pre-trained, and the model parameters of the initial model are adjusted based on each sample port position detection result until a preset loss function meets a preset threshold, thereby obtaining a target detection model.

[0107] In some embodiments, the target detection module 720 is used to obtain a sample video sequence; the sample video sequence is obtained by shooting a light intersection box, and the sample video sequence includes multiple image frames; based on the multiple image frames, multiple frame-to-frame similarities are determined; each frame-to-frame similarity is determined based on the distance between any two image frames in the sample video sequence; based on the multiple frame-to-frame similarities, the multiple image frames are clustered to obtain a first clustering result; the first clustering result includes multiple first clusters, each first cluster includes at least one image frame; the first clustering center of each first cluster is determined; based on each first cluster center, the membership of each image frame in each first cluster is determined; based on the membership of each image frame in each first cluster, the first clustering result is optimized to obtain a second clustering result; the second clustering result includes multiple second clusters, each second cluster includes at least one image frame; the target image frame of each second cluster is determined, and the multiple target image frames are used as multiple initial images to be labeled; each target image frame is the image frame in each second cluster that is closest to the second cluster center of the second cluster.

[0108] In some embodiments, the preset loss function is determined based on confidence loss and position loss, and the target detection model is pre-trained with the goal of minimizing confidence loss and maximizing position loss.

[0109] In some embodiments, the optical delivery box to be inspected includes at least one port and at least one label.

[0110] The occupied port detection module 730 is used to obtain the port number of each port; the port numbers of each row in the optical cross-link box to be inspected are increased from right to left in sequence; based on the port position detection result and the image color characteristics of the image to be detected, image extraction is performed on the image to be detected to obtain the port image of the optical cross-link box to be inspected; the port image is vertically projected to obtain the pixel histogram corresponding to the port image; based on the pixel histogram, the port position of each port of the optical cross-link box to be inspected is determined; based on the image color characteristics, image extraction is performed on the image to be detected to obtain the label image of the optical cross-link box to be inspected; based on the label image and the pixel histogram, the label position of each label of the optical cross-link box to be inspected and the label orientation of each label are determined; based on each port number, each port position, each label position and each label orientation, the currently occupied port of the optical cross-link box to be inspected is determined.

[0111] In some embodiments, the acquisition module 710 is used to shoot the optical cross-box to be inspected to obtain a video sequence to be inspected; perform frame extraction processing on the video sequence to be inspected to obtain an initial image to be inspected; determine whether the port of the optical cross-box to be inspected in the initial image to be inspected is above the label; if the port of the optical cross-box to be inspected in the initial image to be inspected is not above the label, perform image angle correction processing on the initial image to be inspected until the port of the optical cross-box to be inspected in the initial image to be inspected is above the label, so as to obtain the image to be inspected.

[0112] The present application also provides an electronic device, Figure 8 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application, such as Figure 8 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830 and a communication bus 840, wherein the processor 810, the communication interface 820 and the memory 830 communicate with each other through the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute the occupied port detection method.

[0113] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0114] The embodiment of the present application also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the occupied port detection method provided by the above methods is implemented.

[0115] An embodiment of the present application also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the occupied port detection method provided by the above methods.

[0116] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0117] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0118] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting an occupied port, characterized in that: include: Acquire an image to be detected; the image to be detected is obtained by photographing the optical cross box to be inspected; Inputting the image to be inspected into a target detection model to obtain a port position detection result of the optical cross-connect box to be inspected output by the target detection model; Based on the port position detection result, determine the currently occupied port of the optical cross-connect box to be inspected; Among them, the target detection model is obtained by pre-training an initial model constructed based on the YOLO-EF algorithm based on a preset loss function, a sample detection image and a sample port position detection result corresponding to the sample detection image; the sample port position detection result is used to adjust the model parameters of the initial model so that the preset loss function meets the preset threshold.

2. The occupied port detection method according to claim 1, characterized in that: Before inputting the image to be detected into the target detection model and obtaining the port position detection result of the optical cross-connect box to be inspected output by the target detection model, the method further includes: Acquire a plurality of initial images to be annotated; the initial images to be annotated are obtained by photographing a light intersection box; Performing image angle correction processing on the plurality of initial images to be labeled to obtain a plurality of images to be labeled; Respectively mark the port positions and port occupancy of ports at different angles in each of the images to be marked, and mark the label positions of labels at different angles in each of the images to be marked, to obtain a plurality of sample detection images and a sample port position detection result corresponding to each of the sample detection images; Based on the multiple sample detection images and the sample port position detection results corresponding to each of the sample detection images, the initial model constructed based on the YOLO-EF algorithm is pre-trained, and the model parameters of the initial model are adjusted based on each of the sample port position detection results until the preset loss function meets the preset threshold, so as to obtain the target detection model.

3. The occupied port detection method according to claim 2, characterized in that: The step of obtaining a plurality of initial images to be annotated includes: Acquire a sample video sequence; the sample video sequence is obtained by shooting a light cross-box, and the sample video sequence includes a plurality of image frames; Based on the plurality of image frames, determining a plurality of inter-frame similarities; each of the inter-frame similarities is determined based on the distance between any two image frames in the sample video sequence; Based on the plurality of inter-frame similarities, clustering the plurality of image frames to obtain a first clustering result; the first clustering result includes a plurality of first clusters, each of the first clusters includes at least one image frame; determining a first cluster center of each of the first clusters; Determining, based on each of the first cluster centers, a degree of membership of each of the image frames in each of the first clusters; Based on the membership degree of each of the image frames in each of the first clusters, the first clustering result is optimized to obtain a second clustering result; the second clustering result includes a plurality of second clusters, and each of the second clusters includes at least one of the image frames; A target image frame of each second cluster is determined, and a plurality of the target image frames are used as a plurality of the initial images to be labeled; each of the target image frames is an image frame in each second cluster that is closest to the second cluster center of the second cluster.

4. The occupied port detection method according to claim 1, characterized in that: The preset loss function is determined based on the confidence loss and the position loss, and the target detection model is pre-trained with the goal of minimizing the confidence loss and maximizing the position loss.

5. The occupied port detection method according to claim 1, characterized in that: The optical delivery box to be inspected includes at least one port and at least one label; The determining, based on the port position detection result, the currently occupied port of the optical cross-connect box to be inspected, comprises: Obtaining the port number of each port; the port number of each row in the optical delivery box to be inspected increases from right to left; Based on the port position detection result and the image color feature of the image to be detected, image extraction is performed on the image to be detected to obtain the port image of the optical cross-connect box to be inspected; Performing vertical projection on the port image to obtain a pixel histogram corresponding to the port image; Determine the port position of each port of the optical cross-connect box to be inspected based on the pixel histogram; Based on the image color features, image extraction is performed on the image to be detected to obtain a label image of the optical cross-box to be inspected; Determine the label position and label orientation of each label in the optical cross-check box to be inspected based on the label image and the pixel histogram; Based on each of the port numbers, each of the port positions, each of the label positions and each of the label orientations, a currently occupied port of the optical cross-connect box to be inspected is determined.

6. The occupied port detection method according to any one of claims 1 to 5, characterized in that: The step of acquiring the image to be detected comprises: Shooting the optical cross box to be inspected to obtain a video sequence to be inspected; Performing frame extraction processing on the video sequence to be detected to obtain an initial image to be detected; Determine whether the port of the optical delivery box to be inspected in the initial image to be inspected is above the label; If the port of the optical cross-box to be inspected in the initial image to be inspected is not above the label, the initial image to be inspected is subjected to image angle correction processing until the port of the optical cross-box to be inspected in the initial image to be inspected is above the label to obtain the image to be inspected.

7. An occupied port detection device, characterized in that: include: An acquisition module is used to acquire an image to be detected; the image to be detected is obtained by photographing the optical cross box to be inspected; A target detection module, used for inputting the image to be detected into a target detection model, and obtaining a port position detection result of the optical cross-connect box to be inspected output by the target detection model; An occupied port detection module, used to determine the currently occupied port of the optical cross-connect box to be inspected based on the port position detection result; Among them, the target detection model is obtained by pre-training an initial model constructed based on the YOLO-EF algorithm based on a preset loss function, a sample detection image and a sample port position detection result corresponding to the sample detection image; the sample port position detection result is used to adjust the model parameters of the initial model so that the preset loss function meets the preset threshold.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the occupied port detection method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the occupied port detection method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the occupied port detection method according to any one of claims 1 to 6 is implemented.

Images