Container seal identification method and system

Abstract

The application discloses a container lead seal identification method and system; the method comprises the following steps: collecting a container image and performing pretreatment; detecting a lock buckle and a lead seal of the pretreated image through a target detection neural network, and extracting a lock buckle area image detected by the target detection neural network; the target detection neural network is configured to output a detection frame and a corresponding category probability value; performing classification analysis on whether the lock buckle area image exists a lead seal through a classification neural network; the classification neural network is configured to receive the lock buckle area image and output a binary classification result and a confidence degree of whether the lead seal exists; and comprehensively judging according to analysis results of the target detection neural network and the classification neural network to determine a lead seal state and output. The application can accurately identify the lead seal on the container and improve work efficiency.

Description

Technical Field

[0001] This application relates to the field of image recognition, specifically to a method and system for identifying lead seals on containers. Background Technology

[0002] In the container shipping industry, ensuring cargo safety is paramount, and lead seals, as a key component in protecting containerized cargo, are of great significance for accurate detection. With the booming development of global trade and the continuous increase in container throughput, container movement has become increasingly frequent. Against this backdrop, efficiently and accurately identifying the status of container lead seals has become a crucial factor in ensuring cargo transportation safety and improving logistics efficiency.

[0003] In the past, the inspection of container seals mainly relied on manual on-site operations. Workers had to personally go to the container yard and inspect the seals of each container to confirm their integrity and whether they had been opened abnormally. While this manual inspection method could ensure accuracy to a certain extent, its inefficiency became increasingly apparent with the continuous increase in the number of containers and the faster flow of goods. In addition, some attempts were made to use image recognition technology to detect seals, but directly locating seals in images is difficult because the target is relatively small in an image. To prevent missed detections, the system recall rate needs to be improved; however, this reduces the system's accuracy. Relying on a single detection network or classification neural network is insufficient to meet the needs of seal recognition.

[0004] Consequently, due to the significant increase in container throughput and the frequency of movement, the efficiency of manual inspection is far from meeting actual needs, severely impacting the overall efficiency of logistics and transportation. Simultaneously, harsh on-site inspection environments, such as high temperatures, high humidity, and strong winds, affect worker efficiency and expose them to significant safety risks during inspections. More importantly, manual inspection is susceptible to subjective factors, making it difficult to guarantee the accuracy of seal identification and thus failing to effectively protect the safety of containerized goods. Existing image recognition methods, relying on single detection networks or classification neural networks, struggle to balance recall and accuracy, similarly failing to meet the requirement of accurate seal identification. Summary of the Invention

[0005] In order to accurately identify lead seals on containers and improve operational efficiency, this application provides a method and system for identifying lead seals on containers.

[0006] Firstly, this application provides a method for identifying container lead seals, including: Acquire images of the container and perform preprocessing; The preprocessed image is subjected to lock and lead seal detection using a target detection neural network, and the image of the lock region detected by the target detection neural network is extracted; the target detection neural network is configured to output detection boxes and corresponding category probability values; the categories include: lead seal present and lead seal missing. The image of the latch area is classified and analyzed to determine whether a lead seal exists. The classification neural network is configured to receive the image of the latch area and output a binary classification result and confidence level indicating whether a lead seal exists. The state of the lead seal is determined and output based on the analysis results of the target detection neural network and the classification neural network. The comprehensive determination steps include: determining whether the output of the target detection neural network is a single detection box; if it is multiple detection boxes, comparing the probability values ​​of the corresponding categories of the detection boxes, and determining whether a lead seal exists based on the output corresponding to the larger value; if it is a single detection box, directly determining whether a lead seal exists based on the output of the category corresponding to the single detection box; comparing whether there is a conflict between the output of the classification neural network and the output of the target detection neural network determined by the single detection box; if there is a conflict, further determining whether the confidence level of the classification neural network is greater than a preset confidence threshold; if it is greater, directly outputting the analysis result of the classification neural network; if it is not greater, comparing the confidence level of the classification result output by the classification neural network with the probability value of the corresponding category of the detection box output by the target detection neural network determined by the single detection box, and determining whether a lead seal exists based on the output corresponding to the larger value.

[0007] By adopting the above scheme, the target detection neural network is used to detect the latch and lead seal and extract the image of the latch area to obtain the lead seal information in advance; the classification neural network is used to classify and analyze the image of the latch area to further confirm the state of the lead seal; the analysis results of the target detection neural network and the classification neural network are comprehensively judged to accurately determine the state of the lead seal and output the location of the lead seal under the condition of no missed detection, thus effectively determining the existence and location of the container lead seal.

[0008] Preferably, the comprehensive determination step further includes: After determining that the confidence level of the classification neural network is greater than the preset confidence threshold, the stability of the classification neural network output is judged. Based on the stability judgment result, it is decided whether to directly output the analysis result of the classification neural network to replace the original direct output of the analysis result of the classification neural network. The stability determination of the classification neural network output includes: judging whether the difference between the confidence level of the classification result and the preset confidence threshold is less than a preset difference; if it is not less than the preset difference, the classification neural network output is determined to be stable; otherwise, the classification neural network output is determined to be unstable. The step of determining whether to directly output the classification result of the classification neural network based on the stability determination result includes: if the classification neural network output is determined to be stable, the analysis result of the classification neural network is directly output; if the classification neural network output is determined to be unstable, the confidence level of the classification result output by the classification neural network is compared with the probability value of the category corresponding to the detection box output by the target detection neural network determined by a single detection box, and the output corresponding to the larger value is used to determine whether the seal exists.

[0009] By adopting the above scheme, when the classification neural network determines whether the lead seal is missing based on the confidence level, it is further judged whether the output of the classification neural network is stable. In the case of unstable output, the existence of the lead seal is determined by comprehensive comparative analysis, which can improve the accuracy of lead seal identification and solve the problem of inaccurate lead seal identification.

[0010] Preferred options also include: A container seal template library is constructed. The container seal template library stores template images of various container seals in different scenarios. For each template image, the HOG feature of the detection box is extracted, and the seal type and viewpoint are labeled and stored together in the template library. The system synchronously acquires environmental data of the current container image and determines the corresponding environment type. For different environment types, it sets multi-detection box conflict resolution strategies that match the environment type. These strategies include: a basic probability value comparison strategy, a morphological consistency check strategy, and a temporal filtering strategy. The basic probability value comparison strategy involves comparing the probability values ​​of each detection box and determining whether a lead seal exists based on the category with the highest probability value. The morphological consistency check strategy involves extracting the HOG features of each detection box and comparing them with the HOG features of lead seals in all template images in the container lead seal template library. It calculates the cosine similarity, selects the detection box with the highest similarity value that is greater than a preset similarity value, and determines whether a lead seal exists based on the category corresponding to the selected detection box. The temporal filtering strategy involves acquiring detection boxes corresponding to the same latch in adjacent container images, using Kalman filtering to smooth the detection, predicting the current frame's detection box state, and using the filtered detection output to determine whether a lead seal exists. The comprehensive determination step further includes: if the target detection neural network has multiple detection boxes, then a multi-detection box conflict resolution strategy is matched according to the current environment type to replace the original basic probability value comparison strategy to determine whether a lead seal exists.

[0011] By adopting the above scheme, a container lead seal template library is constructed and HOG features of the template image detection boxes are extracted, which can provide a foundation for subsequent lead seal feature comparison. Simultaneously, the current container image environment data is acquired and the environment type is determined, and a multi-detection box conflict resolution strategy matching it can be set. Appropriate strategies can be flexibly selected according to different environment types to improve the accuracy of determining the presence of lead seals in complex situations.

[0012] Preferred options also include: If the output of the classification neural network is unstable in consecutive frames, and the classification neural network is still deemed unstable when performing classification analysis on the locked region image of the current frame, then the region expansion coefficient is calculated based on the confidence level and scene type of the output after classification analysis, and the locked region is expanded to obtain the expanded locked region; the classification analysis is performed again for the new locked region; the expansion and classification analysis are repeated until the classification analysis result for the current frame is determined to be stable by the output of the classification neural network or the preset number of expansions is reached; The step of calculating the region expansion coefficient based on the confidence level and scene type output by the classification analysis includes: setting a basic expansion coefficient; matching a first adjustment factor based on the difference between the output confidence level and the preset confidence level threshold, and setting a matching first adjustment factor for different differences; matching a second adjustment factor based on the scene type, and setting a matching second adjustment factor for each scene type; and calculating the final region expansion coefficient based on the basic expansion coefficient, the first adjustment factor, and the second adjustment factor.

[0013] By adopting the above scheme, when the output of the classification neural network is unstable in consecutive frames, the expansion coefficient is calculated based on the confidence level and scene type to expand the locking area and reclassify and analyze it, which effectively solves the problem of unstable output of the classification neural network and improves the accuracy and stability of lead seal recognition.

[0014] Preferred options also include: Acquire multi-angle images of containers and perform preprocessing; When it is necessary to expand the locking region extracted from the current frame container image, multiple sets of container images from other angles corresponding to the current frame container image are obtained. Lock detection is performed on the preprocessed multiple sets of images through a target detection neural network. All currently detected locking region images are clustered and analyzed to determine the coordinates of the obtained locking positioning box and the locking opening direction. The opening direction is the main one and the other directions are secondary. The current frame locking detection box is non-uniformly expanded according to the expansion coefficient to obtain the expanded locking region.

[0015] By adopting the above scheme, multi-angle container images are acquired and preprocessed. When it is necessary to expand the locking area of ​​the previous frame, multiple sets of images from other angles are used for locking detection and cluster analysis to determine the coordinates of the locking positioning frame and the opening direction. The locking detection frame is then expanded according to the expansion coefficient, which can obtain locking area information more comprehensively and accurately, and improve the accuracy and stability of lead seal recognition.

[0016] Preferred options also include: In the process of detecting buckles and lead seals in the preprocessed image using a target detection neural network, an edge-guided interpolation algorithm is introduced to determine the edge of the buckle region in order to assist in extracting the buckle region image.

[0017] By adopting the above scheme, an edge-guided interpolation algorithm is introduced into the target detection neural network detection process to determine the edge of the locking region and assist in extracting the locking region image more accurately.

[0018] Preferred options also include: In the process of classifying and analyzing whether a lead seal exists in the image of the latch area using a classification neural network, a CBAM attention mechanism is embedded to automatically weight the lead seal feature map.

[0019] By adopting the above scheme, the CBAM attention mechanism is embedded in the classification analysis process of the classification neural network to automatically weight the seal feature map, increase the attention to the seal features, and improve the accuracy of seal recognition.

[0020] Secondly, this application provides a container seal identification system, comprising: The image acquisition module is used to acquire images of the container and perform preprocessing. The target detection module is used to detect buckles and lead seals in the preprocessed image using a target detection neural network, and to extract the buckle region image detected by the target detection neural network; the target detection neural network is configured to output detection boxes and corresponding category probability values; the categories include: lead seal present and lead seal missing. The classification analysis module is used to perform classification analysis on the image of the latch area to determine whether a lead seal exists, using a classification neural network; the classification neural network is configured to receive the image of the latch area and output a binary classification result and confidence level indicating whether a lead seal exists. The comprehensive judgment module is used to comprehensively judge and output the seal status based on the analysis results of the target detection neural network and the classification neural network. The comprehensive judgment steps include: determining whether the output of the target detection neural network is a single detection box; if it is multiple detection boxes, comparing the probability values ​​of the corresponding categories of the detection boxes, and determining whether a seal exists based on the output corresponding to the larger value; if it is a single detection box, directly determining whether a seal exists based on the output of the single detection box corresponding to the category; comparing whether there is a conflict between the output of the classification neural network and the output of the target detection neural network determined by the single detection box judgment; if there is a conflict, further judging whether the confidence level of the classification neural network is greater than a preset confidence threshold; if it is greater, directly outputting the analysis result of the classification neural network; if it is not greater, comparing the confidence level of the classification result output by the classification neural network and the probability value of the corresponding category of the detection box output by the target detection neural network determined by the single detection box judgment, and determining whether a seal exists based on the output corresponding to the larger value.

[0021] By adopting the above scheme and combining the analysis results of target detection neural networks and classification neural networks for comprehensive judgment, the accuracy and reliability of container seal identification can be improved.

[0022] Thirdly, this application provides a computer-readable storage medium including a stored computer program, wherein the computer program, when running, controls the device where the computer-readable storage medium is located to perform the method described above.

[0023] Fourthly, this application provides a computer device, the computer device including a memory, a processor and a program stored in the memory and executable thereon, the program being executed by the processor to implement the steps of the method described above.

[0024] In summary, this application has the following beneficial effects: 1. Use an object detection neural network to detect the latch and lead seal and extract the image of the latch area to obtain preliminary information about the lead seal; use a classification neural network to classify and analyze the image of the latch area to further confirm the state of the lead seal; combine the analysis results of the object detection neural network and the classification neural network to make a comprehensive judgment, thereby improving the accuracy and reliability of container lead seal identification. 2. When the output of the classification neural network is unstable, the presence of the lead seal is more accurately determined by combining the confidence level of the classification neural network and the probability value of the target detection neural network, thereby improving the accuracy and stability of the lead seal recognition results. When the output of the classification neural network is unstable in consecutive frames and the current frame is still unstable, the region expansion coefficient is calculated based on the confidence level of the classification analysis output and the scene type to expand the lock area and reclassify and analyze it, thereby solving the problem of unstable output of the classification neural network and improving the accuracy of lead seal recognition. 3. Construct a container lead seal template library and extract HOG features. Combine different environment types to set up multi-detection box conflict resolution strategies. When there are multiple detection boxes in the target detection neural network, the appropriate strategy can be matched according to the environment to more accurately and flexibly determine whether the container has a lead seal, thereby improving the accuracy and adaptability of lead seal recognition. Attached Figure Description

[0025] Figure 1 This is a flowchart of the container seal identification method described in a specific embodiment; Figure 2 The image shown is the brightened image in the container seal identification method described in the specific embodiment; Figure 3 The image shown is the result of the target detection neural network in the container seal identification method described in the specific embodiment. Figure 4 The image shows the classification analysis result of the classification neural network in the container seal identification method described in the specific embodiment; Figure 5 This is a flowchart illustrating the comprehensive analysis process in the container seal identification method described in a specific embodiment; Figure 6 This is a schematic diagram of the container seal identification system described in a specific embodiment. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0027] like Figure 1 As shown in the figure, this application discloses a method for identifying container seals, including steps such as image acquisition and preprocessing, target detection, classification analysis, and comprehensive judgment. Among them, the comprehensive judgment is achieved by using a combination of target detection neural network and classification neural network to improve the accuracy and efficiency of container seal identification. The following is a further detailed description of each step of this application.

[0028] S1. Acquire container images and perform preprocessing.

[0029] Specifically, in the image acquisition step, the acquisition equipment can employ high-definition cameras or industrial cameras to clearly capture images of the container and operate stably in complex lighting and harsh environments. Multiple camera angles can be set up to acquire multi-angle images of each target container in real time.

[0030] In the image preprocessing step, preprocessing is necessary because the acquired images may contain noise, uneven brightness, and other issues. Specifically, preprocessing methods include denoising and adjusting brightness and contrast. For denoising, Gaussian blur can be used first to eliminate image patch features. Gaussian blur, as a linear smoothing filter, can effectively suppress noise. Then, sharpening techniques are used to enhance the image's edge features, such as the Laplacian sharpening operator, which can enhance the image's edges and details. Considering that some images were taken at night, insufficient lighting may make the latching features less obvious, requiring increased brightness. Therefore, for images with insufficient lighting, histogram normalization can be used to increase brightness and adjust contrast to make the latching features more prominent. The brightened effect is as follows: Figure 2 As shown.

[0031] In addition, during the image acquisition process, environmental data, such as meteorological data like temperature and humidity, and camera location data, are collected simultaneously.

[0032] S2. Use a target detection neural network to detect locks and seals on the preprocessed image.

[0033] Specifically, in the target detection step, the target detection neural network can use the YOLO series detection algorithm; in this embodiment, YOLOv7 is selected. The target detection neural network is configured to take a container image as input and output detection boxes and corresponding category probability values. The output detection boxes include: latch boxes and detection boxes (there may be multiple or none); the corresponding categories for the lead seal detection boxes include: lead seal present and lead seal missing. This is generated through training on historical container images labeled with latch positions and the presence or absence of lead seals. In this embodiment, the target detection neural network analyzes the features of the container latches in the image and identifies them in the image, such as... Figure 3 As shown.

[0034] By inputting the preprocessed image into a trained object detection neural network, latch bounding boxes, detection boxes, and their class probability values ​​are obtained. Based on the latch bounding boxes, image segmentation techniques (a real-time segmentation module based on the U-Net architecture can be embedded in the object detection neural network) are used to extract the latch region image. To improve the extraction of the latch region image, an edge-guided interpolation algorithm is also introduced to determine the edges of the latch region to assist in image extraction. For the detection results, different colors can be used to distinguish latches containing lead seals from those without, and the target name and its probability are labeled above them, making it easier for users to distinguish whether the container is sealed.

[0035] S3. Perform classification analysis on the image of the latch area to determine whether a lead seal exists, using a classification neural network.

[0036] Specifically, in the classification analysis step, the classification neural network can employ the EfficientNet algorithm. This algorithm possesses the ability to quantize and adjust complex networks. By comprehensively adjusting the network depth, width, and input image resolution, it obtains the optimal network parameters for specific needs, enabling the network to simultaneously possess the advantages of both network size and recognition accuracy. EfficientNet receives the extracted lock region image and outputs a binary classification result and confidence score indicating the presence or absence of a lock, including: presence of a lock and its confidence score, or absence of a lock and its confidence score.

[0037] Considering that different environmental factors can affect the recognition results of the classification neural network, several EfficientNets (e.g., eight types from B0 to B7) can be trained and developed to adapt to different environment types. Using the current frame container image corresponding to the lock region image, the environment type corresponding to the synchronously collected environmental data is determined. The classification neural network under the corresponding environment type is then used to perform classification analysis on the lock region image to determine whether a lead seal exists, obtaining a more accurate classification result. In this embodiment, the classification effect of EfficientNetB7 on the lock region image is as follows: Figure 4 As shown.

[0038] In addition, to further improve the accuracy of the classification results output by the classification neural network, a CBAM attention mechanism is embedded in the process of classifying and analyzing whether a lead seal exists in the image of the latch area through the classification neural network, and the lead seal feature map is automatically weighted.

[0039] S4. Based on the analysis results of the target detection neural network and the classification neural network, a comprehensive judgment is made to determine the seal status, and the location where the seal exists is output accordingly.

[0040] Considering that relying solely on either the YOLOv7 target detection neural network or the EfficientNet classification neural network for lead seal detection has its limitations, it is necessary to effectively combine the two. Since all locks detected by YOLOv7 require classification analysis via EfficientNet, a comprehensive analysis method is designed, integrating the detection results from both methods. The workflow is as follows: Figure 5 As shown; the comprehensive determination steps include: Based on the analysis results of the above classification neural network, it is determined whether the lead seal is present or missing. Based on the output of the above target detection neural network, it is determined whether the output of the target detection neural network is a single detection box, to determine whether there is a multi-box detection output result. Combining the outputs of both, the following output determination is made: First, if the classification neural network determines that the seal is missing, and the target detection neural network has only one detection box (i.e., there is no situation where multiple detection boxes are caused by insufficient false detection due to feature ambiguity), then the seal is initially confirmed to be missing. Next, it is determined whether the probability value of the missing seal category is greater than a preset probability (45%). If it is determined to be greater than the preset probability, meaning the target detection neural network YOLOv7 believes that there may be no seal, then the result output by the classification neural network is verified, confirming that the seal is missing. If the probability value of the missing seal category is not greater than the preset probability (45%), it indicates that the target detection neural network YOLOv7 believes that the seal may exist, and there is a conflict between the two. Next, it is determined whether the confidence level of the classification neural network is greater than a preset confidence threshold (95%). If it is greater, it indicates that the accuracy of the classification neural network analysis result is high, and the determination of the missing seal is directly output. If it is not greater, it indicates that the accuracy of the classification neural network analysis result is average, and the comparison between the confidence level of the missing seal output by the classification neural network and the probability value of the corresponding category of the detection box output by the target detection neural network is chosen, with the output corresponding to the larger value determining whether a seal exists.

[0041] In addition to comprehensively analyzing and judging the output results by comparing confidence levels and probability values, a joint probability can also be constructed by weighting the confidence level of the classification results output by the classification neural network and the probability values ​​of the seal presence or absence categories output by the target detection neural network; the formula for constructing the weighted joint probability is as follows: In the formula, is the probability value of the seal presence or absence category output by the target detection neural network; is the confidence level of the classification result output by the classification neural network; in the formula, The probability value of the presence or absence of the lead seal as output by the target detection neural network; The confidence score of the classification result output by the classification neural network; where the weights of the confidence scores are... Weights of probability values The selection criteria can be determined based on the current environment. For example, in a sunny daytime environment at a port, where close-up photography is possible, YOLOv7 is used as the primary choice (with a larger weighting), supplemented by EfficientNet. Alternatively, in a low-light / nighttime environment, or in a rainy, reflective environment, EfficientNet is used as the primary choice (with a larger weighting), supplemented by YOLOv7. Finally, by comparing the joint probability with a preset joint probability threshold, the presence of the lead seal is determined if the joint probability is greater than the preset threshold; otherwise, the absence of the lead seal is considered.

[0042] Secondly, if the classification neural network determines that the seal is missing, and the target detection neural network has multiple detection boxes, then the probability values ​​of the corresponding categories of the detection boxes are compared. The output corresponding to the larger value determines whether a seal exists. If the output category with the larger probability value determines that the seal is missing, then the final determination is that the seal is missing. If the output category with the larger probability value determines that the seal exists, it indicates that the target detection neural network YOLOv7 believes that the seal may exist, and there is a conflict between the two. The system then checks whether the confidence level of the classification neural network is greater than a preset confidence threshold (95%). If it is greater, it indicates that the accuracy of the classification neural network analysis result is high, and the determination of a missing seal is directly output. If it is not greater, it indicates that the accuracy of the classification neural network analysis result is average, and the system compares the confidence level of the missing seal output by the classification neural network with the probability values ​​of the corresponding categories of the detection boxes output by the target detection neural network, using the output corresponding to the larger value to determine whether a seal exists. Similarly, in case of conflict, a weighted comparison method can be used to determine the final determination output.

[0043] Third, if the classification neural network determines that the lead seal exists, and the target detection neural network has only one detection box, then the existence of the lead seal is initially confirmed. The system then checks whether the probability value of the lead seal's existence category is greater than a preset probability (45%). If it is determined to be greater than the preset probability, then the lead seal is confirmed to exist; that is, if YOLOv7 has only one recognition result and YOLOv7 believes there may be a lead seal, then it is determined that a lead seal exists. If the probability value of the lead seal's existence category is not greater than the preset probability (45%), and YOLOv7 has only one recognition result and YOLOv7 believes there may be no lead seal, then there is a conflict. The system then checks whether the confidence level of the classification neural network is greater than a preset confidence threshold. If it is, the accuracy of the classification neural network's analysis result is high, and the system directly outputs that the lead seal exists; otherwise, it indicates that the accuracy of the classification neural network's analysis result is average. In this case, the system compares the confidence level of the lead seal's absence output by the classification neural network with the probability value of the corresponding category of the detection box output by the target detection neural network, using the output with the larger value to determine whether the lead seal exists. Similarly, in case of conflict, a weighted comparison method can be used to determine the final judgment output.

[0044] Fourth, if the classification neural network determines that the lead seal exists, and the target detection neural network has multiple detection boxes, then the probability values ​​of the corresponding categories of the detection boxes are compared, and the output corresponding to the larger value determines whether the lead seal exists. If the output category with the larger probability value determines that the lead seal exists, then the final determination is that the lead seal exists. If the output category with the larger probability value determines that the lead seal is missing, it indicates that the target detection neural network YOLOv7 believes that the lead seal may be missing, and there is a conflict between the two. The system then continues to determine whether the confidence level of the classification neural network is greater than the preset confidence threshold (95%). If it is greater, it indicates that the accuracy of the classification neural network analysis result is high, and the determination of the lead seal's existence is directly output. If it is not greater, it indicates that the accuracy of the classification neural network analysis result is average, and the system then compares the confidence level of the lead seal's absence output by the classification neural network with the probability values ​​of the corresponding categories of the detection boxes output by the target detection neural network, and the output corresponding to the larger value determines whether the lead seal exists. Similarly, in case of conflict, a weighted comparison method can be used to determine the final determination output.

[0045] Based on the above analysis results, the final seal identification result is output, and when the seal is found to be missing, an alarm signal is generated and sent to the user's visual interface. At the same time, the location of the lock corresponding to the missing seal is sent to the user's visual interface.

[0046] In a specific embodiment, considering that the confidence level of the classification neural network may be unstable when it is near the confidence threshold, in order to further improve the accuracy of lead seal status recognition, the method includes judging whether the output of the classification neural network is stable and then completing the final output judgment of whether there is a lead seal. After determining that the confidence level of the classification neural network is greater than the preset confidence threshold, the stability of the classification neural network output is determined. That is, it is determined whether the difference between the confidence level of the classification result and the preset confidence threshold is less than the preset difference. If it is not less than the preset difference, the output of the classification neural network is determined to be stable; otherwise, the output of the classification neural network is determined to be unstable.

[0047] The decision to directly output the analysis result of the classification neural network, replacing the original direct output of the classification neural network, is based on the stability determination result of the classification neural network output. That is, if the classification neural network output is determined to be stable, the analysis result of the classification neural network is directly output; if the classification neural network output is determined to be unstable, the confidence level of the classification result output by the classification neural network is compared with the probability value of the category corresponding to the detection box output by the target detection neural network determined by a single detection box. The output corresponding to the larger value is used to determine whether the seal exists.

[0048] Specifically, taking the classification neural network's determination of a missing seal as an example, and the target detection neural network having only one detection box as an example, the missing seal is initially confirmed. Further, it is determined whether the probability value of the missing seal category is greater than a preset probability (45%). If the probability value is not greater than the preset probability, there is a conflict. The classification neural network's confidence level is then determined whether it is greater than a preset confidence threshold (95%). If it is, but the difference between its confidence level (95.05%) and the preset confidence threshold (95%) (0.05%) is less than a preset difference (1%), the classification neural network's output is considered unstable. The confidence level of the missing seal output by the classification neural network is then compared with the probability value of the corresponding category of the detection box output by the target detection neural network. The output with the larger value is used to determine whether a seal exists, replacing the direct output of a missing seal determination.

[0049] In a specific embodiment, considering complex environments and to ensure the reliability of lead seal recognition results, this method compensates for the shortcomings of a single strategy in different environments by setting multi-detection box conflict resolution strategies for different environment types. The most suitable strategy is selected based on the specific environmental conditions to handle multi-detection box conflicts, thereby improving the accuracy and stability of lead seal recognition. The method further includes: A container seal template library is constructed. The container seal template library stores template images of various container seals under different scenarios (different lighting, different viewing angles, different container movement speeds, etc.). For each template image, the HOG feature of the detection box is extracted, and the seal type and viewing angle are labeled and stored together in the template library.

[0050] Simultaneously acquire environmental data of the current container image and determine the corresponding environment type of the current container image; for different environment types, set a multi-detection box conflict resolution strategy that matches the environment type; specifically, the multi-detection box conflict resolution strategy includes: basic probability value comparison strategy, morphological consistency check strategy, and temporal filtering strategy.

[0051] The basic probability value comparison strategy includes: comparing the probability values ​​of each detection box, and determining whether a lead seal exists based on the category with the highest probability value. For example, if the category with the missing lead seal has the highest probability value, it is determined that the lead seal is missing, i.e., the target detection neural network output is considered to be missing a lead seal. The morphological consistency check strategy includes: extracting the HOG features of each detection box, comparing it with the lead seal HOG features in all template images in the container lead seal template library, calculating the cosine similarity, selecting the detection box with the highest similarity and greater than a preset similarity value, and determining whether a lead seal exists based on the category corresponding to the selected detection box.

[0052] Considering the temporal continuity of the container seal status (present / absent)—in normal scenarios, the seal of the same latch will not change abruptly in adjacent frames (within 100ms). Therefore, based on the continuity of the seal status in adjacent frames, the presence status of the seal in the detection box of the current frame is predicted. Thus, the temporal filtering strategy includes: acquiring the detection boxes corresponding to the same latch in the container images of adjacent frames, smoothing the detection using Kalman filtering, predicting the detection box status of the current frame, and determining the presence of the seal using the filtered detection output. The specific prediction steps include: tracking the presence / absence of the seal and the corresponding latch center position as state variables; selecting three consecutive frames as initial frames; fusing them to obtain the initial seal status X0; initializing the state equation and observation equation of the Kalman filter; setting the process noise Q (representing the probability of abrupt changes in the seal status, with a default small value of Q=0.01) and the observation noise R (representing the detection noise, dynamically adjusted according to image quality); based on the filtered output status of the previous frame, predicting the seal status of the current frame using the state equation; predicting the seal status and position; and ensuring continuity with the previous frame.

[0053] The comprehensive judgment step further includes: if the target detection neural network has multiple detection boxes, then a multi-detection box conflict resolution strategy is matched according to the current environment type to replace the original basic probability value comparison strategy to determine whether a lead seal exists. For example, for stable lighting and a frontal viewing angle, more reliance can be placed on shape consistency checks, so a shape consistency check strategy is selected accordingly. For drastic lighting changes, since HOG features are sensitive to lighting, a basic probability value comparison strategy is used. For rapid container movement or camera shake, a temporal filtering strategy is selected accordingly.

[0054] In a specific embodiment, to effectively address the problem of unstable output from the classification neural network and improve the reliability of lead seal identification, when the classification neural network output is unstable, expanding the lock region and reclassifying and analyzing it can broaden the analysis range, obtain more feature information, and thus improve the classification accuracy. The method further includes: If the output of the classification neural network is unstable across consecutive frames, and the classification neural network is still deemed unstable when performing classification analysis on the current frame's latching region image, it indicates that the current classification neural network is unable to make a stable and accurate determination of the current frame's latching region. In this case, the classification neural network can be optimized, or the current frame's latching region image can be expanded. The expansion size is adjusted according to the region expansion coefficient. In this embodiment, the current frame's latching region image is expanded, and the region expansion coefficient is calculated based on the confidence level and scene type output after classification analysis. The latching region is then expanded according to the expansion coefficient to obtain the expanded latching region.

[0055] The step of calculating the region expansion coefficient based on the confidence level and scene type output by the classification analysis includes: setting the basic expansion coefficient. The first adjustment factor is matched based on the difference between the output confidence level and the preset confidence threshold. And different difference values ​​are set with a matching first adjustment factor. , obtain The adjustment factor decreases as the difference between the confidence level and the preset confidence threshold decreases (a smaller difference indicates lower confidence, more severe feature loss, and a greater need for expansion). Different adjustment factor values ​​can be preset based on expert experience or historical data to match different difference values. A second adjustment factor is matched based on the scene type. And a second adjustment factor is set for each scene type. , obtain For example, in a stable lighting environment, the corresponding scene adjustment factor is 0.8-0.9; in a complex environment with drastic lighting changes, rapid container movement, or camera shake, the corresponding scene adjustment factor is 1.1-1.2. The final region expansion factor is calculated based on the basic expansion factor, the first adjustment factor, and the second adjustment factor.

[0056] Finally, the new locking region is reclassified and analyzed; the expansion and classification analysis are repeated until the classification analysis result for the current frame is determined to be stable for the output of the classification neural network or the preset number of expansions is reached, such as 3 times.

[0057] In one specific embodiment, acquiring multi-angle container images provides more comprehensive information. When expanding the locking area, by performing locking detection and cluster analysis on multiple sets of images from different angles, the position and opening direction of the locking can be determined more accurately. Expanding the area primarily based on the opening direction allows for a more targeted expansion of the analysis scope, obtaining more effective feature information and improving the accuracy of seal identification. Therefore, the method also includes: real-time acquisition of multi-angle container images and preprocessing them for noise reduction.

[0058] When it is necessary to expand the locking region extracted from the current frame container image, multiple sets of other angle container images corresponding to the current frame container image are obtained. Lock detection is performed on the preprocessed multiple sets of images through a target detection neural network. By cluster analysis of all currently detected locking region images, the coordinates of the obtained locking positioning box and the locking opening direction are determined.

[0059] The current frame's latch detection box is non-uniformly expanded according to the expansion coefficient, with the opening direction as the primary factor and other directions as secondary factors. The expanded latch area is then obtained. That is, the expansion coefficient in the opening direction = the basic expansion coefficient * the expansion coefficient, and the expansion coefficient in other directions = the basic expansion coefficient.

[0060] like Figure 6 As shown in the figure, this application discloses a container seal identification system, including: Image acquisition module 100 is used to acquire container images and perform preprocessing; The target detection module 200 is used to detect buckles and lead seals on the preprocessed image through a target detection neural network, and extract the buckle region image detected by the target detection neural network; the target detection neural network is configured to output detection boxes and corresponding category probability values; the categories include: lead seal present and lead seal missing. The classification analysis module 300 is used to perform classification analysis on the image of the latch area to determine whether a lead seal exists, using a classification neural network; the classification neural network is configured to receive the image of the latch area and output a binary classification result and confidence level indicating whether a lead seal exists. The comprehensive judgment module 400 is used to perform a first comprehensive judgment based on the analysis results of the target detection neural network and the classification neural network to determine the state of the lead seal. The first comprehensive judgment step includes: if the classification neural network determines that the lead seal is missing, then the lead seal is initially confirmed to be missing, and the probability value of the missing lead seal category is further judged to be greater than a preset probability. If it is determined to be greater than the preset probability, the lead seal is determined to be missing; otherwise, the lead seal is determined to exist. If the classification neural network determines that the lead seal exists and the target detection neural network has only one detection box, then the lead seal is initially confirmed to exist, and the probability value of the present lead seal category is further judged to be greater than a preset probability. If it is determined to be greater than the preset probability, the lead seal is determined to exist; otherwise, the lead seal is determined to be missing. If the classification neural network determines that the lead seal exists and the target detection neural network has multiple detection boxes, then the probability values ​​of each detection box are compared, and the category with the larger probability value is output to determine whether the lead seal exists.

[0061] In one specific embodiment, the comprehensive judgment module 400 in the system is further configured to, after determining that the confidence level of the classification neural network is greater than a preset confidence level threshold, select to perform a stability determination of the output of the classification neural network, and decide whether to directly output the analysis result of the classification neural network to replace the original direct output of the analysis result of the classification neural network based on the stability determination result of the output of the classification neural network.

[0062] In one specific embodiment, the system further includes: The template library construction module 500 is used to construct a container seal template library. The container seal template library stores template images of various container seals in different scenarios. For each template image, the HOG feature of the detection box is extracted, and the seal type and viewpoint are labeled and stored together in the template library. The environmental data acquisition module 600 is used to synchronously acquire environmental data of the current container image and determine the environmental type corresponding to the current container image; The comprehensive judgment module 400 is also used to set a multi-detection box conflict resolution strategy that matches the environment type for different environment types; if there are multiple detection boxes in the target detection neural network, the multi-detection box conflict resolution strategy is matched according to the current environment type to replace the original basic probability value comparison strategy to determine whether there is a lead seal.

[0063] In a specific embodiment, the comprehensive judgment module 400 is further configured to: if there are consecutive frames where the output of the classification neural network is unstable, and when the classification neural network is used to classify and analyze the locked region image of the current frame, it is still determined that the output of the classification neural network is unstable, then calculate the region expansion coefficient based on the confidence level and scene type of the output after classification analysis, expand the locked region, and obtain the expanded locked region; re-classify and analyze the new locked region; repeat the expansion and classification analysis until the classification analysis result for the current frame is determined to be stable or the preset number of expansions is reached.

[0064] In one specific embodiment, the system further includes: The image acquisition module 100 is also used to acquire multi-angle container images and perform preprocessing; The comprehensive judgment module 400 is also used to acquire multiple sets of other angle container images corresponding to the current frame container image when it is necessary to expand the locking region extracted from the current frame container image. The module then performs locking detection on the preprocessed multiple sets of images through a target detection neural network, performs cluster analysis on all currently detected locking region images, determines the coordinates of the acquired locking positioning box and the locking opening direction, and performs non-uniform expansion of the current frame locking detection box according to the expansion coefficient, with the opening direction as the main direction and other directions as secondary directions, to obtain the expanded locking region.

[0065] This application also discloses a computer-readable storage medium.

[0066] Specifically, the computer-readable storage medium stores a computer program that can be loaded by a processor and executed, such as the container seal identification method described above. The computer-readable storage medium includes, for example, various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0067] This application also discloses a computer device.

[0068] Specifically, the computer device includes a memory and a processor, with the memory storing a computer program that can be loaded by the processor and executed using the aforementioned container seal identification method.

[0069] The above are all preferred embodiments of this application and are not intended to limit the scope of protection of this application. Any feature disclosed in this specification (including the abstract and drawings) may be replaced by other equivalent or similar features unless specifically stated otherwise. That is, unless specifically stated otherwise, each feature is only one example of a series of equivalent or similar features.

Claims

1. A method for identifying lead seals on containers, characterized in that, include: Acquire images of the container and perform preprocessing; The preprocessed image is subjected to buckle and lead seal detection by a target detection neural network, and the buckle area image detected by the target detection neural network is extracted. The target detection neural network is configured to output detection boxes and corresponding category probability values; the categories include: seal present and seal missing. The image of the latch area is classified and analyzed to determine whether a lead seal exists. The classification neural network is configured to receive the image of the latch area and output a binary classification result and confidence level indicating whether a lead seal exists. The state of the lead seal is determined and output based on the analysis results of the target detection neural network and the classification neural network. The comprehensive determination steps include: determining whether the output of the target detection neural network is a single detection box; if it is multiple detection boxes, comparing the probability values ​​of the corresponding categories of the detection boxes, and determining whether a lead seal exists based on the output corresponding to the larger value; if it is a single detection box, directly determining whether a lead seal exists based on the output of the category corresponding to the single detection box; comparing whether there is a conflict between the output of the classification neural network and the output of the target detection neural network determined by the single detection box; if there is a conflict, further determining whether the confidence level of the classification neural network is greater than a preset confidence threshold; if it is greater, directly outputting the analysis result of the classification neural network; if it is not greater, comparing the confidence level of the classification result output by the classification neural network with the probability value of the corresponding category of the detection box output by the target detection neural network determined by the single detection box, and determining whether a lead seal exists based on the output corresponding to the larger value.

2. The container seal identification method according to claim 1, characterized in that, The comprehensive determination step also includes: After determining that the confidence level of the classification neural network is greater than the preset confidence threshold, the stability of the classification neural network output is judged. Based on the stability judgment result, it is decided whether to directly output the analysis result of the classification neural network to replace the original direct output of the analysis result of the classification neural network. The stability determination of the classification neural network output includes: judging whether the difference between the confidence level of the classification result and the preset confidence threshold is less than a preset difference; if it is not less than the preset difference, the classification neural network output is determined to be stable; otherwise, the classification neural network output is determined to be unstable. The step of determining whether to directly output the classification result of the classification neural network based on the stability determination result includes: if the classification neural network output is determined to be stable, the analysis result of the classification neural network is directly output; if the classification neural network output is determined to be unstable, the confidence level of the classification result output by the classification neural network is compared with the probability value of the category corresponding to the detection box output by the target detection neural network determined by a single detection box, and the output corresponding to the larger value is used to determine whether the seal exists.

3. The container seal identification method according to claim 2, characterized in that, Also includes: Build a container seal template library; The container seal template library stores template images of various container seals in different scenarios. For each template image, HOG feature extraction of the detection box is performed, and the seal type and viewpoint are labeled and stored together in the template library. Simultaneously acquire environmental data of the current container image and determine the corresponding environment type of the current container image; For different environment types, set up multi-detection frame conflict resolution strategies that match the environment type; The multi-detection box conflict resolution strategy includes: a basic probability value comparison strategy, a morphological consistency check strategy, and a temporal filtering strategy. The basic probability value comparison strategy involves comparing the probability values ​​of each detection box and determining the presence of a lead seal based on the category with the highest probability value. The morphological consistency check strategy involves extracting the HOG features of each detection box and comparing them with the HOG features of lead seals in all template images in the container lead seal template library. Cosine similarity is calculated, and the detection box with the highest similarity value (greater than a preset similarity value) is selected. The presence of a lead seal is determined based on the category corresponding to the selected detection box. The temporal filtering strategy involves obtaining detection boxes corresponding to the same latch in adjacent frame container images, using Kalman filtering to smooth the detection, predicting the current frame detection box state, and using the filtered detection output to determine the presence of a lead seal. The comprehensive determination step further includes: if the target detection neural network has multiple detection boxes, then a multi-detection box conflict resolution strategy is matched according to the current environment type to replace the original basic probability value comparison strategy to determine whether a lead seal exists.

4. The container seal identification method according to claim 3, characterized in that, Also includes: If the output of the classification neural network is unstable in consecutive frames, and the classification neural network is still deemed unstable when performing classification analysis on the locked region image of the current frame, then the region expansion coefficient is calculated based on the confidence level and scene type of the output after classification analysis, and the locked region is expanded to obtain the expanded locked region; the classification analysis is performed again for the new locked region; the expansion and classification analysis are repeated until the classification analysis result for the current frame is determined to be stable by the output of the classification neural network or the preset number of expansions is reached; The step of calculating the region expansion coefficient based on the confidence level and scene type output by the classification analysis includes: setting a basic expansion coefficient; matching a first adjustment factor based on the difference between the output confidence level and the preset confidence level threshold, and setting a matching first adjustment factor for different differences; matching a second adjustment factor based on the scene type, and setting a matching second adjustment factor for each scene type; and calculating the final region expansion coefficient based on the basic expansion coefficient, the first adjustment factor, and the second adjustment factor.

5. The container seal identification method according to claim 4, characterized in that, Also includes: Acquire multi-angle images of containers and perform preprocessing; When it is necessary to expand the locking region extracted from the current frame container image, multiple sets of container images from other angles corresponding to the current frame container image are obtained. Lock detection is performed on the preprocessed multiple sets of images through a target detection neural network. All currently detected locking region images are clustered and analyzed to determine the coordinates of the obtained locking positioning box and the locking opening direction. The opening direction is the main one and the other directions are secondary. The current frame locking detection box is non-uniformly expanded according to the expansion coefficient to obtain the expanded locking region.

6. The container seal identification method according to claim 2, characterized in that, Also includes: In the process of detecting buckles and lead seals in the preprocessed image using a target detection neural network, an edge-guided interpolation algorithm is introduced to determine the edge of the buckle region in order to assist in extracting the buckle region image.

7. The container seal identification method according to claim 2, characterized in that, Also includes: In the process of classifying and analyzing whether a lead seal exists in the image of the latch area using a classification neural network, a CBAM attention mechanism is embedded to automatically weight the lead seal feature map.

8. A container seal identification system, characterized in that, include: The image acquisition module is used to acquire images of the container and perform preprocessing. The target detection module is used to detect buckles and lead seals in the preprocessed image through a target detection neural network, and to extract the buckle area image detected by the target detection neural network. The target detection neural network is configured to output detection boxes and corresponding category probability values; the categories include: seal present and seal missing. The classification analysis module is used to perform classification analysis on the image of the latch area to determine whether a lead seal exists, using a classification neural network; the classification neural network is configured to receive the image of the latch area and output a binary classification result and confidence level indicating whether a lead seal exists. The comprehensive judgment module is used to perform a first comprehensive judgment based on the analysis results of the target detection neural network and the classification neural network to determine the state of the lead seal. The first comprehensive judgment step includes: if the classification neural network determines that the lead seal is missing, then the lead seal is initially confirmed to be missing, and the probability value of the missing lead seal category is further judged to be greater than a preset probability. If it is determined to be greater than the preset probability, the lead seal is determined to be missing; otherwise, the lead seal is determined to exist. If the classification neural network determines that the lead seal exists and the target detection neural network has only one detection box, then the lead seal is initially confirmed to exist. The probability value of the present lead seal category is further judged to be greater than a preset probability. If it is determined to be greater than the preset probability, the lead seal is determined to exist; otherwise, the lead seal is determined to be missing. If the classification neural network determines that the lead seal exists and the target detection neural network has multiple detection boxes, then the probability values ​​of each detection box are compared, and the category with the larger probability value is output to determine whether the lead seal exists.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein, when the computer program is executed, it controls the device on which the computer-readable storage medium is located to perform the method as described in any one of claims 1 to 7.

10. A computer device, characterized in that, The computer device includes a memory, a processor, and a program stored in and executable on the memory, the program being executed by the processor to implement the steps of the method as described in any one of claims 1 to 7.

Images