A container seal detection and recording method and system

Abstract

The present application provides a container seal detection and recording method and system, which is applied to the field of container seal technology, wherein the container seal detection method includes: marking a number of marked areas from the container image according to the characteristics of the door handle and door lock of the container; wherein the number of marked areas at least include the following areas where the seal may be: the door handle area and the door lock area in the container door area; classifying and identifying the door handle area and the door lock area according to the seal characteristics; wherein, when the door lock area has a seal feature, the door lock area is classified as a seal area, otherwise the door lock area is classified as a non-seal area; when the door handle area has a seal feature, the door handle area is classified as a seal area, otherwise the door handle area is classified as a non-seal area. Therefore, based on the two-stage method, the small seal targets in the container can be detected efficiently and accurately, and the detection scheme has high flexibility and is convenient for deployment and application.

Description

Technical Field

[0001] The present application relates to the technical field of container seals, and in particular to a method and system for detecting and recording container seals. Background Art

[0002] Container seals are an important security measure to ensure that containers are not opened without authorization during transportation. They are designed to prevent goods from being tampered with, stolen or lost, and play a key role in modern logistics and supply chain management. In port operations, it is necessary to inspect the seals of containers and enter the seal information. At present, the statistics of container seal information are mainly relied on by port staff, which leads to a large demand for personnel, low efficiency and poor accuracy of manual statistics, and many problems for subsequent container management. Even if visual algorithms are introduced in the existing solutions, the visual algorithms have weak anti-interference capabilities and are difficult to adapt to the complex environment of the port.

[0003] Based on this, a new technical solution that can identify container seals is needed. Summary of the invention

[0004] In view of this, the embodiments of this specification provide a container seal detection and recording method and system, which detects the seal by a two-stage recognition method, which can significantly reduce the image area that needs to be processed and ensure the efficiency and accuracy of container seal detection.

[0005] The embodiments of this specification provide the following technical solutions:

[0006] The embodiment of this specification provides a container seal detection method, including:

[0007] According to the characteristics of the door handle and door lock of the container, a number of marked areas are marked from the container image; wherein the number of marked areas at least include the following areas where the lead seal may be located: the door handle area and the door lock area in the container door area;

[0008] The door lock area and the door handle area are classified and identified respectively according to the lead seal characteristics; when the door lock area has the lead seal characteristics, the door lock area is classified as a lead-sealed area, otherwise the door lock area is classified as a non-lead-sealed area; when the door handle area has the lead seal characteristics, the door handle area is classified as a lead-sealed area, otherwise the door handle area is classified as a non-lead-sealed area.

[0009] Compared with the prior art, the at least one technical solution adopted in the embodiments of this specification can achieve the following beneficial effects:

[0010] Since the seal is usually small in the image, it is difficult to detect the whole image directly. Therefore, the present invention adopts a two-stage method to detect the seal. First, the system locates all the door handles and door lock areas on the box door, because the seal can only appear in these specific areas, and there is at most one seal in each area. In this way, the system can significantly reduce the image area that needs to be processed, thereby improving the efficiency and accuracy of detection.

[0011] Specifically, in the first stage, advanced object detection algorithms are used to accurately identify and locate the door handle and door lock areas on the box door. Then, in the second stage, these located areas are further classified to determine whether they contain lead seals.

[0012] Therefore, this two-stage detection method can not only significantly improve the accuracy and speed of seal detection, but also accurately obtain the number of seals and the specific area where they are located, thereby fully realizing the goal of seal detection. It can more effectively adapt to various complex scenarios and conditions, ensure that seals can be reliably detected in different environments, and improve the safety of logistics and goods. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0014] Figure 1 It is a schematic diagram of a two-stage architecture for detecting container seals using computer vision in this application;

[0015] Figure 2a It is a flow chart of a container seal detection method in this application;

[0016] Figure 2b It is a schematic diagram of the seal detection result of collecting the container image from the upper right angle in this application;

[0017] Figure 2c It is a schematic diagram of the seal detection result of collecting the container image from the upper left angle in this application;

[0018] Figure 2d It is a flowchart of the training phase and the reasoning phase in this application;

[0019] Figure 3a This is a schematic diagram of the container image collected in the night scene environment in this application;

[0020] Figure 3bThis is a schematic diagram of a container door area image extracted from a container image in a night scene environment in this application;

[0021] Figure 3c This is a schematic diagram of marking the door handle and door lock areas in the container image under the night scene environment in this application;

[0022] Figure 3d This is a schematic diagram of the overall result after the lead seal detection of the container image is completed in the night scene environment in this application;

[0023] Figure 4a This is a schematic diagram of a container image collected in a daytime environment in this application;

[0024] Figure 4b This is a schematic diagram of a door area image extracted from a container image in a daytime environment in this application;

[0025] Figure 4c This is a schematic diagram of marking the door handle and door lock areas in the container image in the daytime environment in this application;

[0026] Figure 4d This is a schematic diagram of the overall result after the container image is completed with the lead seal detection in the daytime environment in this application;

[0027] Figure 5 This is a schematic diagram of the data set preparation, model training and application for container seal detection in the first phase of this application;

[0028] Figure 6 It is a schematic diagram of the data set preparation, model training and application for container seal classification in the second phase of this application;

[0029] Figure 7 It is a structural schematic diagram of a container seal detection system in this application;

[0030] Figure 8 It is a structural schematic diagram of a container seal recording system in this application;

[0031] Fig. 9 It is a flow chart of a container sealing recording method in this application. DETAILED DESCRIPTION

[0032] The embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0033] The following describes the implementation methods of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The present application can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that, in the absence of conflict, the following embodiments and the features in the embodiments can be combined with each other. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work belong to the scope of protection of the present application.

[0034] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein can be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present application, it should be understood by those skilled in the art that an aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number and aspect described herein can be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein can be used to implement this device and / or practice this method.

[0035] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present application. The drawings only show components related to the present application rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0036] Additionally, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, it will be understood by those skilled in the art that the examples can be practiced without these specific details.

[0037] At present, seal detection mainly relies on manual inspection and traditional image recognition. However, manual inspection is time-consuming and laborious, and traditional image recognition requires manual design of features, poor robustness, and slow speed. For example, existing document 1 (CN110503046A, a seal identification method based on image recognition technology) requires manual positioning of the seal position, and existing document 2 (CN116524018A, a method for automatic identification of seals based on computer vision) is difficult to process different types of seal results due to insufficient consideration of complex scenes.

[0038] Therefore, manual or traditional image processing solutions can no longer meet the needs of container seal inspection.

[0039] In view of this, through the detection of container seals, and in-depth research and improvement exploration of related image processing of seals, it is found that:

[0040] On the one hand, since the pixel area occupied by the lead seal in the entire image is usually very small, if we continue to use the traditional image processing method, that is, directly detecting the lead seal based on the full image of the container, we will inevitably encounter many difficulties, such as it is difficult to determine whether there is a lead seal in the image, or even identifying the lead seal as other objects.

[0041] Therefore, the technical idea of ​​segmented processing can be adopted: in the container field, lead seals may only appear in some specific areas, such as door handles, door locks and other local areas that may be opened, and each door lock area has at most one lead seal, and the position of door handles, door locks, etc. in the image, the size of the area occupied, etc. are generally very obvious, so door handles, door locks and other areas can be very conveniently located and identified by traditional image processing methods. Therefore, in the first stage of processing, all door handles, door locks and other areas on the container door can be quickly and accurately located from the captured container image, and then in the second stage of processing, only these specific areas that have been located are classified and identified for the presence or absence of lead seals.

[0042] In other words, because the positioning work in the first stage is based on relatively significant features, the positioning process is relatively simple, and does not require a complex positioning algorithm model. It is possible to crop out (or mark out) a small number of areas where lead seals may exist from the entire image (such as a very limited area such as a door lock and the door handle where the door lock is located), which significantly reduces the difficulty, complexity and data processing volume of image processing. In the second stage of classification processing, the classification processing only processes these reduced small image areas, and can rely on the significant features unique to lead seals (such as corner features) to determine whether there are lead seals in these areas. That is, the classification processing process in the second stage is also relatively simple, and similarly does not require a complex algorithm model to complete lead seal classification and identification, so it can also significantly reduce the difficulty, complexity and data processing volume of image processing.

[0043] Through two-stage processing, the difficulty and complexity of detection can be significantly reduced, and the image content that needs to be processed can be significantly reduced, ensuring the efficiency and accuracy of detection, and being able to flexibly meet the deployment and application of port container scenarios.

[0044] Secondly, although computer vision-based target detection and recognition and classification schemes have been widely used in the field of image target detection, these recognition and classification models are usually large-scale models and require training based on a huge amount of data to obtain a usable model. That is, the algorithm model and model parameters are very large, which makes the model design and training difficult, and the model is more difficult to deploy in actual application scenarios (for example, too many computing resources are required, resulting in many occasions where it cannot be deployed, etc.). In particular, when these intelligent recognition and classification models are transferred to the field of container technology, the overall performance of the model is poor and it can basically not be used for container seal detection applications.

[0045] Therefore, based on the above segmentation technology ideas, the model is also split into two simple small models: positioning model and classification model. Among them, the positioning model is used to locate and identify the designated areas where lead seals may exist from the container image, such as the door handle area, door lock area and other specific areas. Since these areas have very obvious identification features in the container, they can be quickly and accurately located and identified from the captured container image. In addition, due to the significant features of these areas, the training of the positioning model is very simple, and the efficiency and accuracy of positioning and identification can be guaranteed; the classification model is used to identify the presence or absence of lead seals from the designated areas provided by the positioning model.

[0046] Based on this, the embodiment of this specification proposes a container seal detection solution: Figure 1 As shown, for the collected data (i.e., container images), in the first stage of processing, the door handles, door locks and other significant features are first used to quickly and accurately identify the areas where the lead seals are likely to be from the entire container image, that is, a mature positioning model is used to extract (such as by cropping or marking) the image blocks of the container door and the door handles, door locks and other designated areas on the container door; then in the second stage of processing, the lead seal features are used to quickly classify and identify these designated areas, that is, since the lead seals have specific features, the classification model can be used to quickly complete the classification based on the lead seal features, and the classification only needs to determine the presence or absence of the lead seal, so that the statistical port can complete the container lead seal statistics.

[0047] In addition, after classifying whether there are lead seals in these designated areas, these areas can be further added as labeled data to the training data set of the positioning model and / or classification model, thereby further improving the model performance by using labeled data.

[0048] The technical solutions provided by various embodiments of the present application are described below in conjunction with the accompanying drawings.

[0049] like Figure 2a As shown, the embodiment of this specification provides a container seal detection method, comprising the following steps:

[0050] Step S202, the first stage of positioning and identification processing: according to the features of the door handle and door lock of the container, a number of marked areas are marked from the container image.

[0051] In implementation, when positioning and marking the designated area, the positioned and marked areas include at least the following areas where the seal may be: the door handle area and the door lock area in the box door area.

[0052] refer to Figure 2b and Figure 2c In the first stage of positioning and recognition, the door area is quickly determined from the entire image collected, as shown in the following figure: Figure 2b , 2c The local image indicated by the large marked box on the left side of the figure, and then in the local image of the door area, the areas corresponding to each door handle and door lock are quickly marked, as shown in the figure below. Figure 2b , 2c The local image indicated by the small annotation box on the left side of the figure is as follows: Figure 2b , 2c The middle is a partial enlarged schematic diagram of the image in the dotted box on the left.

[0053] It should be noted that when displaying the annotated data using a schematic frame (such as a rectangular frame), only the door handle area can be annotated, only the door lock area can be annotated, or both can be annotated with a schematic frame to facilitate manual observation of the annotated frames. Moreover, in terms of data processing logic, both will be annotated with corresponding schematic frames, so that these annotated areas are convenient for schematic display and will not affect the classification processing of the annotated areas in the second stage.

[0054] Step S204, second stage classification and identification processing: classify and identify the door lock area and the door handle area respectively according to the lead seal features.

[0055] Since the lead seal recognition process is performed on the local area (such as the door handle area and the door lock area), and the lead seal is usually in the door handle area and the door lock area, when the door handle area and / or the door lock area are detected to have a lead seal, the corresponding area with the lead seal can be classified as a lead-sealed area, otherwise it can be classified as a non-lead-sealed area. For example, when the door lock area has a lead seal feature, the door lock area can be classified as a lead-sealed area, otherwise it can be classified as a non-lead-sealed area; for example, when the door handle area has a lead seal feature, the door handle area can be classified as a lead-sealed area, otherwise it can be classified as a non-lead-sealed area.

[0056] During implementation, considering that traditional containers are generally only sealed on the right door handle and right door lock, and containers generally have two right door handles, two right door locks, etc., if the type of container being detected is clearly sealed only on the right door handle and / or right door lock, the door handle area and door lock area in this application can be directly used to carry out the first and second stage work on the right door handle area and right door lock area. At this time, the features of the door handle, door lock, etc. are more significant, so the trained first stage positioning model, second stage classification model, etc. are completed using these significant features, which has better performance and is more targeted, and can significantly improve the positioning performance of the first stage and the classification and recognition performance of the second stage, thereby ensuring the efficiency and effect of container seal detection.

[0057] refer to Figure 2b and Figure 2c Illustration: Based on the marked area located in the first stage, the classification and recognition model in the second stage is used to quickly determine whether the marked area has a seal (a seal is marked as haveseal, and a seal is marked as noseal).

[0058] Therefore, the container seal detection solution of the present application can be divided into two stages in terms of processing logic, and both stages can use prior information about containers and seals to quickly complete the work, avoiding the defect of traditional solutions that directly process the entire captured image, and providing a simple and effective detection solution for traditional containers when conducting seal detection.

[0059] refer to Figure 2d The illustrated training stage is as follows: first, the labeled data of door handles, door locks, etc. are used as the training set, and then the detection model of door handles and door locks used in the first stage is trained. The training is completed after the corresponding loss of the classification model is obtained. In addition, the detection model can extract the door handle and door lock areas and label them (in the specific implementation, the detection area can be expanded to obtain the expanded detection frame). The data labeled with the labeled frame is used to form the training set required for the classification model for classification and recognition in the second stage. The classification and recognition model of the second stage is trained using the training set, and the training is completed after the corresponding loss of the classification model is obtained.

[0060] refer to Figure 2d Schematic reasoning stage: the collected image is input into the trained detection model, and the door handle and door lock detection model performs the first stage of positioning and recognition, and then extracts the areas corresponding to the door handle (such as the right door handle) and the door lock (such as the right door lock), and marks the corresponding detection frames (the expanded detection frames can also be expanded). The data related to the labeled detection frames obtained in the first stage enters the classification model in the second stage, and the classification model classifies and recognizes the door handle area (such as the area where the right door handle is located), the door lock area (such as the area where the right door lock is located), etc., and marks the classification results of whether there is a lead seal or not.

[0061] In addition, in the first stage of detection result processing, if no abnormal conditions such as right door handle or right door lock are found in the detection results, abnormal result processing will be performed. For example, if there is no right door handle, it will be prompted that the right door handle is not detected; for example, if there is no door lock, it will be prompted that there is no door lock.

[0062] Figure 3a-3d This is an example of conducting seal inspection on containers at night. Figures 4a-4d This is an example of conducting seal inspection on a container during daytime.

[0063] refer to Figure 3a-3d and Figures 4a-4d In the first stage of positioning and recognition, the whole image (such as Figure 3a , Figure 4a The door area is located in the captured image (such as Figure 3b , Figure 4b Then, based on the door handles, door locks and other prominent features on the door, the door handle area and door lock area are located from the door area, such as Figure 3c , 4c Schematic marked area.

[0064] refer to Figure 3a-3d and Figures 4a-4d In the second stage of classification and recognition, it is determined whether there is a lead seal feature in the specified area, and then the area with the lead seal feature is classified as a lead seal area (such as Figure 3c , Figure 4c The light-colored marked box part indicated in the figure), otherwise it is classified as the lead-free seal area (such as Figure 3c , Figure 4c (as indicated by the light-colored box in the figure). Figure 3c , 3d and Figure 4c , 4d In the diagrams, the lead-sealed area is schematically marked with a light-colored frame, and the unsealed area is schematically marked with a black frame.

[0065] It should be noted that different colors can be used to display lead-sealed areas and non-lead-sealed areas during implementation, so as to facilitate quick and accurate manual confirmation of the marked areas.

[0066] In some embodiments, the feature data such as door handle features and door lock features in the present application can be deep-level feature data extracted based on mature deep learning algorithms, such as deep-level image features automatically extracted through deep learning models such as convolutional neural networks (CNN). Moreover, deep learning models such as convolutional neural networks (CNN) used for feature extraction can refer to mature existing solutions, simplifying the design of feature extraction models, and feature extraction based on these models can ensure that the extracted features have significant characteristics, which is conducive to ensuring the efficiency and accuracy of lead seal recognition.

[0067] Through the two-stage processing of positioning and classification, the defects of traditional image detection in directly identifying lead seals from the entire picture are avoided. In addition, in both stages, the lead seal recognition is closely combined with the container's own characteristics to be tailored, making the container lead seal detection solution based on computer vision algorithm more in line with the lead seal recognition process.

[0068] Therefore, this application can solve several major pain points and difficulties in container seal identification in ports:

[0069] 1. Detection performance

[0070] High accuracy: Deep learning models such as convolutional neural networks (CNNs) automatically extract deep features of container seal images and use them to locate and classify the area where the seal is located, which can ensure the efficiency and accuracy of recognition. Moreover, compared with traditional image processing methods, feature extraction based on deep learning models performs better in dealing with complex backgrounds, lighting changes, occlusions, etc., and the recognition results are more reliable;

[0071] Real-time processing: The present invention implements real-time reasoning of deep learning models on high-performance computing hardware, which can quickly process large-scale container seal images and achieve real-time recognition to meet practical application needs;

[0072] Strong robustness: The deep learning model of the present invention has strong adaptability to various environmental changes, such as lighting, angle, occlusion, etc., ensuring the stability and consistency of the recognition results;

[0073] 2. Testing costs

[0074] Reduce labor costs: Traditional manual inspection methods require a lot of manpower and are prone to missed detection and false detection. This invention significantly reduces labor costs and improves overall detection efficiency through automated identification;

[0075] Reduced equipment costs: Compared with solutions that require dedicated hardware devices, the present invention is based on standard computer vision hardware, such as cameras and GPU servers, which reduces equipment costs and maintenance expenses;

[0076] 3. Optimization of testing process

[0077] Automated process: The present invention simplifies the multi-step operation in the traditional method through a fully automated lead seal identification process, reduces manual intervention and operational complexity, and improves detection efficiency and consistency;

[0078] Real-time monitoring and early warning: The detection results can be used to implement real-time monitoring and early warning functions in the system. For example, when an abnormal seal is identified, early warning information can be issued in time, allowing relevant personnel to take quick measures and enhance the security and reliability of the system.

[0079] In some implementations, in the process of using the seal feature for classification, it may be based on the corner feature of the seal, so as to quickly and accurately determine whether there is a seal in the marked area (det_area).

[0080] In implementation, the corner feature of the lead seal is used as the lead seal feature for classification. Therefore, in the classification and identification of each door handle area, the door handle area is first expanded to the left, right and bottom according to the preset first ratio as the first detection area to determine whether there is a lead seal in the door handle area, and then it is determined whether there is a corner feature of the lead seal in the first detection area. If so, the door handle area is determined as a lead seal area (haveseal), otherwise it is determined as a lead-free area (noseal); and in the classification and identification of each door lock area, the door lock area is first expanded downward according to the preset second ratio as the second detection area to determine whether there is a lead seal in the door lock area, and then it is determined whether there is a corner feature of the lead seal in the second detection area. If so, the door lock area is determined as a lead seal area, otherwise it is determined as a lead-free area.

[0081] It should be noted that when using corner point features for identification and classification, a local window can be selected and moved in various directions to calculate the difference between the pixels in the window before and after the movement to determine whether the area corresponding to the window is a corner point.

[0082] By performing corner point judgment in the above-mentioned multiple overlapping expanded areas, that is, judging whether there are lead-sealed corner points inside the expanded door handle area, door lock area and other areas, the accuracy of the results can be better guaranteed than traditional single area judgment.

[0083] In some embodiments, the expansion ratio can be preset and adjusted according to the container detection application, for example, the preset first ratio includes 10%-40%, and the preset second ratio includes 10%-30%, so as to cover the possible location area of ​​the container seal, reduce the amount of image data, and take into account both detection efficiency and accuracy.

[0084] In some embodiments, in both the positioning and classification stages, images of areas where seals may be located may be cropped from captured images to form samples in a data set, thereby ensuring that positioning and classification processes have sufficient and reliable data samples for training and testing.

[0085] The first detection area and the second detection area may be cropped out from the container image, and a label data set may be formed after the cropped image is labeled according to whether it belongs to a lead-sealed area or a lead-free area.

[0086] It should be noted that, in the present application, the cropped designated area may be a marked area that has not been expanded, or may be an area that has been expanded according to the above ratio.

[0087] In some embodiments, in both stages of processing, relevant algorithm models can be used to complete image positioning and classification, that is, the positioning model can be used to quickly identify relevant areas that need to detect lead seals from the entire image, such as specific areas such as box doors, door handles, and door locks.

[0088] For example, a positioning model is constructed using a deep learning algorithm, and then the positioning model is used to quickly mark several areas where the seal may be located (such as the aforementioned door handle area, door lock area, etc.) from the entire container image collected based on the door handle features of the container. The positioning model is a deep learning model trained based on the data set samples of the door handle features and door lock features.

[0089] For example, a classification model is constructed using a deep learning algorithm, and then the classification model is used to classify and identify the door lock area according to the lead seal feature. The classification model is a deep learning model obtained by sample training based on a data set of lead seal features.

[0090] It should be noted that since the door handle and door lock are a type of prominent feature on the door, the existing technology can be referred to to use these feature data to train a positioning model, without any specific limitation; similarly, the lead seal feature is also a type of prominent feature in the door lock area, and the classification model can be trained using these prominent features, which can also refer to the existing technology, without any specific limitation here.

[0091] In some embodiments, the algorithm model used in the above two stages can preferably be the single-stage target detection algorithm model of yolov5, wherein the positioning model is preferably the detection model of yolov5, and the classification model is preferably the classification model of yolov5. The excellent performance of yolov5 can be used to better balance the efficiency and accuracy of lead seal detection, and the difficulty of model design and deployment is simplified.

[0092] In some implementations, the image data in each embodiment may be enhanced data obtained after data enhancement, so as to further improve the adaptability of the solution to different environments and severe weather conditions and enhance robustness.

[0093] Among them, at least one of the following data enhancements is performed on the data set: simulated rainy day noise, foggy day noise. Therefore, even if there are many unstable factors in the outdoor environment, after data enhancement, such as simulated rainy day noise and foggy day noise, the efficiency and accuracy of lead seal detection can be guaranteed.

[0094] In practice, the image enhancement of the present application refers to adding some simulated noise of the environment in which the container is located to the original image, so as to simulate various container environments and then expand the useful data set. In addition, the image enhancement means may also include flipping, translation, rotation, scaling, etc., which are not limited here.

[0095] By adopting data enhancement, we can not only improve the impact of different environments on container images, but also provide container data set samples in various environments for deep learning, and solve the problems of traditional image methods in identifying container seals incorrectly in complex environments, poor environmental adaptability, and low accuracy in small target detection.

[0096] In some embodiments, the preparation of data sets and the use of data sets to carry out model training and application processes can be specifically referred to Figure 5 and Figure 6 Indicates the content.

[0097] refer to Figure 5 As shown in the figure, in the first stage of processing, the images are first annotated to form a data set. The annotated objects in the first stage are the door, door handle and door lock. Finding the location of the door handle and door lock through the door will also help the task in the second stage. Then the detection model is trained based on the data set and verified using the corresponding evaluation indicators to obtain a usable positioning model. Finally, in the application stage, the image to be detected can be input into the positioning model to obtain the specified area where the lead seal may exist, such as the door handle, door lock and other areas. In addition, the positioning model can be the detection model of yolov5 or other algorithm models, which are not limited here.

[0098] refer to Figure 6In the second stage of processing, the image is first labeled. The labeling object of the second stage is the presence or absence of lead seals. The regional image located in the first stage is used as the labeling object of the second stage. Then the classification model is used to train the data set and verified using the corresponding evaluation indicators. Finally, the image of the area where the lead seal may exist is input into the model to obtain the classification result of whether there is a lead seal. In addition, when establishing a data set, a variety of data enhancement methods can be used to expand the data set samples to cope with the influence of different environments and bad weather; the classification model can be the classification model of yolov5 or other algorithm models, which are not limited here.

[0099] In some implementations, the feature data in each implementation example, namely, the door handle feature, the door lock feature and / or the lead seal feature, is subjected to multi-scale feature extraction processing to introduce different scale perception mechanisms in the feature extraction process.

[0100] During implementation, a multi-scale feature extraction module can be added to the deep learning model, such as the positioning model for locating the area and the classification model for judging the presence or absence of lead seals, so as to introduce a scale perception mechanism in the feature extraction process to enhance the detection capability of targets of different sizes.

[0101] By introducing multi-scale feature extraction into the model, the model can dynamically adjust the receptive field to adapt to targets of different scales, effectively capture the features of small and large targets, fuse feature maps of different scales, and improve the detection capability of small targets.

[0102] In some embodiments, for port container deployment applications, container images can be collected by shooting the container door from above according to the special use requirements of the port, wherein the camera acquisition unit can be fixedly installed at a specific location such as a gate or under a bridge crane, which will not affect normal container operations and can obtain useful container images. The image direction obtained by shooting from above can refer to the above Figure 3a , Figure 4a The actual collected container image.

[0103] Therefore, the container image in each example can be an image of the container door in an actual scene captured at a designated camera position (such as a gate, bridge crane or other specific location), where the camera takes a bird's-eye view, and the angle between the bird's-eye view and the horizontal direction is in the range of 10-20 degrees, preferably a bird's-eye view angle of 15 degrees.

[0104] Based on the same inventive concept, the present application also provides a container seal detection system corresponding to the aforementioned method example, so as to complete the container seal detection by using the system.

[0105] refer to Figure 7 Schematically, a container seal detection system includes:

[0106] The detection unit 101 of the first stage: marks a number of marked areas from the container image according to the features of the door handle and door lock of the container; wherein the number of marked areas at least include the following areas where the lead seal may be located: the door handle area and the door lock area in the container door area;

[0107] The classification unit 103 of the second stage: classifies and identifies the door lock area and the door handle area respectively according to the lead seal feature; wherein, when the lead seal feature exists in the door lock area, the door lock area is classified as a lead-sealed area, otherwise the door lock area is classified as a non-lead-sealed area; and when the lead seal feature exists in the door handle area, the door handle area is classified as a lead-sealed area, otherwise the door handle area is classified as a non-lead-sealed area.

[0108] The two-stage processing unit avoids the defect of the traditional image processing system that it cannot detect small lead-sealed targets based on the entire image, and can also ensure detection efficiency and accuracy.

[0109] Based on the same inventive concept, the present application also provides a container seal recording method and system, so as to automatically count and record the number and position of seals that should exist on the container door according to the number and position of seals obtained by seal detection, so as to facilitate the automated processing of container seals at the port.

[0110] Based on the same inventive concept, the present application also provides a container seal recording system.

[0111] refer to Figure 8 Schematic diagram of a container seal recording system, comprising:

[0112] Camera unit: used to collect container images. In practice, the camera can be set up at the gate or under the bridge crane, and take a bird's-eye view of the container door side in the actual scene at an angle of 10-20 degrees to the horizontal direction to obtain the container image. The collected results can refer to the above Figure 2b Figure 2c , Figure 3a , Figure 4a Wait for the result to indicate;

[0113] A data processing unit: performs seal detection on the container image, and calculates the number and position of the seals in the container image; wherein the seal detection is performed based on the container seal detection method described in any one of the embodiments of the present application;

[0114] Data recording unit: records the number and position of the seals in association with the box door.

[0115] During the implementation, a high-resolution camera is first used to capture the container door handle and door lock area. The image size can be 1920x1080 pixels. The door, door handle and door lock areas are located based on a pre-trained target detection model (such as the YOLOv5 detection model). Then the images of these areas are input into a classification model (such as the YOLOv5 classification model) for seal detection. Since a large number of annotated field data sets are used in the training process, the detection system can achieve a high accuracy rate under different lighting and angles.

[0116] In some embodiments, the camera unit can be installed at a gate or under a bridge crane, wherein the camera unit forms an angle of 10-20 degrees with the horizontal direction, and then obtains the container image by capturing one side of the container door in the actual scene through a bird's-eye view.

[0117] In some implementations, corresponding warnings are issued after real-time monitoring for the recorded results of the lead seal detection. For example, if monitoring finds that there is a problem with the lead seal, the on-site personnel are promptly informed through a warning.

[0118] In some embodiments, since seal detection does not require a complex algorithm model, the data processing unit can be set at the yard side, such as integrated with a camera, or at the back end, such as in a processing terminal. Also, the data recording unit can be executed by the processing terminal. Alternatively, the data processing unit and / or the data recording unit is a GPU-based server unit or a general computer system.

[0119] In some embodiments, the container seal recording system further includes: a monitoring and early warning unit, which is used to monitor the number and position of seals on the container in real time, and issue an early warning message when an abnormal seal is identified.

[0120] In some implementations, the image data collected by the camera unit may be sent to the processing terminal via a network, and the processing terminal may also send warning information to the storage yard side via the network.

[0121] The network transmission mode may be HTTP mode or other modes. For example, after the camera unit acquires the container image, it sends the image to the data recording unit via HTTP.

[0122] Based on the same inventive concept, the present application provides a container seal recording method, comprising:

[0123] Step S402: collecting at least one image of a container including a container door by a camera unit;

[0124] Step S404: perform seal detection on the container image, and calculate the number and positions of seals in the container image; wherein the seal detection is performed based on the container seal detection method described in any one of the embodiments of the present application;

[0125] Step S406: associate the seal quantity and seal position with the box door and record them.

[0126] The working principles of each component are as follows: the camera unit (such as a camera) acquires images, the positioning model locates the door handle and door lock area, and the classification model classifies the seals, thereby achieving accurate detection and positioning of the seals.

[0127] The present invention can significantly improve the efficiency and reliability of container seal detection in applications at shipping terminals and other container scenarios.

[0128] It should be noted that a container seal recording method can be Figure 8 The system is executed without limitation here.

[0129] refer to Fig. 9 As shown in the figure, the following is an example to illustrate the process of container lead seal detection and recording: first, an image is input through HTTP, then it enters the first stage detection model, locates the location area where the lead seal may appear, then crops the area and inputs it into the second stage classification model, which determines whether there is a lead seal, and outputs the lead seal recognition result when there is a lead seal. In this way, the number of lead seals on the door image and the area where they are located can be obtained, thereby achieving the purpose of detecting lead seals, which is better than directly detecting the entire image range. In addition, the results can also be output through HTTP, and the recognition results include result data and pictures of the recognition results.

[0130] In this specification, the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the embodiments described later, the description is relatively simple, and the relevant parts can be referred to the partial description of the previous embodiments.

[0131] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A container seal detection method, characterized in that: include: Using the positioning model, a plurality of marked areas are marked from the container image according to the features of the door handle and door lock of the container; wherein the plurality of marked areas at least include the following areas where the lead seal may exist: the door handle area and the door lock area in the container door area; marking the plurality of marked areas from the container image includes: first locating the container door area from the entire container image, and then locating the door handle area and the door lock area from the container door area according to the features of the door handle and door lock on the container door; The door lock region and the door handle region are classified and identified according to the lead seal feature by using a classification model; wherein, when the door lock region has the lead seal feature, the door lock region is classified as a lead-sealed region, otherwise, the door lock region is classified as a non-lead-sealed region; when the door handle region has the lead seal feature, the door handle region is classified as a lead-sealed region, otherwise, the door handle region is classified as a non-lead-sealed region; The positioning model is a deep learning model trained based on a training data set of door handle features and door lock features, and the classification model is a deep learning model trained based on a training data set of lead seal features; The lead seal feature includes a corner feature of the lead seal; in classification and identification, the door handle area is first expanded to the left, right and bottom according to a preset first ratio as a first detection area for whether a lead seal exists in the door handle area, and the door lock area is expanded downward according to a preset second ratio as a second detection area for whether a lead seal exists in the door lock area, and then it is judged whether the lead seal corner feature exists in the first detection area and the second detection area respectively. If the lead seal corner feature exists in the first detection area, the corresponding door handle area is determined as a lead seal area, otherwise it is determined as a lead-free area. If the lead seal corner feature exists in the second detection area, the corresponding door lock area is determined as a lead seal area, otherwise it is determined as a lead-free area. The first detection area and the second detection area are cropped out from the container image, and labels are marked according to whether the cropped images belong to the lead-sealed area or the lead-unsealed area to form a label data set, and the label data set is added to the training data set of the positioning model and / or the classification model.

2. The container seal detection method according to claim 1, characterized in that: The preset first ratio includes 10%-40%, and the preset second ratio includes 10%-30%.

3. The container seal detection method according to claim 1, characterized in that: The positioning model is the detection model of yolov5; the classification model is the classification model of yolov5; And / or, the container seal detection method further includes: performing at least one of the following data enhancements on the data set: simulating noise on rainy days or foggy days.

4. The container seal detection method according to any one of claims 1 to 3, characterized in that: The container image is an image of the container door in the actual scene captured at a specified camera position, where the camera takes a bird's-eye view with an angle between the bird's-eye view and the horizontal direction within a range of 10-20 degrees; And / or, the container seal detection method further includes: performing multi-scale feature extraction processing on the door handle features, and / or the door lock features, and / or the seal features, so as to introduce different scale perception mechanisms in the feature extraction process.

5. A container seal recording method, characterized in that: include: Capturing at least one image of a container including a container door by a camera unit; Performing seal detection on the container image, and calculating the number and positions of seals in the container image; wherein the seal detection is performed based on the container seal detection method according to any one of claims 1 to 4; The number of seals and the position of the seals are associated with the corresponding box doors and recorded.

6. A container seal detection system, characterized in that: include: The first stage detection unit: using the positioning model to mark a number of marked areas from the container image according to the features of the door handle and door lock of the container; wherein the number of marked areas at least include the following areas where the lead seal may exist: the door handle area and the door lock area in the container door area; marking the number of marked areas from the container image includes: first locating the container door area from the entire container image, and then locating the door handle area and the door lock area from the container door area according to the features of the door handle and door lock on the container door; The second stage classification unit: using the classification model to classify and identify the door lock area and the door handle area respectively according to the lead seal feature; wherein, when the door lock area has the lead seal feature, the door lock area is classified as a lead-sealed area, otherwise the door lock area is classified as a non-lead-sealed area, and when the door handle area has the lead seal feature, the door handle area is classified as a lead-sealed area, otherwise the door handle area is classified as a non-lead-sealed area, wherein the positioning model is a deep learning model trained based on the training data set of the door handle feature and the door lock feature, and the classification model is a deep learning model trained based on the training data set of the lead seal feature; The lead seal feature includes a corner feature of the lead seal; in classification and identification, the door handle area is first expanded to the left, right and bottom according to a preset first ratio as a first detection area for whether a lead seal exists in the door handle area, and the door lock area is expanded downward according to a preset second ratio as a second detection area for whether a lead seal exists in the door lock area, and then it is judged whether the lead seal corner feature exists in the first detection area and the second detection area respectively. If the lead seal corner feature exists in the first detection area, the corresponding door handle area is determined as a lead seal area, otherwise it is determined as a lead-free area. If the lead seal corner feature exists in the second detection area, the corresponding door lock area is determined as a lead seal area, otherwise it is determined as a lead-free area. The first detection area and the second detection area are cropped out from the container image, and labels are marked according to whether the cropped images belong to the lead-sealed area or the lead-free area to form a label data set, and the label data set is added to the training data set of the positioning model and / or the classification model.

7. A container seal recording system, characterized in that: include: Camera unit: used to collect container images. It is installed at the gate or under the bridge crane and takes a bird's-eye view of the container door side in the actual scene at an angle of 10-20 degrees to the horizontal direction to obtain the container image. A data processing unit: performs seal detection on the container image, and calculates the number and position of the seals in the container image; wherein the seal detection is performed based on the container seal detection method according to any one of claims 1 to 4; Data recording unit: records the number of seals and the position of the seals in association with the corresponding box doors.

8. The container seal recording system according to claim 7, characterized in that: After acquiring the container image, the camera unit sends the image to the data recording unit via HTTP; And / or, the data processing unit and / or the data recording unit are both GPU-based server units; And / or, the container seal recording system further includes: a monitoring and early warning unit, which is used to monitor the number and position of seals on the container in real time, and issue an early warning message when an abnormal seal is identified.

Images