Container packing photo detection method and system based on computer vision

Abstract

The invention relates to a container encasement photo detection method and system based on computer vision, based on a YOLO network and a CRNN network, a container opening and closing state, a container number position, cargo packaging and other target detection model and a container number identification model are respectively constructed, an interface receives encasement photo address, container number and other data uploaded by a business system, and the data are sent to a server; after the corresponding picture is successfully loaded, the specific content of the box type and the box number can be comprehensively and accurately obtained through each optimized target detection model and each optimized box number identification model, finally, the information identified by each model is sorted, and the information corresponding to the box number in the business system is found according to the identified box number for cross verification. According to the technical scheme, the compliance of the uploaded photos is judged, intelligent detection of the boxed photos is completed, the workload of customer service staff is greatly reduced, illegal or problematic photos are effectively recognized, the risk and cost caused by wrong detection or missing detection of the illegal photos are reduced, and the boxed photo detection efficiency and accuracy can be greatly improved.

Description

Technical Field

[0001] The present invention relates to the technical field of ship shipping container packing photo detection, in particular to a container packing photo detection method and system based on computer vision. Background Art

[0002] In the container shipping business, customers need to upload the packing photos of the goods to the shipping company in time after packing to prove that the goods are intact and identify the types of packed goods; prevent the concealment and false reporting of dangerous goods, ensure the transportation safety of the goods and ships on board; prevent customers from taking advantage of low-priced goods to guarantee the efficiency of the company's routes; the shipping company arranges the loading plan according to the goods situation to improve the transportation efficiency and safety; it is used as an important document archive in the transportation process, quality monitoring record, and future query and reference and other business scenarios. The photos must be able to reflect the types of container doors fully open, half open and fully closed. The image quality of the photos is particularly important in the business. The container number in the photo must be clearly visible, the container number in the photo must be consistent with the order container number, and the goods must be clearly seen in the half-open and full-open photos. However, in actual production, the quality of the packing photos uploaded by customers varies, and many do not meet the requirements. For example, uploaded photos that are completely unrelated to packing, photos with low clarity, photos with shooting angles that do not meet the requirements, photos with inconsistent container numbers, etc., which seriously affect the subsequent business processes and progress.

[0003] However, due to the huge number of photos, the traditional verification of packing photos can only rely on manual spot checks by customer service personnel. A lot of manpower is spent on the spot check work, the verification speed is slow, and the proportion of manual spot checks is not high, and full coverage cannot be achieved. There is a risk of misdetection and omission of illegal photos and subsequent business disputes, and the verification accuracy rate is low.

[0004] At present, some container number recognition methods have gradually emerged, but the existing solutions have a relatively single angle of detecting containers. They can only identify the container number, but do not pay more attention to the detection of the goods and packaging types in the container, and cannot achieve comprehensive detection.

[0005] Therefore, there is an urgent need for a more efficient, accurate and comprehensive method to optimize the traditional manual spot check of packing photos, realize the automatic identification and verification of packing photos, replace manual verification work to the greatest extent, greatly reduce the workload of customer service, and reduce the risks and costs caused by misdetection or missed detection of illegal photos, thereby improving the overall competitiveness and benefits of the enterprise. Summary of the invention

[0006] The present invention aims at the problems that the traditional manual spot check of container packing photos costs a lot of manpower and there are risks caused by wrong detection and missed detection of illegal photos, and the current container box number recognition detection angle is single and can only recognize the container box number without comprehensive detection. A container packing photo detection method based on computer vision is proposed. Based on the YOLO network and the CRNN network, a target detection model and a box number recognition model such as the container opening and closing state, the box number position and the cargo packaging are constructed. The interface receives the data such as the packing photo address and the box number transmitted from the business system. After the corresponding picture is successfully loaded, the specific content of the box type and box number will be comprehensively and accurately obtained through the optimized target detection model and the box number recognition model. Finally, the information recognized by each model is sorted out, and the information corresponding to the box number in the business system is found according to the recognized box number for cross-checking, and the compliance of the uploaded photos is judged, which greatly reduces the workload of customer service personnel, effectively identifies illegal or problematic photos, reduces the risks and costs caused by wrong detection or missed detection of illegal photos, and can greatly improve the efficiency and accuracy of packing photo detection. The present invention also relates to a container packing photo detection system based on computer vision.

[0007] The technical solution of the present invention is as follows:

[0008] A method for detecting container packing photos based on computer vision, characterized in that it comprises the following steps:

[0009] A data set acquisition and processing step is to acquire a large number of container packing photos from a business system server, wherein the container packing photos include fully open, half-open, and fully closed types of packing conditions, and filter the container packing photos according to a preset filtering condition to obtain a filtered data set;

[0010] The target detection model generation step generates several target detection models based on the YOLO network, wherein the target detection models include a container opening and closing state detection model, a cargo position detection model, and a cargo packaging type detection model, and respectively marks the container opening and closing state, the cargo position, and the cargo packaging type on the screened data set through a marking tool, and then divides the marked data set into a target detection training set, a target detection verification set, and a target detection test set according to a preset ratio, and inputs the divided target detection training set and the target detection verification set into each target detection model for training, and automatically verifies the corresponding target detection model currently obtained after a preset number of training rounds, evaluates the performance of each target detection model, and then inputs the divided target detection test set into each trained and verified target detection model for testing, so as to generate optimized target detection models;

[0011] The box number recognition model generation step is to generate a box number recognition model based on the CRNN network, and mark the box number of the filtered data set through a marking tool, and then divide the marked data set into a box number recognition training set, a box number recognition verification set and a box number recognition test set according to a preset ratio, input the divided box number recognition training set and the box number recognition verification set into the box number recognition model for training, and automatically verify the current box number recognition model after a preset number of training rounds to evaluate the performance of the box number recognition model, and then input the divided box number recognition test set into the trained and verified box number recognition model for testing to generate an optimized box number recognition model;

[0012] The packing photo detection processing step is to receive several container packing photos uploaded by the user through the API interface, detect the opening and closing status of the container through the optimized container opening and closing status detection model, and detect the cargo position through the optimized cargo position detection model when it is detected that the container has an open or closed state and is in a fully open or half-open type, and detect the cargo packaging type through the optimized cargo packaging type detection model, and output the cargo position and cargo packaging type detection results; when it is detected that the container has an open or closed state and is in a fully closed or half-open type, identify the container number through the optimized container number recognition model and output the container number recognition result; then cross-validate the cargo position and cargo packaging type detection results with the container number recognition results to complete the intelligent detection of packing photos.

[0013] Preferably, in the target detection model generation step, the target detection model generated based on the YOLO network also includes a seal number position detection model and a box number position detection model, and the filtered data set is marked with the seal number position and the box number position by a marking tool, and the marked data set is divided into a target detection training set, a target detection verification set and a target detection test set according to a preset ratio, and then each target detection model is trained, verified and tested respectively to generate each optimized target detection model;

[0014] In the packing photo detection processing step, when it is detected that the container is not in an open or closed state, the seal number position is detected by the optimized seal number position detection model, and the seal number position detection result is output; when it is detected that the container is in an open or closed state and is in a fully closed or half-open type, the container number position is first detected by the optimized box number position detection model, and then the box number is identified by the optimized box number recognition model, and the box number position detection result and the box number recognition result are output; then the cargo position, cargo packaging type, seal number position and box number position detection results are cross-validated with the box number recognition results to complete the intelligent detection of the packing photos.

[0015] Preferably, in the target detection model generation step, the target detection model generated based on the YOLO network also includes a recognition model for each component of the box number, and the filtered data set is marked with each component of the box number by a marking tool, and the marked data set is divided into a target detection training set, a target detection verification set and a target detection test set according to a preset ratio, and then the recognition model for each component of the box number is trained, verified and tested to generate an optimized recognition model for each component of the box number;

[0016] In the packing photo detection processing step, when it is detected that the container is in an open or closed state and is in a fully closed or half-open type, the container number position is first detected by the optimized container number position detection model, and then the container number components are identified by the optimized container number component recognition model, and then the container number is identified by the optimized container number recognition model, and then the container number is identified by the optimized container number recognition model, and the container number position detection result, the container number component recognition result and the container number recognition result are output; then the cargo position, cargo packaging type, lead seal number position and container number position detection results, the container number component recognition results and the container number recognition result are cross-validated to complete the intelligent detection of the packing photo.

[0017] Preferably, in the packing photo detection processing step, after receiving a number of container packing photos uploaded by the user through the API interface, the container position detection is first performed on the photos, and when the container is detected, the container opening and closing status is detected by using an optimized container opening and closing status detection model.

[0018] Preferably, in the packing photo detection processing step, when a container is detected, the packing photo direction detection is automatically performed, and the packing photo is automatically rotated by 0 degrees, 90 degrees, 180 degrees or 270 degrees according to the detected packing photo direction using a picture rotation tool, and after the rotation, the container opening and closing state is detected using an optimized container opening and closing state detection model.

[0019] Preferably, in the data set acquisition processing step, the screened data set is also subjected to image graying, binarization, balancing of the number of samples in each category, image size adjustment, normalization and tensorization processing to obtain a screened and preprocessed data set; the subsequent target detection model generation step and box number recognition model generation step are to use a marking tool to mark the screened and preprocessed data set accordingly, and the corresponding training set, validation set and test set are divided in a ratio of 3:1:1.

[0020] A container packing photo detection system based on computer vision, characterized in that it comprises a data set acquisition processing module, a target detection model generation module, a box number recognition model generation module and a packing photo detection processing module, wherein the data set acquisition processing module is respectively connected to the target detection model generation module and the box number recognition model generation module, and the target detection model generation module and the box number recognition model generation module are both connected to the packing photo detection processing module.

[0021] The data set acquisition processing module acquires a large number of container packing photos from the business system server, wherein the container packing photos include fully open, half open and fully closed types of packing conditions, and screens the container packing photos according to preset screening conditions to obtain a screened data set;

[0022] The target detection model generation module generates several target detection models based on the YOLO network, and the target detection models include a container opening and closing state detection model, a cargo position detection model, and a cargo packaging type detection model, and respectively marks the container opening and closing state, cargo position, and cargo packaging type on the screened data set through a marking tool, and then divides the marked data set into a target detection training set, a target detection verification set, and a target detection test set according to a preset ratio, and inputs the divided target detection training set and target detection verification set into each target detection model for training, and automatically verifies the corresponding target detection model currently obtained after a preset number of training rounds, evaluates the performance of each target detection model, and then inputs the divided target detection test set into each trained and verified target detection model for testing, so as to generate optimized target detection models;

[0023] The box number recognition model generation module generates a box number recognition model based on the CRNN network, and marks the box number of the filtered data set through a marking tool, and then divides the marked data set into a box number recognition training set, a box number recognition verification set and a box number recognition test set according to a preset ratio, inputs the divided box number recognition training set and the box number recognition verification set into the box number recognition model for training, and automatically verifies the currently obtained box number recognition model after a preset number of training rounds, evaluates the performance of the box number recognition model, and then inputs the divided box number recognition test set into the trained and verified box number recognition model for testing, so as to generate an optimized box number recognition model;

[0024] The packing photo detection processing module receives a number of container packing photos uploaded by the user through the API interface, detects the opening and closing state of the container through the optimized container opening and closing state detection model, detects the cargo position through the optimized cargo position detection model when it is detected that the container has an open or closed state and is in a fully open or half-open type, and detects the cargo packaging type through the optimized cargo packaging type detection model, and outputs the cargo position and cargo packaging type detection results; when it is detected that the container has an open or closed state and is in a fully closed or half-open type, identifies the container number through the optimized container number recognition model and outputs the container number recognition result; then cross-validates the cargo position and cargo packaging type detection results with the container number recognition result to complete the intelligent detection of the packing photos.

[0025] Preferably, in the target detection model generation module, the target detection model generated based on the YOLO network also includes a seal number position detection model and a box number position detection model, and the filtered data set is marked with the seal number position and the box number position by a marking tool, and the marked data set is divided into a target detection training set, a target detection verification set and a target detection test set according to a preset ratio, and then each target detection model is trained, verified and tested respectively to generate each optimized target detection model;

[0026] In the packing photo detection processing module, when it is detected that the container is not in an open or closed state, the seal number position is detected by the optimized seal number position detection model, and the seal number position detection result is output; when it is detected that the container is in an open or closed state and is in a fully closed or half-open type, the container number position is first detected by the optimized box number position detection model, and then the box number is identified by the optimized box number recognition model, and the box number position detection result and the box number recognition result are output; then the cargo position, cargo packaging type, seal number position and box number position detection results are cross-validated with the box number recognition results to complete the intelligent detection of the packing photos.

[0027] Preferably, in the target detection model generation module, the target detection model generated based on the YOLO network also includes recognition models for each component of the box number, and the filtered data set is marked with each component of the box number by a marking tool, and the marked data set is divided into a target detection training set, a target detection verification set and a target detection test set according to a preset ratio, and then the recognition models for each component of the box number are trained, verified and tested to generate an optimized recognition model for each component of the box number;

[0028] In the packing photo detection and processing module, when it is detected that the container is in an open or closed state and is in a fully closed or half-open type, the box number position is first detected by the optimized box number position detection model, and then the box number components are identified by the optimized box number component identification model, and then the box number is identified by the optimized box number identification model, and the box number is outputted. The box number position detection result, the box number component identification result and the box number identification result are outputted; then the cargo position, cargo packaging type, lead seal number position and box number position detection results, the box number component identification results and the box number identification result are cross-validated to complete the intelligent detection of the packing photo.

[0029] Preferably, in the packing photo detection processing module, after receiving several container packing photos uploaded by the user through the API interface, the container position detection is first performed on the photos, and when the container is detected, the packing photo direction detection is automatically performed, and the packing photo is automatically rotated by 0 degrees, 90 degrees, 180 degrees or 270 degrees according to the detected packing photo direction using a rotation picture direction tool, and after the rotation, the container opening and closing state is detected by the optimized container opening and closing state detection model.

[0030] The technical effects of the present invention are as follows:

[0031] The present invention relates to a method for detecting container packing photos based on computer vision. After acquiring a large number of container packing photos, a plurality of target detection models including a container opening and closing state detection model, a cargo position detection model and a cargo packaging type detection model are generated based on a YOLO network, and a container number recognition model is generated based on a CRNN network. The data sets are respectively marked, proportionally segmented, model trained, verified and tested, and optimized target detection models and container number recognition models are respectively generated. Then, an API interface receives a plurality of container packing photos uploaded by a user, and the container opening and closing state is detected by the optimized container opening and closing state detection model. When it is detected that the container has an opening and closing state and is in a fully open or half-open type, the cargo position is detected by the optimized cargo position detection model, and the cargo packaging type is detected by the optimized cargo packaging type detection model, and the cargo position and cargo packaging type detection results are output; when it is detected that the container has an opening and closing state and is in a fully closed or half-open type, the optimized cargo position detection model is used to detect the cargo position, and the cargo packaging type is detected by the optimized cargo packaging type detection model, and the cargo position and cargo packaging type detection results are output; when it is detected that the container has an opening and closing state and is in a fully closed or half-open type, the optimized cargo position detection model is used to detect the cargo position, and the cargo packaging type is detected by the optimized cargo packaging type detection model, and the cargo position and cargo packaging type detection results are output. When the container number is identified, the optimized container number recognition model is used to identify the container number and output the container number recognition result; the present invention can accurately identify the container opening and closing state, cargo position, cargo packaging type and container number in the container packing photo based on the YOLO network and the CRNN network through the deep learning model. That is to say, on the basis of accurately identifying the container number, the container opening and closing state, cargo position, cargo packaging type, container orientation, lead seal number position, and box type code in the container packing photo can be further identified. It combines computer vision technologies such as image processing, machine learning, feature extraction, target detection and recognition, and cross-validates the information corresponding to the identified container number and the detected container number (cargo position and cargo packaging type detection results) to complete the intelligent detection of the packing photo, which can greatly reduce the workload of the shipping company's customer service, realize full coverage verification of the packing photos, reduce the risks and costs caused by misdetection or missed detection of illegal photos, and greatly improve the efficiency and accuracy of packing photo detection.

[0032] The present invention is based on the computer vision container packing photo detection method, through the use of deep learning model to quickly, accurately, full coverage detection of packing photos, replacing the traditional manual cumbersome and inefficient spot check; the existing scheme to detect the orientation angle of the container is relatively single, and can only identify the container number, while the present invention can further identify whether the goods and packaging types match the information stored under the same container number in the business system on the basis of accurate identification of the container number; through the API service interface, the reuse requirements between the shipowner's systems can be realized, and it has good robustness and scalability. The present invention optimizes the traditional manual spot check of packing photos through a more efficient, accurate, and full coverage method, realizes the automatic identification and verification of packing photos, and replaces the manual verification work to the greatest extent. It only needs to screen out the suspected problem photos after cross-validation and hand them over to the customer service staff for further manual review, while the normal qualified photos that are consistent with the cross-validation do not require manual intervention, greatly reducing the workload of customer service, and reducing the risks and costs caused by misdetection or missed detection of illegal photos, and improving the overall competitiveness and benefits of the enterprise.

[0033] The present invention also relates to a computer vision-based container packing photo detection system, which corresponds to the above-mentioned computer vision-based container packing photo detection method, and can be understood as a system that implements the above-mentioned computer vision-based container packing photo detection method, including a data set acquisition processing module, a target detection model generation module, a box number recognition model generation module and a packing photo detection processing module. Each module works together to build a target detection model and a box number recognition model for the container opening and closing state, box number location and cargo packaging based on a YOLO network and a CRNN network through a deep learning model. The interface receives the packing photo address and box number data transmitted from the business system, and after successfully loading the corresponding picture , will respectively use the target detection models such as the opening and closing status of the container, the location of the container number and the packaging of the goods. At the same time, the pictures with the container number will be passed through the text recognition algorithm of the container number recognition model, so as to comprehensively and accurately obtain the specific content of the container type and number. Finally, the information recognized by each model will be sorted out, and the information corresponding to the container number in the business system will be found according to the recognized container number for cross-checking, and the compliance of the uploaded photos will be judged to select the problem data. The problem data can be handed over to the customer service staff for further verification, which greatly reduces the workload of the customer service staff, effectively identifies illegal or problematic photos, reduces the risks and costs caused by misdetection or missed detection of illegal photos, can greatly improve the efficiency and accuracy of packaging photo detection, and has certain engineering application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 The present invention is a flowchart of a method for detecting container packing photos based on computer vision.

[0035] Figure 2Schematic of the steps to generate an object detection model.

[0036] Figure 3 Schematic diagram of the steps for generating a bin number recognition model.

[0037] Figure 4 The figure is a preferred flow chart of the packing photo detection processing steps.

[0038] Figure 5 The structure diagram of the container packing photo detection system based on computer vision of the present invention is shown in FIG. DETAILED DESCRIPTION

[0039] The present invention will be described below in conjunction with the accompanying drawings.

[0040] The present invention relates to a method for detecting container packing photos based on computer vision, the process of which is as follows: Figure 1 As shown, the following steps are included:

[0041] S1: a data set acquisition processing step, obtaining a large number of container packing photos from the business system server, wherein the container packing photos include fully open, half open (i.e., half closed) and fully closed types of packing conditions, and filtering the container packing photos according to preset filtering conditions to obtain a filtered data set.

[0042] Preferably, this step may include boxing photo collection, boxing photo screening and image preprocessing.

[0043] S11: Packing photo collection: Obtain a large amount of container packing photos from the business system server (i.e., obtain historical data stored in the business system server), including fully open, half-open, and fully closed types of packing conditions;

[0044] S12: Filtering of container loading photos: Filtering container loading photos according to preset filtering conditions to remove invalid photos such as photos without containers or duplicate photos. The filtered photos are used as training, verification and test data sets;

[0045] S13: Image preprocessing: Preferably, the filtered data set is subjected to image grayscale, binarization, balancing of the number of samples in each category, image resizing, normalization and tensorization processing to reduce interference from factors such as lighting and blur, improve recognition effect after training, and obtain a filtered and preprocessed data set; through a series of processing steps, the original data is converted into a form suitable for model training.

[0046] S2: Target detection model generation step, based on the YOLO network, several target detection models are generated, the target detection models include a container opening and closing state detection model, a cargo location detection model and a cargo packaging type detection model, and the filtered data set is marked with a marking tool for the container opening and closing state, cargo location and cargo packaging type, and then the marked data set is divided into a target detection training set, a target detection verification set and a target detection test set according to a preset ratio, and the divided target detection training set and target detection verification set are input into each target detection model for training, and the corresponding target detection model currently obtained is automatically verified after a preset number of training rounds to evaluate the performance of each target detection model, and then the divided target detection test set is input into each trained and verified target detection model for testing, so as to generate optimized target detection models. This step is based on marking the filtered and preprocessed data set, and inputting it into each target detection network model for training and testing. Specifically, Figure 2 Schematic diagram shown.

[0047] S21: generating an initial target detection model: generating several target detection models based on the YOLO network, wherein the target detection models include a container opening and closing state detection model, a seal number position detection model, a box number position detection model, a box number component recognition model, a cargo position detection model, and a cargo packaging type detection model;

[0048] S22: Target detection data set labeling: Label the data set filtered and preprocessed in S13 by a labeling tool (such as IabelImg labeling tool), preferably, label the container opening and closing status (box door opening and closing status), seal number position, box number position, each component of the box number, cargo position, and cargo packaging type (or special packaging type) respectively; wherein the marked components of the box number include the box owner code, box registration code, check number, and box type code; the IabelImg labeling tool can directly generate a label format suitable for target detection model training;

[0049] S23: Target detection data set segmentation: the labeled data set in S22 is segmented into a target detection training set, a target detection verification set, and a target detection test set according to a preset ratio; preferably, the segmentation ratio of the corresponding training set, verification set, and test set can be set to 3:1:1, and it is necessary to ensure that there is no overlap between the data sets, so as to ensure the accuracy and reliability of the training data;

[0050] S24: Target detection model training and verification: The target detection training set and target detection verification set divided in S23 are input into each target detection model network for training, and after a specified number of training rounds, the network automatically verifies the corresponding target detection model currently obtained, evaluates the performance of the current corresponding target detection model, and formulates subsequent training strategies to obtain the current optimal performance model;

[0051] S25: Target detection model testing: The target detection test sets divided in S23 are input into each target detection model trained and verified in S24 for testing. If the performance is poor, S24 is repeated for iterative training to optimize the model, and finally optimized target detection models are generated.

[0052] It is worth noting that in the S2 target detection model generation step, its target detection model is based on the YOLO network, which is an end-to-end training model that can achieve fast and accurate target detection. Its detection process can be simply regarded as a black box, and you only need to pay more attention to the images input to the model and the detection results of the model.

[0053] In order to improve the overall recognition effect of the boxed photos, each different detection target needs to correspond to a detection model. If multiple detection targets are mixed in one model, it is easy to cause misrecognition between the targets, and it is difficult to determine the number of detection frames of each category. If there are multiple containers in the photo, more containers that do not need to be recognized will be recognized, which will interfere with the overall result and lead to low recognition rate. At the same time, each detection target is trained separately, which can have better scalability. If new detection targets need to be added later, a new detection model can be trained separately.

[0054] S3: The step of generating a box number recognition model, generating a box number recognition model based on the CRNN network, and marking the box number of the screened data set through a marking tool, and then dividing the marked data set into a box number recognition training set, a box number recognition verification set and a box number recognition test set according to a preset ratio, and inputting the divided box number recognition training set and box number recognition verification set into the box number recognition model for training, and automatically verifying the current box number recognition model after a preset number of training rounds, evaluating the performance of the box number recognition model, and then inputting the divided box number recognition test set into the trained and verified box number recognition model for testing, so as to generate an optimized box number recognition model. This step is based on marking the screened and preprocessed data set, and inputting it into the box number recognition model for training and testing. Specifically, Figure 3 Schematic diagram shown.

[0055] S31: Generation of initial container number recognition model: Generate a container number recognition model based on the CRNN network, which is a text recognition network model; preferably, based on the CRNN network, a text recognition model suitable for container number recognition is trained through transfer learning. The CRNN network is a convolutional recurrent neural network, and is also an end-to-end training model, mainly composed of CNN+RNN+CTC. The CNN convolutional layer is mainly used for image processing, extracting feature sequences from a given image, and the RNN recurrent layer is mainly used for natural language processing, making predictions based on the feature sequences generated by the CNN convolutional layer. CTC can organically combine the two and output the predicted character sequence of the RNN recurrent layer as a label, thereby realizing the recognition of indefinite-length container number text;

[0056] S32: box number recognition data set generation: through the optimized box number position detection model obtained in S25, the data set image after screening and preprocessing in S13 is intercepted to obtain images of each component of the box number area (box owner code, box registration code, check number and box type code);

[0057] S33: marking and segmenting the box number data set: marking the data set generated in S32, and segmenting it into a box number recognition training set, a box number recognition verification set, and a box number recognition test set according to a preset ratio; preferably, the segmentation ratio of the corresponding training set, verification set, and test set can be set to 3:1:1;

[0058] S34: training of container number recognition model: inputting the container number recognition training set and the container number recognition verification set divided in S33 into the container number recognition model network for training, and after the specified number of training rounds, the network automatically verifies the currently obtained container number recognition model, evaluates the performance of the current container number recognition model, and formulates subsequent training strategies to obtain the current optimal performance model;

[0059] S35: Container number recognition model test: The container number recognition test set divided in S33 is input into the container number recognition model trained in S34 for testing. If the performance is poor, S34 is repeated for iterative training to optimize the model, and finally an optimized container number recognition model is generated.

[0060] S4: Packing photo detection processing step, receiving several container packing photos uploaded by the user through the API interface, detecting the opening and closing status of the container through the optimized container opening and closing status detection model, detecting the cargo position through the optimized cargo position detection model when it is detected that the container has an open or closed state and is in a fully open or half-open type, and detecting the cargo packaging type through the optimized cargo packaging type detection model, and outputting the cargo position and cargo packaging type detection results; when it is detected that the container has an open or closed state and is in a fully closed or half-open type, identifying the container number through the optimized container number recognition model and outputting the container number recognition result; then cross-verifying the cargo position and cargo packaging type detection results with the container number recognition results to complete the intelligent detection of packing photos. Specifically, this step can be understood as the deployment and application of the packing photo detection model. By writing the detection code and generating the API interface, the optimized target detection model and box number recognition model finally generated in S2 and S3 are integrated, and the interface is deployed to the server, so that the packing photo can be input through the business system, and the interface returns information in the form of outputting the corresponding container position (i.e. container position), opening and closing status, lead seal number position, box number position, cargo position, cargo packaging type, box number and box type code in the photo. The business system further performs verification processing based on the returned detection results, such as Figure 4 The preferred flow chart is shown.

[0061] exist Figure 4 In the process, the interface receives the address of the packing photo and the box number and other data transmitted from the business system. After successfully loading the corresponding picture, it will pass through the target detection models such as the opening and closing status of the container, the location of the box number and the packaging of the goods. At the same time, the picture with the box number will pass through the box number recognition model based on the text recognition algorithm, so as to fully and accurately obtain the specific content of the box type and number. Finally, the information recognized by each model is sorted out, and the information corresponding to the box number in the business system is found according to the recognized box number for cross-checking, and the compliance of the uploaded photos is judged to select the problem data. After screening, it will be handed over to the customer service staff for further verification, which greatly reduces the workload of the customer service staff while effectively identifying illegal or problematic photos. Specifically,

[0062] S41: Writing detection code and generating interface: Using development tools, write code according to the established packing photo detection process, generate an API service interface, and receive parameters passed in by the business system; preferably, Python can be used for development, the development tool is PyCharm, and the API service interface transmission format is JSON;

[0063] S42: Interface deployment and detection result generation: The interface code written in S41 is deployed to the server. The business system passes the parameters required by each detection model through the interface. After receiving several container loading photos uploaded by the user through the API interface, the container position detection can be performed on the photos first, that is, the presence or absence of container photos can be judged. When there are no container photos (no container in the photos), the detection ends. When a container is detected, the container opening and closing state is detected by the optimized container opening and closing state detection model. When the container is detected to be in the open and closed state and in the fully closed type, the container number position is first detected by the optimized container number position detection model. The optimized container number component recognition model is used to recognize the container number components, and then the container number is recognized by the optimized container number recognition model, and the container number position detection result, the container number component recognition result and the container number recognition result are output, and then returned to the business system through the interface; when it is detected that the container has an open or closed state and is in a fully open or half-open type (that is, not a fully closed type), the cargo position is detected by the optimized cargo position detection model, and the cargo packaging type is detected by the optimized cargo packaging type detection model, and the cargo position and cargo packaging type detection results are output, and then returned to the business system through the interface, or further as follows Figure 4 As shown, when the type is fully open, it is returned to the business system through the interface, and when the type is half open, the optimized box number position detection model is used to detect the box number position, and then the optimized box number component recognition model is used to recognize the box number components, and then the optimized box number recognition model is used to recognize the box number, and the box number position detection result, the box number component recognition result and the box number recognition result are output, and then returned to the business system through the interface;

[0064] When it is detected that the container is not in the open or closed state, the seal number position is detected through the optimized seal number position detection model, and the seal number position detection result is output. The detection result is returned to the business system through the interface. The detection ends when the seal number position is not detected;

[0065] S43: Business system cross-verification: The business system cross-verifies the test results generated in S42 (box number position test results, box number component identification results, box number identification results, seal number position test results, cargo position and cargo packaging type test results) according to the box number and the business information stored in the system to check whether it is a compliant container or seal number, whether all three door opening and closing states (fully open, half open and fully closed) are available, whether there is cargo, and whether the packaging type, box number and box type code are consistent with those stored in the business system. Qualified photos that are consistent with the cross-verification do not require manual intervention to complete the intelligent detection of packing photos, and only the inconsistent cross-verification results are provided to the customer service for further processing.

[0066] The present invention optimizes the traditional manual spot check of packing photos through a more efficient, accurate and comprehensive method, realizes automatic identification and verification of packing photos, and replaces manual verification work to the greatest extent. It only needs to screen out suspected problematic photos after cross-verification and hand them over to customer service personnel for further manual review. Container packing photos that are consistent with the cross-verification are considered to be normal and qualified photos and do not require manual intervention, which greatly reduces the workload of customer service, reduces the risks and costs caused by misdetection or missed detection of illegal photos, and improves the overall competitiveness and benefits of the enterprise.

[0067] Furthermore, in the S4 packing photo detection processing step, in S42, when a container is detected, the packing photo direction detection can be automatically performed. According to the detected packing photo direction, the packing photo can be automatically rotated by 0 degrees, 90 degrees, 180 degrees or 270 degrees using the image rotation direction tool. After the rotation, the opening and closing state of the container is detected by the optimized container opening and closing state detection model, so as to further improve the intelligent detection efficiency of the packing photo.

[0068] The present invention also relates to a computer vision-based container packing photo detection system, which corresponds to the above-mentioned computer vision-based container packing photo detection method, and can be understood as a system that implements the above-mentioned computer vision-based container packing photo detection method, such as Figure 5 As shown, it includes a data set acquisition processing module, a target detection model generation module, a box number recognition model generation module and a packing photo detection processing module, wherein the data set acquisition processing module is respectively connected to the target detection model generation module and the box number recognition model generation module, the target detection model generation module is connected to the box number recognition model generation module, and the target detection model generation module and the box number recognition model generation module are both connected to the packing photo detection processing module.

[0069] The data set acquisition processing module acquires a large number of container packing photos from the business system server, wherein the container packing photos include fully open, half open and fully closed types of packing conditions, and screens the container packing photos according to preset screening conditions to obtain a screened data set;

[0070] The target detection model generation module generates several target detection models based on the YOLO network, and the target detection models include a container opening and closing state detection model, a cargo position detection model, and a cargo packaging type detection model, and respectively marks the container opening and closing state, cargo position, and cargo packaging type on the screened data set through a marking tool, and then divides the marked data set into a target detection training set, a target detection verification set, and a target detection test set according to a preset ratio, and inputs the divided target detection training set and target detection verification set into each target detection model for training, and automatically verifies the corresponding target detection model currently obtained after a preset number of training rounds, evaluates the performance of each target detection model, and then inputs the divided target detection test set into each trained and verified target detection model for testing, so as to generate optimized target detection models;

[0071] The box number recognition model generation module generates a box number recognition model based on the CRNN network, and marks the box number of the filtered data set through a marking tool, and then divides the marked data set into a box number recognition training set, a box number recognition verification set and a box number recognition test set according to a preset ratio, inputs the divided box number recognition training set and the box number recognition verification set into the box number recognition model for training, and automatically verifies the currently obtained box number recognition model after a preset number of training rounds, evaluates the performance of the box number recognition model, and then inputs the divided box number recognition test set into the trained and verified box number recognition model for testing, so as to generate an optimized box number recognition model;

[0072] The packing photo detection processing module receives a number of container packing photos uploaded by the user through the API interface, detects the opening and closing state of the container through the optimized container opening and closing state detection model, detects the cargo position through the optimized cargo position detection model when it is detected that the container has an open or closed state and is in a fully open or half-open type, and detects the cargo packaging type through the optimized cargo packaging type detection model, and outputs the cargo position and cargo packaging type detection results; when it is detected that the container has an open or closed state and is in a fully closed or half-open type, identifies the container number through the optimized container number recognition model and outputs the container number recognition result; then cross-validates the cargo position and cargo packaging type detection results with the container number recognition result to complete the intelligent detection of the packing photos.

[0073] Furthermore, in the target detection model generation module, the target detection model generated based on the YOLO network also includes a seal number position detection model and a box number position detection model, and the filtered data set is marked with the seal number position and the box number position by a marking tool, and the marked data set is divided into a target detection training set, a target detection verification set and a target detection test set according to a preset ratio, and then each target detection model is trained, verified and tested respectively to generate each optimized target detection model;

[0074] In the packing photo detection processing module, when it is detected that the container is not in an open or closed state, the seal number position is detected by the optimized seal number position detection model, and the seal number position detection result is output; when it is detected that the container is in an open or closed state and is in a fully closed or half-open type, the container number position is first detected by the optimized box number position detection model, and then the box number is identified by the optimized box number recognition model, and the box number position detection result and the box number recognition result are output; then the cargo position, cargo packaging type, seal number position and box number position detection results are cross-validated with the box number recognition results to complete the intelligent detection of the packing photos.

[0075] Furthermore, in the target detection model generation module, the target detection model generated based on the YOLO network also includes a recognition model for each component of the box number, and the filtered data set is marked with each component of the box number by a marking tool, and the marked data set is divided into a target detection training set, a target detection verification set and a target detection test set according to a preset ratio, and then the recognition model for each component of the box number is trained, verified and tested to generate an optimized recognition model for each component of the box number;

[0076] In the packing photo detection and processing module, when it is detected that the container is in an open or closed state and is in a fully closed or half-open type, the box number position is first detected by the optimized box number position detection model, and then the box number components are identified by the optimized box number component identification model, and then the box number is identified by the optimized box number identification model, and the box number is outputted. The box number position detection result, the box number component identification result and the box number identification result are outputted; then the cargo position, cargo packaging type, lead seal number position and box number position detection results, the box number component identification results and the box number identification result are cross-validated to complete the intelligent detection of the packing photo.

[0077] Furthermore, in the packing photo detection processing module, after receiving several container packing photos uploaded by the user through the API interface, the container position detection is first performed on the photos, and when the container is detected, the packing photo direction detection is automatically performed. According to the detected packing photo direction, the packing photo is automatically rotated by 0 degrees, 90 degrees, 180 degrees or 270 degrees using the image rotation direction tool. After the rotation, the container opening and closing state is detected by the optimized container opening and closing state detection model.

[0078] It should be noted that the above-described specific implementations can enable those skilled in the art to more fully understand the invention, but do not limit the invention in any way. Therefore, although this specification has described the invention in detail with reference to the drawings and embodiments, those skilled in the art should understand that the invention can still be modified or replaced by equivalents. In short, all technical solutions and improvements that do not deviate from the spirit and scope of the invention should be included in the protection scope of the patent for the invention.

Claims

1. A method for detecting container packing photos based on computer vision, characterized in that: The following steps are involved: A data set acquisition and processing step is to acquire a large number of container packing photos from a business system server, wherein the container packing photos include fully open, half-open, and fully closed types of packing conditions, and filter the container packing photos according to a preset filtering condition to obtain a filtered data set; The target detection model generation step generates several target detection models based on the YOLO network, wherein the target detection models include a container opening and closing state detection model, a cargo position detection model, and a cargo packaging type detection model, and respectively marks the container opening and closing state, the cargo position, and the cargo packaging type on the screened data set through a marking tool, and then divides the marked data set into a target detection training set, a target detection verification set, and a target detection test set according to a preset ratio, and inputs the divided target detection training set and the target detection verification set into each target detection model for training, and automatically verifies the corresponding target detection model currently obtained after a preset number of training rounds, evaluates the performance of each target detection model, and then inputs the divided target detection test set into each trained and verified target detection model for testing, so as to generate optimized target detection models; The box number recognition model generation step is to generate a box number recognition model based on the CRNN network, and mark the box number of the filtered data set through a marking tool, and then divide the marked data set into a box number recognition training set, a box number recognition verification set and a box number recognition test set according to a preset ratio, input the divided box number recognition training set and the box number recognition verification set into the box number recognition model for training, and automatically verify the current box number recognition model after a preset number of training rounds to evaluate the performance of the box number recognition model, and then input the divided box number recognition test set into the trained and verified box number recognition model for testing to generate an optimized box number recognition model; The packing photo detection processing step is to receive several container packing photos uploaded by the user through the API interface, detect the opening and closing status of the container through the optimized container opening and closing status detection model, and detect the cargo position through the optimized cargo position detection model when it is detected that the container has an open or closed state and is in a fully open or half-open type, and detect the cargo packaging type through the optimized cargo packaging type detection model, and output the cargo position and cargo packaging type detection results; when it is detected that the container has an open or closed state and is in a fully closed or half-open type, identify the container number through the optimized container number recognition model and output the container number recognition result; then cross-validate the cargo position and cargo packaging type detection results with the container number recognition results to complete the intelligent detection of packing photos.

2. The method for detecting container packing photos based on computer vision according to claim 1, characterized in that: In the target detection model generation step, the target detection model generated based on the YOLO network also includes a seal number position detection model and a box number position detection model, and the filtered data set is marked with the seal number position and the box number position by a marking tool, and the marked data set is divided into a target detection training set, a target detection verification set and a target detection test set according to a preset ratio, and then each target detection model is trained, verified and tested respectively to generate each optimized target detection model; In the packing photo detection processing step, when it is detected that the container is not in an open or closed state, the seal number position is detected by the optimized seal number position detection model, and the seal number position detection result is output; when it is detected that the container is in an open or closed state and is in a fully closed or half-open type, the container number position is first detected by the optimized box number position detection model, and then the box number is identified by the optimized box number recognition model, and the box number position detection result and the box number recognition result are output; then the cargo position, cargo packaging type, seal number position and box number position detection results are cross-validated with the box number recognition results to complete the intelligent detection of the packing photos.

3. The method for detecting container packing photos based on computer vision according to claim 2, characterized in that: In the target detection model generation step, the target detection model generated based on the YOLO network also includes a recognition model for each component of the box number, and the filtered data set is marked with each component of the box number by a marking tool, and the marked data set is divided into a target detection training set, a target detection verification set and a target detection test set according to a preset ratio, and then the recognition model for each component of the box number is trained, verified and tested to generate an optimized recognition model for each component of the box number; In the packing photo detection processing step, when it is detected that the container is in an open or closed state and is in a fully closed or half-open type, the container number position is first detected by the optimized container number position detection model, and then the container number components are identified by the optimized container number component recognition model, and then the container number is identified by the optimized container number recognition model, and then the container number is identified by the optimized container number recognition model, and the container number position detection result, the container number component recognition result and the container number recognition result are output; then the cargo position, cargo packaging type, lead seal number position and container number position detection results, the container number component recognition results and the container number recognition result are cross-validated to complete the intelligent detection of the packing photo.

4. The method for detecting container packing photos based on computer vision according to any one of claims 1 to 3, characterized in that: In the packing photo detection processing step, after receiving a number of container packing photos uploaded by the user through the API interface, the container position is first detected on the photos, and when the container is detected, the container opening and closing status is detected by using the optimized container opening and closing status detection model.

5. The method for detecting container packing photos based on computer vision according to claim 4, characterized in that: In the packing photo detection processing step, when a container is detected, the packing photo direction detection is automatically performed, and the packing photo is automatically rotated 0 degrees, 90 degrees, 180 degrees or 270 degrees using a rotation picture direction tool according to the detected packing photo direction. After the rotation, the container opening and closing state is detected using an optimized container opening and closing state detection model.

6. The method for detecting container packing photos based on computer vision according to claim 4, characterized in that: In the data set acquisition processing step, the screened data set is also subjected to image graying, binarization, balancing of the number of samples in each category, image size adjustment, normalization and tensorization processing to obtain the screened and preprocessed data set; the subsequent target detection model generation step and box number recognition model generation step are to use a marking tool to mark the screened and preprocessed data set accordingly, and the corresponding training set, validation set and test set are divided in a ratio of 3:1:

1.

7. A container packing photo detection system based on computer vision, characterized in that: It includes a data set acquisition processing module, a target detection model generation module, a box number recognition model generation module and a packing photo detection processing module. The data set acquisition processing module is connected to the target detection model generation module and the box number recognition model generation module respectively. The target detection model generation module and the box number recognition model generation module are both connected to the packing photo detection processing module. The data set acquisition processing module acquires a large number of container packing photos from the business system server, wherein the container packing photos include fully open, half open and fully closed types of packing conditions, and screens the container packing photos according to preset screening conditions to obtain a screened data set; The target detection model generation module generates several target detection models based on the YOLO network, and the target detection models include a container opening and closing state detection model, a cargo position detection model, and a cargo packaging type detection model, and respectively marks the container opening and closing state, cargo position, and cargo packaging type on the screened data set through a marking tool, and then divides the marked data set into a target detection training set, a target detection verification set, and a target detection test set according to a preset ratio, and inputs the divided target detection training set and target detection verification set into each target detection model for training, and automatically verifies the corresponding target detection model currently obtained after a preset number of training rounds, evaluates the performance of each target detection model, and then inputs the divided target detection test set into each trained and verified target detection model for testing, so as to generate optimized target detection models; The box number recognition model generation module generates a box number recognition model based on the CRNN network, and marks the box number of the filtered data set through a marking tool, and then divides the marked data set into a box number recognition training set, a box number recognition verification set and a box number recognition test set according to a preset ratio, inputs the divided box number recognition training set and the box number recognition verification set into the box number recognition model for training, and automatically verifies the currently obtained box number recognition model after a preset number of training rounds, evaluates the performance of the box number recognition model, and then inputs the divided box number recognition test set into the trained and verified box number recognition model for testing, so as to generate an optimized box number recognition model; The packing photo detection processing module receives a number of container packing photos uploaded by the user through the API interface, detects the opening and closing state of the container through the optimized container opening and closing state detection model, detects the cargo position through the optimized cargo position detection model when it is detected that the container has an open or closed state and is in a fully open or half-open type, and detects the cargo packaging type through the optimized cargo packaging type detection model, and outputs the cargo position and cargo packaging type detection results; when it is detected that the container has an open or closed state and is in a fully closed or half-open type, identifies the container number through the optimized container number recognition model and outputs the container number recognition result; then cross-validates the cargo position and cargo packaging type detection results with the container number recognition result to complete the intelligent detection of the packing photos.

8. The computer vision-based container packing photo detection system according to claim 7, characterized in that: In the target detection model generation module, the target detection model generated based on the YOLO network also includes a seal number position detection model and a box number position detection model. The filtered data set is marked with the seal number position and the box number position by a marking tool, and the marked data set is divided into a target detection training set, a target detection verification set and a target detection test set according to a preset ratio. Then, each target detection model is trained, verified and tested respectively to generate optimized target detection models. In the packing photo detection processing module, when it is detected that the container is not in an open or closed state, the seal number position is detected by the optimized seal number position detection model, and the seal number position detection result is output; when it is detected that the container is in an open or closed state and is in a fully closed or half-open type, the container number position is first detected by the optimized box number position detection model, and then the box number is identified by the optimized box number recognition model, and the box number position detection result and the box number recognition result are output; then the cargo position, cargo packaging type, seal number position and box number position detection results are cross-validated with the box number recognition results to complete the intelligent detection of the packing photos.

9. The computer vision-based container packing photo detection system according to claim 8, characterized in that: In the target detection model generation module, the target detection model generated based on the YOLO network also includes a recognition model for each component of the box number, and the filtered data set is marked with each component of the box number through a marking tool, and the marked data set is divided into a target detection training set, a target detection verification set and a target detection test set according to a preset ratio, and then the recognition model for each component of the box number is trained, verified and tested to generate an optimized recognition model for each component of the box number; In the packing photo detection and processing module, when it is detected that the container is in an open or closed state and is in a fully closed or half-open type, the box number position is first detected by the optimized box number position detection model, and then the box number components are identified by the optimized box number component identification model, and then the box number is identified by the optimized box number identification model, and the box number is outputted. The box number position detection result, the box number component identification result and the box number identification result are outputted; then the cargo position, cargo packaging type, lead seal number position and box number position detection results, the box number component identification results and the box number identification result are cross-validated to complete the intelligent detection of the packing photo.

10. The computer vision-based container packing photo detection system according to any one of claims 7 to 9, characterized in that: In the packing photo detection processing module, after receiving several container packing photos uploaded by the user through the API interface, the container position detection is first performed on the photos. When the container is detected, the packing photo direction detection is also automatically performed. According to the detected packing photo direction, the packing photo is automatically rotated 0 degrees, 90 degrees, 180 degrees or 270 degrees using the image rotation direction tool. After the rotation, the container opening and closing state is detected by the optimized container opening and closing state detection model.

Images