Cargo congestion identification method and device, electronic equipment and storage medium

Abstract

The application provides a cargo congestion identification method and device, electronic equipment and storage medium; the cargo congestion identification method in the application comprises: acquiring initial video information of a cargo conveying line sent by a shooting device; inputting the initial video information into a preset image segmentation model to extract effective video information from the initial video information through the image segmentation model; inputting the effective video information into a preset video classification model to identify the effective video information through the video classification model and obtain a cargo congestion identification result; in the application, the initial video information is segmented through the image segmentation model to obtain effective video information, and the effective video information is classified and identified through the video classification model to obtain the cargo congestion identification result, so that the amount of video information processed during cargo congestion identification is small, and the efficiency and accuracy of cargo congestion identification are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of logistics, in particular to a cargo congestion identification method and device, electronic equipment and storage medium. BACKGROUND

[0002] With the development of the Internet and online shopping, the logistics industry has also thrived. After receiving express items, the logistics company generally performs multiple steps of sorting and packaging processing. In these processes, a conveyor belt is generally used to transport express items. For example, a belt conveyor is used. In the logistics industry, a transmission belt is the most common transmission mechanism applied to loading and unloading ports. Basically, every express item needs to pass through the transmission belt for transmission. Therefore, the goods on the transmission belt have good operation, which can improve the efficiency of goods transportation.

[0003] At present, there are many factors affecting the operation of goods on the transmission belt. Cargo congestion is a common problem and a common cause of express item damage. At present, the number of transmission belts in the logistics network is generally large, and the occupied area is large, especially multi-layer transmission belts and cross-floor transmission belts, which are much higher than the height of a person. Therefore, the running state of the transmission belt cannot be seen at all, so when congestion occurs, it cannot be discovered in time. At present, monitoring equipment is used for monitoring, and then a prompt is given for cargo congestion. However, such a cargo congestion identification method usually requires a large amount of cargo accumulation before the congestion prompt is given. Such a congestion identification method is low in efficiency and prone to errors. SUMMARY

[0004] The present application provides a cargo congestion identification method, device, electronic equipment and storage medium, which aims to solve the technical problems of large data processing amount, low data processing efficiency and low recognition accuracy of the existing cargo congestion identification data.

[0005] In one aspect, the present application provides a cargo congestion identification method, which comprises the following steps:

[0006] Obtaining initial video information of a cargo conveying line sent by a shooting device;

[0007] Inputting the initial video information into a preset image segmentation model to extract effective video information from the initial video information through the image segmentation model;

[0008] Inputting the effective video information into a preset video classification model to identify the effective video information through the video classification model and obtaining a cargo congestion identification result.

[0009] In some embodiments of the present application, before the initial video information is input into the preset image segmentation model to extract the effective video information from the initial video information through the image segmentation model, the method comprises:

[0010] Obtain image samples of the marked cargo conveyor line area, and summarize the image samples to form an image sample set;

[0011] A preset proportion of image samples are extracted from the image sample set at one time to construct an initial image segmentation model using the image samples;

[0012] Iteratively extract a preset proportion of image samples from the image sample set to train the initial image segmentation model using the image samples, thereby obtaining an image segmentation training model;

[0013] Obtain the segmentation accuracy of the image segmentation training model, and use the image segmentation training model with a segmentation accuracy higher than the preset segmentation accuracy as the preset image segmentation model.

[0014] In some embodiments of this application, the step of inputting the initial video information into a preset image segmentation model to extract valid video information from the initial video information through the image segmentation model includes:

[0015] The initial video information is input into a preset image segmentation model to segment the initial video information into frames, thereby obtaining continuous initial video frames;

[0016] The initial video frames are segmented one by one using the preset image segmentation model to determine the cargo conveyor area of ​​each initial video frame;

[0017] The region other than the cargo conveyor area in the initial video frame is removed to obtain valid video information.

[0018] In some embodiments of this application, the step of inputting the valid video information into a preset video classification model to identify the valid video information through the video classification model and obtain a cargo congestion identification result includes:

[0019] The valid video information is input into a preset video classification model to determine whether the conveyor belt in the cargo conveyor area corresponding to the valid video information is in a stationary state.

[0020] If the conveyor belt in the cargo conveyor area is stationary, the preset video classification model is used to identify whether there is cargo on the stationary conveyor belt.

[0021] If there are no goods on the conveyor belt in the stationary state, the result of no congestion is output.

[0022] If there are goods on the conveyor belt in the stationary state, the congestion identification result is output.

[0023] In some embodiments of this application, after inputting the valid video information into a preset video classification model to determine whether the conveyor belt in the cargo conveyor area corresponding to the valid video information is in a stationary state, the method includes:

[0024] If the conveyor belt in the cargo conveyor area is in operation, the preset video classification model is used to determine whether there are relatively moving goods on the conveyor belt in operation.

[0025] If there are goods in relative motion on the conveyor belt during operation, a conveyor belt slippage warning will be output.

[0026] In some embodiments of this application, the step of outputting a cargo congestion identification result if there are goods on the conveyor belt in the stationary state includes:

[0027] If there are goods on the conveyor belt in the stationary state, the preset video classification model is used to determine whether the number of goods on the conveyor belt in the stationary state has increased.

[0028] If the number of goods on the conveyor belt in the stationary state increases, the congestion identification result is output.

[0029] If the number of goods on the conveyor belt in the stationary state does not increase, then it is determined whether there are goods within the preset range of the conveyor belt in the stationary state.

[0030] If there are no goods within the preset range of the conveyor belt in the stationary state, the result of goods not being congested is output.

[0031] If there are goods within the preset range of the conveyor belt in the stationary state, the goods congestion identification result will be output.

[0032] In some embodiments of this application, before acquiring the initial video information of the cargo conveyor line sent by the shooting device, the method further includes:

[0033] Receive a cargo congestion identification command and obtain the target monitoring range corresponding to the cargo congestion identification command;

[0034] Obtain the shooting range of a preset fixed camera, and determine whether the shooting range of the preset fixed camera covers the target monitoring range;

[0035] The acquisition of initial video information of the cargo conveyor line sent by the shooting device includes:

[0036] If the shooting range of the preset fixed camera does not cover the target monitoring range, a call command is sent to the preset aerial drone to call the aerial drone to shoot the uncovered area within the target monitoring range, thereby obtaining the initial video information of the cargo conveyor line sent by the preset fixed camera and the aerial drone.

[0037] On the other hand, this application provides a cargo congestion identification device, the cargo congestion identification device comprising:

[0038] The video acquisition module is used to acquire the initial video information of the cargo conveyor line sent by the shooting device.

[0039] The segmentation and extraction module is used to input the initial video information into a preset image segmentation model, so as to extract effective video information from the initial video information through the image segmentation model;

[0040] The input recognition module is used to input the valid video information into a preset video classification model, so as to identify the valid video information through the video classification model and obtain the cargo congestion recognition result.

[0041] In some embodiments of this application, the cargo congestion identification device includes:

[0042] The sample acquisition module is used to acquire image samples of the marked cargo conveyor line area and summarize the image samples to form an image sample set;

[0043] The model building module is used to extract a preset proportion of image samples from the image sample set at one time, so as to build an initial image segmentation model through the image samples;

[0044] The model training module is used to iteratively extract a preset proportion of image samples from the image sample set to train the initial image segmentation model through the image samples and obtain an image segmentation training model.

[0045] The model determination module is used to obtain the segmentation accuracy of the image segmentation training model and to use the image segmentation training model with a segmentation accuracy higher than the preset segmentation accuracy as the preset image segmentation model.

[0046] In some embodiments of this application, the segmentation and extraction module is specifically used for:

[0047] The initial video information is input into a preset image segmentation model to segment the initial video information into frames, thereby obtaining continuous initial video frames;

[0048] The initial video frames are segmented one by one using the preset image segmentation model to determine the cargo conveyor area of ​​each initial video frame;

[0049] The region other than the cargo conveyor area in the initial video frame is removed to obtain valid video information.

[0050] In some embodiments of this application, the input recognition module is specifically used for:

[0051] The valid video information is input into a preset video classification model to determine whether the conveyor belt in the cargo conveyor area corresponding to the valid video information is in a stationary state.

[0052] If the conveyor belt in the cargo conveyor area is stationary, the preset video classification model is used to identify whether there is cargo on the stationary conveyor belt.

[0053] If there are no goods on the conveyor belt in the stationary state, the result of no congestion is output.

[0054] If there are goods on the conveyor belt in the stationary state, the congestion identification result is output.

[0055] In some embodiments of this application, the input recognition module is specifically used for:

[0056] If the conveyor belt in the cargo conveyor area is in operation, the preset video classification model is used to determine whether there are relatively moving goods on the conveyor belt in operation.

[0057] If there are goods in relative motion on the conveyor belt during operation, a conveyor belt slippage warning will be output.

[0058] In some embodiments of this application, the input recognition module is specifically used for:

[0059] If there are goods on the conveyor belt in the stationary state, the preset video classification model is used to determine whether the number of goods on the conveyor belt in the stationary state has increased.

[0060] If the number of goods on the conveyor belt in the stationary state increases, the congestion identification result is output.

[0061] If the number of goods on the conveyor belt in the stationary state does not increase, then it is determined whether there are goods within the preset range of the conveyor belt in the stationary state.

[0062] If there are no goods within the preset range of the conveyor belt in the stationary state, the result of goods not being congested is output.

[0063] If there are goods within the preset range of the conveyor belt in the stationary state, the goods congestion identification result will be output.

[0064] In some embodiments of this application, the cargo congestion identification device further includes:

[0065] The receiving and acquiring module is used to receive cargo congestion identification instructions and obtain the target monitoring range corresponding to the cargo congestion identification instructions;

[0066] The range determination module is used to obtain the shooting range of the camera and determine whether the shooting range of the camera covers the target monitoring range;

[0067] The device invocation module is used to invoke a drone to photograph the uncovered area within the target monitoring range if the camera's shooting range does not cover the target monitoring range.

[0068] Using the camera and the drone as shooting devices, the video acquisition module is executed to acquire the initial video information of the cargo conveyor line sent by the shooting device.

[0069] On the other hand, this application also provides an electronic device, the electronic device comprising:

[0070] One or more processors;

[0071] Memory; and

[0072] One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the processor to implement the cargo congestion identification method.

[0073] On the other hand, this application also provides a computer-readable storage medium having a computer program stored thereon, the computer program being loaded by a processor to perform the steps in the cargo congestion identification method.

[0074] The technical solution of this application acquires initial video information of the cargo conveyor line sent by the shooting device; inputs the initial video information into a preset image segmentation model to extract effective video information from the initial video information; inputs the effective video information into a preset video classification model to identify the effective video information and obtain cargo congestion identification results. In this embodiment, the initial video information is segmented by the image segmentation model to obtain effective video information, and the effective video information is classified and identified by the video classification model to obtain cargo congestion identification results. This eliminates the need to analyze the captured noisy video information, directly extracts the effective video information corresponding to the cargo conveyor line from the initial video information, and identifies the effective video information to obtain cargo congestion identification results. The technical solution of this application has a small amount of video information processing, which improves the efficiency and accuracy of cargo congestion identification. Attached Figure Description

[0075] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0076] Figure 1 This is a schematic diagram of a cargo congestion identification scenario provided in an embodiment of this application;

[0077] Figure 2 This is a schematic flowchart of an embodiment of the cargo congestion identification method provided in this application.

[0078] Figure 3 This is a schematic diagram of an embodiment of the image segmentation model construction in the cargo congestion identification method of this application;

[0079] Figure 4 This is a flowchart illustrating one embodiment of the cargo congestion identification method provided in this application.

[0080] Figure 5 This is a flowchart illustrating another embodiment of the cargo congestion identification method provided in this application.

[0081] Figure 6 This is a flowchart illustrating an embodiment of the cargo congestion identification method provided in this application for obtaining initial video information;

[0082] Figure 7 This is a schematic diagram of an embodiment of the cargo congestion identification device provided in this application.

[0083] Figure 8 This is a schematic diagram of an embodiment of the electronic device provided in this application. Detailed Implementation

[0084] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of the present invention.

[0085] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientation or positional relationships based on the orientation or positional relationships shown in the accompanying drawings, are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include one or more of the stated features. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0086] In this application, the term "exemplary" is used to mean "serving as an example, illustration, or description." Any embodiment described as "exemplary" in this application is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is provided to enable any person skilled in the art to make and use the invention. Details are set forth in the following description for purposes of explanation. It should be understood that those skilled in the art will recognize that the invention can be made without using these specific details. In other instances, well-known structures and processes will not be described in detail to avoid obscuring the description of the invention with unnecessary detail. Therefore, the invention is not intended to be limited to the embodiments shown, but is consistent with the broadest scope of the principles and features disclosed in this application.

[0087] This application provides a method, apparatus, device, and computer-readable storage medium for identifying cargo congestion, which will be described in detail below.

[0088] The cargo congestion identification method in this application embodiment is applied to a cargo congestion identification device, which is set in an electronic device. The electronic device is provided with one or more processors, a memory, and one or more applications, wherein one or more applications are stored in the memory and configured to be executed by the processor to implement the cargo congestion identification method. The electronic device can be a terminal, such as a mobile phone or a tablet computer. The electronic device can also be a server or a service cluster composed of multiple servers.

[0089] like Figure 1 As shown, Figure 1This is a schematic diagram of a scenario for cargo congestion identification according to an embodiment of this application. The scenario for cargo congestion identification in this embodiment includes an electronic device 100 (the electronic device 100 integrates a cargo congestion identification device). The electronic device 100 runs a computer-readable storage medium corresponding to cargo congestion identification to perform the steps of cargo congestion identification.

[0090] Understandable Figure 1 The electronic devices in the scenario of cargo congestion identification, or the devices contained in the electronic devices, do not constitute a limitation on the embodiments of this application. That is, the number or type of devices in the scenario of cargo congestion identification, or the number or type of devices contained in each device, do not affect the overall implementation of the technical solution in the embodiments of this application, and can all be considered as equivalent substitutions or derivatives of the technical solutions claimed in the embodiments of this application.

[0091] In this embodiment, the electronic device 100 is mainly used to acquire initial video information of the cargo conveyor line sent by the shooting device; input the initial video information into a preset image segmentation model to extract effective video information from the initial video information through the image segmentation model; input the effective video information into a preset video classification model to identify the effective video information through the video classification model and obtain cargo congestion identification results.

[0092] In this embodiment, the electronic device 100 can be an independent electronic device, or it can be a network or cluster of electronic devices. For example, the electronic device 100 described in this embodiment includes, but is not limited to, a computer, a network host, a single network electronic device, a set of multiple network electronic devices, or a cloud electronic device composed of multiple electronic devices. The cloud electronic device is composed of a large number of computers or network electronic devices based on cloud computing.

[0093] Those skilled in the art will understand that Figure 1 The application environment shown is merely one application scenario of the solution in this application and does not constitute a limitation on the application scenario of the solution in this application. Other application environments may include those that are more specific to this application. Figure 1 The number of more or fewer electronic devices shown, or the network connections of electronic devices, for example Figure 1 Only one electronic device is shown in the image. It is understood that the scenario for cargo congestion identification may also include one or more other electronic devices, which are not specifically limited here. The electronic device 100 may also include a memory for storing data, such as storing video information obtained by shooting.

[0094] Furthermore, in the scenario of cargo congestion identification in this application, the electronic device 100 may be equipped with a display device, or the electronic device 100 may not have a display device but may communicate with an external display device 200. The display device 200 is used to output the results of the cargo congestion identification method executed by the electronic device. The electronic device 100 can access the background database 300 (the background database may be located in the local storage of the electronic device or it may be located in the cloud). The background database 300 stores information related to cargo congestion identification, such as information consistent with surveillance video.

[0095] It should be noted that, Figure 1 The schematic diagram of the cargo congestion identification scenario shown is merely an example. The cargo congestion identification scenario described in this application embodiment is intended to more clearly illustrate the technical solution of this application embodiment and does not constitute a limitation on the technical solution provided in this application embodiment.

[0096] Based on the above-mentioned scenario of cargo congestion identification, an embodiment of the cargo congestion identification method is proposed.

[0097] like Figure 2 As shown, Figure 2 This is a schematic flowchart of an embodiment of the cargo congestion identification method in this application, which includes the following steps 201-203:

[0098] 201. Obtain the initial video information of the cargo conveyor line sent by the shooting device.

[0099] The cargo congestion identification method in this application embodiment is applied to electronic devices. The type of electronic device is not specifically limited. In this application embodiment, a terminal is used as an example for explanation. The terminal is communicatively connected to a shooting device. The type and number of shooting devices are not specifically limited. For example, the shooting device can be a camera or a drone, etc. The shooting device is used to capture initial video information of the cargo transportation line. The camera sends the captured initial video information to the terminal. The terminal receives the initial video information of the cargo transportation line sent by the camera.

[0100] Understandably, each video frame of the initial video information contains the cargo transport line area and some noisy areas, such as aisles, cargo handling workers, etc. The terminal analyzes the initial video information to determine whether there is cargo congestion on the cargo transport line, and then outputs the congestion identification result.

[0101] Because directly identifying cargo congestion from the initial video information results in a large amount of video information processing for the terminal, low data processing efficiency, and a high risk of errors; therefore, in this embodiment, the terminal first performs segmentation processing on the initial video information. Specifically, the terminal has a preset image segmentation model, which refers to an analysis algorithm obtained through neural network learning. The image segmentation model performs image segmentation on the video frames in the initial video information to locate the cargo conveyor line in the video frames.

[0102] It should be further noted that the segmentation strategy of the image segmentation model in this application embodiment is not specifically limited. Specifically, the image segmentation model performs image segmentation in two strategies: Strategy 1: Divide the image into regions based on the similarity or homogeneity of the gray values ​​of each video frame in the initial video information. The cargo conveyor line region in the initial video information is generated by merging some regions, such as thresholding, clustering, region separation, and region fusion. Strategy 2: Find the location of the target object or its outline based on the discontinuity and abrupt change of the gray values ​​of each video frame in the initial video information, and extend it spatially based on the location. That is, perform image segmentation by detecting the feature points, lines, and surfaces of the image, such as edge detection, to determine the cargo conveyor line region in the initial video information.

[0103] 202. Input the initial video information into a preset image segmentation model to extract valid video information from the initial video information through the image segmentation model.

[0104] Initial video information is input into a preset image segmentation model. The image segmentation model divides the initial video information into frames, forming continuous video frames. Then, the terminal segments each video frame using the image segmentation model to finally obtain valid video information, specifically including:

[0105] (1) Input the initial video information into a preset image segmentation model to divide the initial video information into frames and obtain continuous initial video frames;

[0106] (2) The initial video frames are segmented one by one using the preset image segmentation model to determine the cargo conveyor area of ​​each initial video frame;

[0107] (3) Remove the area in the initial video frame other than the cargo conveyor area to obtain valid video information.

[0108] That is, the initial video information is input into a preset image segmentation model, which segments the initial video information into frames to obtain continuous initial video frames. For example, if the length of the initial video information is 10 seconds, the terminal outputs a frame segmentation setting prompt so that the user can set the frame segmentation time interval or the total number of frames. The terminal segments the initial video information into frames according to the set frame segmentation time interval or the total number of frames to obtain continuous initial video frames. Then, the terminal segments the initial video frames one by one using the preset image segmentation model to determine the cargo conveyor area of ​​each initial video frame. That is, the image segmentation model first determines the boundary line of the cargo conveyor or the feature points of the cargo conveyor, identifies the cargo conveyor area of ​​each initial video frame through the image segmentation model, and finally, the terminal removes the areas other than the cargo conveyor area from the initial video frames, retaining the cargo conveyor area. The terminal retains each continuous video frame of the retained cargo conveyor area to obtain valid video information.

[0109] In this embodiment, the terminal obtains effective video information of the cargo conveyor line area in the initial video information by segmenting the initial video information. In this way, when identifying cargo congestion, only the cargo conveyor line area needs to be identified, reducing the amount of information analysis required for video information identification.

[0110] 203. Input the valid video information into a preset video classification model to identify the valid video information through the video classification model and obtain the cargo congestion identification result.

[0111] The terminal in this application embodiment has a preset video classification model. The video classification model refers to a classification algorithm obtained through deep learning of neural networks. That is, the terminal trains the initial video classification model with image samples to obtain the video classification model, such as a Support Vector Machine (SVM) video classifier.

[0112] The terminal inputs valid video information into a preset video classification model. The video classification model first identifies the transport belt in the valid video information. Then, it combines and analyzes multiple video frames to determine whether the transport belt is stationary. If the transport belt is in motion, it outputs a result indicating that the goods are not congested. If the transport belt is stationary, it further identifies whether there are goods on the stationary transport belt. If there are no goods on the stationary transport belt, it outputs a result indicating that the goods are not congested. If there are goods on the stationary transport belt, it outputs a result indicating that the goods are congested.

[0113] For example, conveyor belt states can typically be categorized into three types: 1. The conveyor belt is operating and goods are being transported normally along it; 2. The conveyor belt is stationary but there are no goods on it; 3. The conveyor belt is stationary and goods are piled up statically on it. The third state is the congestion state that needs to be identified in this embodiment. Through video classification models, it can be seen that the states the video classification model needs to identify are dynamic, meaning that a single frame image cannot determine whether the conveyor belt and goods are moving or stationary; multiple frames must be used to determine this.

[0114] In this embodiment, the initial video information is segmented using an image segmentation model to obtain valid video information. The valid video information is then classified and identified using a video classification model to obtain the cargo congestion identification result. This eliminates the need to analyze the captured noisy video information. Instead, the valid video information corresponding to the cargo conveyor line is directly extracted from the initial video information, and the valid video information is identified to obtain the cargo congestion identification result. The technical solution of this application requires less video information processing, thus improving the efficiency and accuracy of cargo congestion identification.

[0115] like Figure 3 As shown, Figure 3 This is a schematic diagram of an embodiment of the cargo congestion identification method in this application for constructing an image segmentation model.

[0116] In some embodiments of this application, the steps for constructing an image segmentation model are specifically described, including the following steps 301 to 304:

[0117] 301. Obtain image samples of the marked cargo conveyor area, and summarize the image samples to form an image sample set.

[0118] Specifically, the terminal can acquire massive amounts of image information through various channels (such as images captured by various cameras at the network points). The terminal uses the image information containing the goods conveyor line as image samples. The terminal outputs annotation prompts to prompt the user to annotate the goods conveyor line area and non-goods conveyor line areas in the image samples. The terminal acquires image samples with annotated goods conveyor line areas and summarizes the image samples with annotated goods conveyor line areas to form an image sample set, so as to train the image segmentation model using the image samples in the image sample set.

[0119] 302. Extract a preset proportion of image samples from the image sample set at one time to construct an initial image segmentation model using the image samples.

[0120] Specifically, a preset proportion of image samples can be extracted from the image sample set at once (the preset proportion can be flexibly set according to the specific scenario, for example, the preset proportion can be set to 1%). The terminal constructs an initial image segmentation model by extracting the preset proportion of image samples at once. That is, the terminal extracts feature points in the image samples, and then uses the feature points to construct a segmentation function. The terminal uses the segmentation function as the initial image segmentation model.

[0121] 303. Iteratively extract a preset proportion of image samples from the image sample set to train the initial image segmentation model using the image samples, thereby obtaining an image segmentation training model.

[0122] Specifically, a preset proportion of image samples (which can be flexibly set according to specific scenarios, for example, the preset proportion can be set to 1%) can be iteratively extracted from the image sample set. The terminal trains an initial image segmentation model using the image samples to obtain an image segmentation training model. That is, the terminal obtains the feature points of the image samples and then adjusts the parameters of the segmentation function according to the feature points to iteratively train the constructed initial image segmentation model to obtain the trained image segmentation model.

[0123] 304. Obtain the segmentation accuracy of the image segmentation training model, and use the image segmentation training model with a segmentation accuracy higher than the preset segmentation accuracy as the preset image segmentation model.

[0124] The terminal obtains the segmentation accuracy of the image segmentation training model and compares it with a preset segmentation accuracy (the preset segmentation accuracy refers to a pre-set image segmentation accuracy threshold; if the accuracy of the trained image segmentation model is higher than the threshold, training can be stopped; otherwise, if the accuracy is lower than the threshold, iterative training continues; the preset segmentation accuracy can be set to 98%). If the segmentation accuracy is lower than the preset accuracy, iterative training continues; if the accuracy is higher than the preset accuracy, the trained image segmentation function is considered converged, and the terminal uses the image segmentation training model with a higher accuracy as the preset image segmentation model.

[0125] This application embodiment specifically illustrates the construction steps of the image segmentation model. By constructing the image segmentation model to segment the initial video information, the amount of data analysis during cargo congestion identification can be reduced, and noise information can be effectively eliminated, thereby improving the accuracy of cargo congestion identification.

[0126] Reference Figure 4 , Figure 4This is a flowchart illustrating one embodiment of the cargo congestion identification method provided in this application. In some embodiments of this application, the steps for obtaining cargo congestion identification results through effective video information analysis using a video classification model are specifically described, including the following steps 401-404:

[0127] 401. Input the valid video information into a preset video classification model to determine whether the conveyor belt in the cargo conveyor area corresponding to the valid video information is in a stationary state.

[0128] Specifically, the terminal inputs the valid video information into a preset video classification model. The terminal compares adjacent video frames in the valid video information using the video classification model. If the similarity of adjacent video frames is not higher than the preset video frame similarity (the preset video frame similarity can be set according to the specific scenario, for example, the preset video frame similarity is set to 95%), then the conveyor belt in the goods conveyor area is determined to be in motion. Conversely, if the similarity of adjacent video frames is higher than the preset video frame similarity, then the conveyor belt in the goods conveyor area is determined to be in a stationary state.

[0129] 402. If the conveyor belt in the cargo conveyor area is stationary, the preset video classification model is used to identify whether there is cargo on the stationary conveyor belt.

[0130] If the conveyor belt in the cargo conveyor area is stationary, the terminal performs bounding box detection using a video classification model. If there are feature points on the conveyor belt corresponding to rectangular bounding boxes, the terminal determines that there are goods on the stationary conveyor belt; if there are no feature points on the conveyor belt corresponding to rectangular bounding boxes, the terminal determines that there are no goods on the stationary conveyor belt.

[0131] 403. If there are no goods on the conveyor belt in the stationary state, output the result of goods not being congested.

[0132] If there are no goods on the conveyor belt when it is stationary, the terminal outputs a no-congestion identification result. In this embodiment, the terminal outputs a no-congestion identification result when the conveyor belt is stationary and there are no goods.

[0133] 404. If there are goods on the conveyor belt in the stationary state, output the goods congestion identification result.

[0134] If there are goods on the conveyor belt when it is stationary, the terminal further outputs a congestion identification result based on changes in the goods on the conveyor belt, specifically including:

[0135] (1) If there are goods on the conveyor belt in the stationary state, the number of goods on the conveyor belt in the stationary state is determined by the preset video classification model.

[0136] (2) If the number of goods on the conveyor belt in the stationary state increases, the congestion identification result of the goods is output.

[0137] (3) If the number of goods on the conveyor belt in the stationary state does not increase, then determine whether there are goods within the preset range of the conveyor belt in the stationary state.

[0138] (4) If there are no goods within the preset range of the conveyor belt in the stationary state, output the goods non-congestion identification result;

[0139] (5) If there are goods within the preset range of the conveyor belt in the stationary state, the goods congestion identification result is output.

[0140] That is, if there are goods on the stationary conveyor belt, the terminal determines whether the number of goods on the stationary conveyor belt has increased using a preset video classification model; if the number of goods on the stationary conveyor belt increases, it indicates that the conveyor belt has a conveying task, and the terminal outputs a goods congestion identification result; if the number of goods on the stationary conveyor belt does not increase, the terminal further determines whether there are goods in the preset area around the stationary conveyor belt (the preset range can be set according to the specific scenario, for example, the preset range is 1 meter of the conveyor belt); if there are no goods in the preset range of the stationary conveyor belt, the terminal outputs a goods non-congestion identification result; if there are goods in the preset range of the stationary conveyor belt, the terminal outputs a goods congestion identification result.

[0141] In this embodiment of the application, when the terminal identifies cargo congestion, in order to prevent incorrect judgment of cargo congestion when there is cargo on the conveyor belt, the terminal combines the change in the number of goods on the conveyor belt with the change in the number of goods within a preset range on the conveyor belt. In this way, it can identify whether the conveyor belt has stopped working or the cargo transportation has failed. In this embodiment of the application, cargo congestion can be accurately identified.

[0142] Reference Figure 5 , Figure 5 This is a flowchart illustrating another embodiment of the cargo congestion identification method provided in this application.

[0143] In some embodiments of this application, after the terminal inputs valid video information into a preset video classification model and determines whether the conveyor belt in the cargo conveyor area corresponding to the valid video information is in a stationary state, the following steps 501-502 are further included:

[0144] 501. If the conveyor belt in the cargo conveyor area is in operation, the preset video classification model is used to determine whether there are relatively moving goods on the conveyor belt in operation.

[0145] In this embodiment, the terminal determines that the conveyor belt in the cargo conveying line area is in operation. That is, the terminal compares adjacent video frames in the effective video information and determines that the similarity of adjacent video frames is not higher than the preset video frame similarity. The terminal then determines whether there are relatively moving goods on the conveyor belt in operation through a preset video classification model. The purpose of the terminal's determination of whether there are relatively moving goods on the conveyor belt in operation in this embodiment is to determine whether the conveyor belt is slipping.

[0146] 502. If there are goods moving relative to each other on the conveyor belt in the operating state, an alert indicating that the conveyor belt is slipping will be output.

[0147] If there are goods moving relative to the conveyor belt in operation, a conveyor belt slippage warning is output; otherwise, if there are no goods moving relative to the conveyor belt in operation, the terminal determines that the conveyor belt is in normal working condition. In this embodiment, when the conveyor belt is moving, there are goods moving relative to the conveyor belt in operation. This can effectively monitor whether the conveyor belt is slipping, avoid the loss of small items during transportation, and improve the intelligence of cargo congestion identification.

[0148] Reference Figure 6 , Figure 6 This is a flowchart illustrating an embodiment of the cargo congestion identification method provided in this application for obtaining initial video information.

[0149] In some embodiments of this application, the steps for obtaining initial video information are specifically described, including the following steps 601 to 604:

[0150] 601. Receive a cargo congestion identification instruction and obtain the target monitoring range corresponding to the cargo congestion identification instruction.

[0151] The terminal receives a cargo congestion identification command. The triggering method of the cargo congestion identification command is not specifically limited. That is, the cargo congestion identification command can be triggered manually by the user. For example, the user can type "cargo congestion identification" on the terminal's display interface and click the confirmation button to trigger the cargo congestion identification command. In addition, the cargo congestion identification command can also be triggered automatically by the terminal. For example, the terminal is pre-set to automatically trigger the cargo congestion identification command from 8:00 to 22:00 every day.

[0152] After receiving the cargo congestion identification instruction, the terminal obtains the target monitoring range corresponding to the cargo congestion identification instruction. The target monitoring range can be flexibly set according to the specific scenario. For example, the target monitoring range can be set as a circular range with the unloading position and the loading position as the diameter.

[0153] 602. Obtain the shooting range of the preset fixed camera and determine whether the shooting range of the preset fixed camera covers the target monitoring range.

[0154] The terminal obtains the shooting range of preset fixed cameras (preset fixed cameras refer to cameras that are pre-set to monitor a specific area, and the preset fixed cameras are communicatively connected to the terminal). The terminal combines the monitoring ranges of all preset fixed cameras to determine whether the shooting range of the preset fixed cameras covers the target monitoring range. If the preset fixed cameras can capture all areas within the target monitoring range, the terminal determines that the shooting range of the preset fixed cameras covers the target monitoring range; otherwise, if the preset fixed cameras cannot capture all areas within the target monitoring range, the terminal determines that the shooting range of the preset fixed cameras does not cover the target monitoring range.

[0155] 603. If the shooting range of the preset fixed camera does not cover the target monitoring range, a call command is sent to the preset aerial drone to call the aerial drone to shoot the uncovered area within the target monitoring range, and to obtain the initial video information of the cargo conveyor line sent by the preset fixed camera and the aerial drone.

[0156] If the preset fixed camera's shooting range does not cover the target monitoring range, the terminal first identifies the uncovered area, and then calls upon a drone to film the uncovered area within the target monitoring range; this is to avoid missing any instances of cargo congestion during identification. The terminal uses both the preset fixed camera and the drone as shooting devices, acquiring initial video information of the cargo conveyor line transmitted by both devices.

[0157] In this embodiment, after receiving the cargo congestion identification instruction, the terminal first determines the target monitoring range that needs to be monitored. After the target monitoring range is determined, the terminal selects a preset fixed camera and a drone as shooting devices according to the target monitoring range. The shooting devices capture images of all cargo conveyor areas within the target monitoring range, which ensures comprehensive monitoring and avoids omissions.

[0158] To better implement the cargo congestion identification method in the embodiments of this application, a cargo congestion identification device is also provided in the embodiments of this application, such as... Figure 7 As shown, Figure 7 This is a schematic diagram of an embodiment of a cargo congestion identification device, which includes the following modules 701-703:

[0159] The video acquisition module 701 is used to acquire the initial video information of the cargo conveyor line sent by the shooting device;

[0160] The segmentation and extraction module 702 is used to input the initial video information into a preset image segmentation model, so as to extract effective video information from the initial video information through the image segmentation model;

[0161] The input recognition module 703 is used to input the valid video information into a preset video classification model, so as to identify the valid video information through the video classification model and obtain the cargo congestion recognition result.

[0162] In some embodiments of this application, the cargo congestion identification device includes:

[0163] The sample acquisition module is used to acquire image samples of the marked cargo conveyor line area and summarize the image samples to form an image sample set;

[0164] The model building module is used to extract a preset proportion of image samples from the image sample set at one time, so as to build an initial image segmentation model through the image samples;

[0165] The model training module is used to iteratively extract a preset proportion of image samples from the image sample set to train the initial image segmentation model through the image samples and obtain an image segmentation training model.

[0166] The model determination module is used to obtain the segmentation accuracy of the image segmentation training model and to use the image segmentation training model with a segmentation accuracy higher than the preset segmentation accuracy as the preset image segmentation model.

[0167] In some embodiments of this application, the segmentation and extraction module is specifically used for:

[0168] The initial video information is input into a preset image segmentation model to segment the initial video information into frames, thereby obtaining continuous initial video frames;

[0169] The initial video frames are segmented one by one using the preset image segmentation model to determine the cargo conveyor area of ​​each initial video frame;

[0170] The region other than the cargo conveyor area in the initial video frame is removed to obtain valid video information.

[0171] In some embodiments of this application, the input recognition module is specifically used for:

[0172] The valid video information is input into a preset video classification model to determine whether the conveyor belt in the cargo conveyor area corresponding to the valid video information is in a stationary state.

[0173] If the conveyor belt in the cargo conveyor area is stationary, the preset video classification model is used to identify whether there is cargo on the stationary conveyor belt.

[0174] If there are no goods on the conveyor belt in the stationary state, the result of no congestion is output.

[0175] If there are goods on the conveyor belt in the stationary state, the congestion identification result is output.

[0176] In some embodiments of this application, the input recognition module is specifically used for:

[0177] If the conveyor belt in the cargo conveyor area is in operation, the preset video classification model is used to determine whether there are relatively moving goods on the conveyor belt in operation.

[0178] If there are goods in relative motion on the conveyor belt during operation, a conveyor belt slippage warning will be output.

[0179] In some embodiments of this application, the input recognition module is specifically used for:

[0180] If there are goods on the conveyor belt in the stationary state, the preset video classification model is used to determine whether the number of goods on the conveyor belt in the stationary state has increased.

[0181] If the number of goods on the conveyor belt in the stationary state increases, the congestion identification result is output.

[0182] If the number of goods on the conveyor belt in the stationary state does not increase, then it is determined whether there are goods within the preset range of the conveyor belt in the stationary state.

[0183] If there are no goods within the preset range of the conveyor belt in the stationary state, the result of goods not being congested is output.

[0184] If there are goods within the preset range of the conveyor belt in the stationary state, the goods congestion identification result will be output.

[0185] In some embodiments of this application, the cargo congestion identification device further includes:

[0186] The instruction receiving module is used to receive cargo congestion identification instructions and obtain the target monitoring range corresponding to the cargo congestion identification instructions;

[0187] The range determination module is used to obtain the shooting range of a preset fixed camera and determine whether the shooting range of the preset fixed camera covers the target monitoring range;

[0188] The video acquisition module 701 includes:

[0189] If the shooting range of the preset fixed camera does not cover the target monitoring range, a call command is sent to the preset aerial drone to call the aerial drone to shoot the uncovered area within the target monitoring range, thereby obtaining the initial video information of the cargo conveyor line sent by the preset fixed camera and the aerial drone.

[0190] In this embodiment of the cargo congestion identification device, the initial video information is segmented using an image segmentation model to obtain valid video information. The valid video information is then classified and identified using a video classification model to obtain the cargo congestion identification result. This eliminates the need to analyze the captured noisy video information; instead, the valid video information corresponding to the cargo conveyor line is directly extracted from the initial video information. The valid video information is then identified to obtain the cargo congestion identification result. The technical solution of this application involves a small amount of video information processing, which improves the efficiency and accuracy of cargo congestion identification.

[0191] This application also provides an electronic device, such as... Figure 8 As shown, Figure 8 This is a schematic diagram of an embodiment of the electronic device provided in this application. The electronic device integrates any of the cargo congestion identification devices provided in this application, and the electronic device includes:

[0192] Pre-set shooting device;

[0193] Accelerometer;

[0194] One or more processors;

[0195] Memory; and

[0196] One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the processor in the steps of the cargo congestion identification method described in any of the embodiments of the above-described cargo congestion identification method.

[0197] Specifically, an electronic device may include components such as a processor 801 with one or more processing cores, a memory 802 with one or more computer-readable storage media, a power supply 803, and an input unit 804. Those skilled in the art will understand that... Figure 8 The electronic device structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein:

[0198] The processor 801 is the control center of the electronic device. It connects various parts of the electronic device via various interfaces and lines. By running or executing software programs and / or modules stored in the memory 802, and by calling data stored in the memory 802, it performs various functions and processes data, thereby providing overall monitoring of the electronic device. Optionally, the processor 801 may include one or more processing cores; preferably, the processor 801 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 801.

[0199] The memory 802 can be used to store software programs and modules. The processor 801 executes various functional applications and data processing by running the software programs and modules stored in the memory 802. The memory 802 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 802 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 802 may also include a memory controller to provide the processor 801 with access to the memory 802.

[0200] The electronic device also includes a power supply 803 that supplies power to the various components. Preferably, the power supply 803 can be logically connected to the processor 801 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 803 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0201] The electronic device may also include an input unit 804, which can be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0202] Although not shown, the electronic device may also include a display unit, etc., which will not be described in detail here. Specifically, in the embodiments of this application, the processor 801 in the electronic device loads the executable files corresponding to the processes of one or more applications into the memory 802 according to the following instructions, and the processor 801 runs the applications stored in the memory 802 to realize various functions, as follows:

[0203] Acquire initial video information of the cargo conveyor line sent by the camera device;

[0204] The initial video information is input into a preset image segmentation model to extract valid video information from the initial video information through the image segmentation model;

[0205] The valid video information is input into a preset video classification model to identify the valid video information and obtain the cargo congestion identification result.

[0206] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0207] Therefore, embodiments of this application provide a computer-readable storage medium, which may include: read-only memory (ROM), random access memory (RAM), a magnetic disk, or an optical disk, etc. A computer program is stored thereon, which is loaded by a processor to execute the steps in any of the cargo congestion identification methods provided in embodiments of this application. For example, the computer program loaded by the processor can execute the following steps:

[0208] Acquire initial video information of the cargo conveyor line sent by the camera device;

[0209] The initial video information is input into a preset image segmentation model to extract valid video information from the initial video information through the image segmentation model;

[0210] The valid video information is input into a preset video classification model to identify the valid video information and obtain the cargo congestion identification result.

[0211] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the detailed descriptions of other embodiments above, which will not be repeated here.

[0212] In practice, each of the above units or structures can be implemented as an independent entity or can be arbitrarily combined to be implemented as the same or several entities. For the specific implementation of each of the above units or structures, please refer to the previous method embodiments, which will not be repeated here.

[0213] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0214] The above provides a detailed description of a cargo congestion identification method provided by the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for identifying cargo congestion, characterized in that, The cargo congestion identification method includes: Acquire initial video information of the cargo conveyor line sent by the camera device; The initial video information is input into a preset image segmentation model to extract valid video information from the initial video information through the image segmentation model; The valid video information is input into a preset video classification model to determine whether the conveyor belt in the cargo conveyor area corresponding to the valid video information is in a stationary state. If the conveyor belt in the cargo conveyor area is stationary, the preset video classification model is used to identify whether there is cargo on the stationary conveyor belt. If there are no goods on the conveyor belt in the stationary state, the result of no congestion is output. If there are goods on the conveyor belt in the stationary state, the goods congestion identification result is output. If the conveyor belt in the cargo conveyor area is in operation, the preset video classification model is used to determine whether there are relatively moving goods on the conveyor belt in operation. If there are goods in relative motion on the conveyor belt during operation, a conveyor belt slippage warning will be output.

2. The cargo congestion identification method according to claim 1, characterized in that, Before inputting the initial video information into a preset image segmentation model to extract valid video information from the initial video information through the image segmentation model, the method further includes: Obtain image samples of the marked cargo conveyor line area, and summarize the image samples to form an image sample set; A preset proportion of image samples are extracted from the image sample set at one time to construct an initial image segmentation model using the image samples; Iteratively extract a preset proportion of image samples from the image sample set to train the initial image segmentation model using the image samples, thereby obtaining an image segmentation training model; Obtain the segmentation accuracy of the image segmentation training model, and use the image segmentation training model with a segmentation accuracy higher than the preset segmentation accuracy as the preset image segmentation model.

3. The cargo congestion identification method according to claim 1, characterized in that, The step of inputting the initial video information into a preset image segmentation model to extract valid video information from the initial video information through the image segmentation model includes: The initial video information is input into a preset image segmentation model to segment the initial video information into frames, thereby obtaining continuous initial video frames; The initial video frames are segmented one by one using the preset image segmentation model to determine the cargo conveyor area of ​​each initial video frame; The region other than the cargo conveyor area in the initial video frame is removed to obtain valid video information.

4. The cargo congestion identification method according to claim 1, characterized in that, If there are goods on the conveyor belt in the stationary state, the output of the goods congestion identification result includes: If there are goods on the conveyor belt in the stationary state, the preset video classification model is used to determine whether the number of goods on the conveyor belt in the stationary state has increased. If the number of goods on the conveyor belt in the stationary state increases, the congestion identification result is output. If the number of goods on the conveyor belt in the stationary state does not increase, then it is determined whether there are goods within the preset range of the conveyor belt in the stationary state. If there are no goods within the preset range of the conveyor belt in the stationary state, the result of goods not being congested is output. If there are goods within the preset range of the conveyor belt in the stationary state, the goods congestion identification result will be output.

5. The cargo congestion identification method according to any one of claims 1 to 4, characterized in that, Before acquiring the initial video information of the cargo conveyor line sent by the shooting device, the method further includes: Receive a cargo congestion identification command and obtain the target monitoring range corresponding to the cargo congestion identification command; Obtain the shooting range of a preset fixed camera, and determine whether the shooting range of the preset fixed camera covers the target monitoring range; The acquisition of initial video information of the cargo conveyor line sent by the shooting device includes: If the shooting range of the preset fixed camera does not cover the target monitoring range, a call command is sent to the preset aerial drone to call the aerial drone to shoot the uncovered area within the target monitoring range, thereby obtaining the initial video information of the cargo conveyor line sent by the preset fixed camera and the aerial drone.

6. A cargo congestion identification device, characterized in that, The cargo congestion identification device includes: The video acquisition module is used to acquire the initial video information of the cargo conveyor line sent by the shooting device. The segmentation and extraction module is used to input the initial video information into a preset image segmentation model, so as to extract effective video information from the initial video information through the image segmentation model; The input recognition module is used to input the valid video information into a preset video classification model to determine whether the conveyor belt in the corresponding goods conveyor area is stationary. If the conveyor belt in the goods conveyor area is stationary, the preset video classification model is used to identify whether there is cargo on the stationary conveyor belt. If there is no cargo on the stationary conveyor belt, a cargo non-congestion recognition result is output. If there is cargo on the stationary conveyor belt, a cargo congestion recognition result is output. If the conveyor belt in the goods conveyor area is in operation, the preset video classification model is used to determine whether there is relatively moving cargo on the operating conveyor belt. If there is relatively moving cargo on the operating conveyor belt, a conveyor belt slippage warning is output.

7. An electronic device, characterized in that, The electronic device includes: One or more processors; Memory; and One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the processor to implement the cargo congestion identification method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, It stores a computer program, which is loaded by a processor to perform the steps in the cargo congestion identification method according to any one of claims 1 to 5.

Images