Intelligent port cargo management method and system based on deep learning

Abstract

The invention provides an intelligent port cargo management method and system based on deep learning, and relates to the technical field of new-generation information. Comprising the steps of obtaining historical and real-time video data of an intelligent port loading area, and performing frame extraction processing to obtain a to-be-processed image and a real-time target image; performing target annotation on the to-be-processed image according to the annotation target list, and establishing a target annotation data set to train a detection and recognition model; the real-time target image is detected and recognized, the operation stage of the current moment is judged according to the position relation of the target recognition frames at the current moment and the previous moment, and when the judged operation stage is a shipment completion stage, the number of the recognized steel bar bundles is accumulated to the total shipment number; otherwise, continuously detecting the position relation according to the time sequence until the ship loading completion stage is judged. According to the method, knowledge migration and an improved YOLOv8 network training detection and recognition model are fused, and the target recognition speed and accuracy are remarkably improved when the method is applied to an intelligent port cargo management scene.

Description

Technical Field

[0001] The present invention relates to the field of new generation information technology, and in particular to a smart port cargo management method and system based on deep learning. Background Art

[0002] Based on the continuous growth of port cargo throughput, the development trend of port intelligence and the competitive landscape of global ports, the construction demand of smart ports needs to be addressed urgently. With the continuous development of global trade, the construction of smart ports can realize the digitization and intelligence of ports, improve the competitiveness of ports, and better adapt to market changes and demands. The operation mode of traditional ports has been difficult to meet the needs. The construction of smart ports can improve port operation efficiency and reduce logistics costs; at the same time, the construction of smart ports can realize the digitization and intelligence of ports, improve the competitiveness of ports, and better adapt to market changes and demands; in addition, the construction of smart ports can promote the green and sustainable development of ports, reduce the impact on the environment, and achieve a dual improvement in economic and social benefits.

[0003] The primary application of smart ports in realizing port digitization and intelligence is in port access management and cargo management. For example, in the traditional port operation mode, the main management task of the finished steel bar loading process in the loading area is to count the number of steel bar bundles loaded on the loading ship. The number of steel bar bundles is estimated by calculating the weight of a single steel bar bundle and the total weight of the loaded steel bars. If you want to know the exact number of steel bar bundles, you can only use manual statistics. Manual statistics not only requires a lot of manpower and financial costs, but also has safety hazards and omissions due to the harsh working environment. Smart ports can use computer vision systems to intelligently identify and count the number of steel bar bundles loaded by the overhead crane during the steel bar loading process, and then send it to the business system to count the total number of steel bar bundles loaded. Compared with traditional ports, cargo management has the characteristics of high accuracy, all-weather uninterrupted operation, reduced labor costs, reduced safety accidents, and improved production efficiency.

[0004] Computer vision systems based on deep learning can show excellent performance in preset experimental environments, especially in tasks such as target recognition, and their accuracy and efficiency are widely praised. However, when these algorithms are applied to actual industrial environments, they often encounter many challenges. For example, factors such as fluctuations in lighting conditions, irregularities in target placement, and density differences will have a significant impact on the recognition effect. Specifically, in industrial sites, lighting conditions may change due to changes in weather, time, or obstructions, which will have a negative impact on image quality and reduce the accuracy of the recognition algorithm. In addition, the placement of the recognition targets is usually not regular enough, and sometimes even overlaps or occlusions occur, which undoubtedly increases the difficulty of recognition. At the same time, due to different production requirements or storage conditions, the density of the recognition targets may also change, further increasing the complexity of recognition. Therefore, target recognition and counting technologies still need to be continuously improved and enhanced to cope with more complex and changeable industrial application scenarios.

[0005] Therefore, when applying deep learning-based computer vision systems in smart port cargo management scenarios, new solutions need to be proposed to improve the accuracy and stability of smart port cargo management and provide more reliable technical support for industrial applications. Summary of the invention

[0006] The purpose of the present invention is to provide a smart port cargo management method and system based on deep learning, which can achieve more stable and accurate target recognition results by deepening the architecture of the target detection and recognition model and the extraction of target features, combined with the position changes of the target recognition results at the current moment and the previous moment, and provide more reliable technical support for the application of computer vision systems in smart ports.

[0007] To achieve the above object, the present invention proposes the following technical solutions: First, a smart port cargo management method based on deep learning is proposed, including: Acquire historical video data of the loading area of ​​the smart port, and perform frame extraction processing on the historical video data to obtain an image to be processed; wherein the historical video data is a historical operation process of loading steel bar bundles onto a ship using a loading machine; For each of the images to be processed, target annotation is performed in sequence according to the annotation target list to obtain a number of pre-processed images; wherein the annotation target list includes loading machinery, steel bar bundles, and ship hulls, and the annotation information of the target annotation includes object position, name, width, height, number of image channels, and label category; According to the preprocessed image, a target annotation data set is established for any target in the annotated target list, and a detection and recognition model corresponding to each target is established and trained according to the target annotation data set; wherein the detection and recognition model is a target detection model based on deep learning, including a first detection and recognition model for identifying a ship loading machinery, a second detection and recognition model for identifying a steel bar bundle, and a third detection and recognition model for identifying a ship hull; Acquire real-time video data of the loading area of ​​the smart port, and perform frame extraction processing on the real-time video data to obtain a real-time target image; wherein the real-time video data is the real-time operation process of loading steel bar bundles onto a ship using a loading machine; The detection and recognition models corresponding to each target are used to detect and recognize the real-time target images respectively, and the operation stage of the loading machinery for loading steel bar bundles is judged at the current moment according to the position relationship between the target recognition frames at the current moment and the previous moment; wherein the operation stage includes: the loading machinery no-load stage, the loading machinery picking-up stage, the loading machinery delivery stage, the loading machinery loading stage, and the loading completion stage; When the operation stage is judged to be the loading completion stage, the count of the steel bar bundles detected and identified in the loading completion stage is added to the total number of loadings, completing the cargo counting management of one loading cycle of the loading machinery; otherwise, the positional relationship of each target recognition frame at the current moment and the previous moment is continuously detected in chronological order until the operation stage is judged to be the loading completion stage.

[0008] Furthermore, the process of establishing and training the detection and recognition models corresponding to each target based on the target annotation data set is as follows: Based on the YOLOv8 network, a network structure of the detection and recognition model is constructed, including a backbone network, a neck network and a head module; wherein the backbone network includes several pixel adaptive extraction modules based on the attention mechanism, and the pixel adaptive extraction modules are used to extract different weight features of the input image and transmit them to the neck network respectively; the neck network includes a weight alignment module and a dense residual block, and the weight alignment module and the dense residual block are used to effectively align the target features in feature images of different scales; the head module is used to output feature images of three different sizes for final recognition; For a target annotation data set of any target in the annotated target list, divide it into a training set, a validation set and a test set; Configure the loss function of the detection and recognition model, use the Wise-IOUv1 loss function with dual attention characteristics, and based on the Wise-IOUv1 loss function, outlier β , monotone focusing coefficient of L-negative loss function of Wise-IOUv1 , construct Wise-IOUv3 loss function; Model training, using a knowledge transfer training method to train and optimize the detection and recognition model according to the training set and the validation set, and performing model testing in the test set to obtain a detection and recognition model for the corresponding target; The coordinates of a set point on the recognition prediction box of the detection recognition model are defined as ( x , y ), and its corresponding coordinates on the real frame are ( x t , y t ), then the calculation formula of Wise-IOUv1 loss function and Wise-IOUv3 loss function is expressed as: ; ; ; ; ; in, w c , h c Indicates the width and height of the minimum bounding rectangle of the prediction box and the real box. b , b t Respectively represent the center points of the predicted box and the real box, represents the scale factor, represents the smoothing parameter and α represents the focusing coefficient.

[0009] Furthermore, the pixel adaptive extraction module includes a 1×1 convolution unit, a horizontal weight extraction module, a vertical weight extraction module, a trunk weight extraction module and a fully connected unit; The 1×1 convolution unit is used to increase the input feature dimension. The high-dimensional features output by the convolution unit are equally divided into three parts through a segmentation operation and are respectively input into a horizontal weight extraction module, a vertical weight extraction module and a trunk weight extraction module. After obtaining the horizontal weight and the vertical weight output by the horizontal weight extraction module and the vertical weight extraction module, the sigmoid activation function is used to activate the horizontal weight and the vertical weight by element multiplication. The two activated weights are fused with the trunk weight output by the trunk weight extraction module in the fully connected unit to achieve weight fine-tuning. The expression of the Sigmoid activation formula is: ; The formula for the fully connected unit to output three types of weight information after weight splicing is: ; in, represents the output of the pixel adaptive extraction module, W C represents the horizontal weight, W V represents the vertical weight, W T represents the backbone weight, Concat For full connection operation, f Represents the input features.

[0010] Furthermore, the weight alignment module and the dense residual block in the neck network are used to effectively align target features in feature images of different scales in the following process: The input horizontal weight or vertical weight is subjected to Gaussian smoothing to remove the interference noise in the feature map and smooth the distribution area of ​​the weight; wherein the calculation formula of the Gaussian smoothing is: ; in, represents the point coordinates of the input image, Indicates the standard deviation of the mean value in the neighborhood of the corresponding subscript; Based on the structural measurement index, a wide-area weighted similarity ratio is set, and based on the wide-area weighted similarity ratio, a neighborhood with a weight close to the trunk is screened; wherein the calculation formula of the wide-area weighted similarity ratio is: ; in, Indicates the wide-area weight similarity ratio between the horizontal weight or vertical weight and the trunk weight, H and W Represent the height and width of the feature map respectively, represents the average value of the corresponding neighborhood, N (3 i , 3) indicates the The 3×3 pixel neighborhood of the recognition prediction box on the feature map, N (3 j , 3) indicates the A 3×3 neighborhood of the recognition prediction box, represents the standard deviation of the mean value in the neighborhood of the corresponding subscript, C 1. C 2 is a constant to avoid the denominator being zero; Based on the neighborhood obtained by screening, outputting an alignment weight space of the horizontal weight or the vertical weight and the trunk weight; A dense residual block structure is used to perform network feature fusion on the input alignment weight space so that the head module outputs feature images of different sizes; wherein the formula of the dense residual block structure is: ; ; in, Indicates dense connection. F Residual represents the desired output of the network, H (·) represents the internal nonlinear transformation module Conv-ReLU operation, x i Represents the input features of the corresponding layer.

[0011] Furthermore, the process of judging the operation stage of the loading machinery for loading steel bar bundles on the ship at the current moment according to the position relationship of each target recognition frame at the current moment and the previous moment is: When the first target recognition frame in the real-time target image at the current moment and the real-time target image at the previous moment does not contain the second target recognition frame, and the first target recognition frame gradually moves away from the third target recognition frame, it is determined that the operation stage of the ship loading machinery loading steel bar bundles at the current moment is the no-load stage of the ship loading machinery; wherein the first target recognition frame is the target recognition frame of the ship loading machinery, the second target recognition frame is the target recognition frame of the steel bar bundle, and the third target recognition frame is the target recognition frame of the hull; When the first target recognition frame in the real-time target image at the current moment and the real-time target image at the previous moment both contain the second target recognition frame, and the number of the second target recognition frames in the first target recognition frame gradually increases, it is determined that the operation stage of the loading machinery for loading steel bar bundles is the loading machinery picking-up stage at the current moment; When the first target recognition frame in the real-time target image at the current moment and the real-time target image at the previous moment both contain an equal number of second target recognition frames, and the first target recognition frame gradually approaches the third target recognition frame, it is determined that the operation stage of the loading machinery for loading steel bar bundles is the loading machinery delivery stage at the current moment; When the first target recognition frame and the third target recognition frame in the real-time target image at the current moment and the real-time target image at the previous moment are partially overlapped, and the number of the second target recognition frames in the first target recognition frame gradually decreases, it is determined that the operation stage of the loading machinery for loading steel bar bundles at the current moment is the loading machinery loading stage; When the overlap between the first target identification frame and the third target identification frame in the real-time target image at the current moment and the real-time target image at the previous moment reaches a set ratio, and the number of second target identification frames in the first target identification frame reaches a set value, it is determined that the operation stage of the loading machinery for loading steel bar bundles at the current moment is the loading completion stage.

[0012] Furthermore, it also includes: The real-time target images corresponding to the loading stage and the loading completion stage of the loading machinery, as well as the real-time video data of the time period corresponding to the real-time target images are intercepted and saved to the cargo management system of the smart port for data traceability.

[0013] Secondly, a smart port cargo management system based on deep learning is proposed, including: The first acquisition module is used to acquire historical video data of the loading area of ​​the smart port, and perform frame extraction processing on the historical video data to obtain an image to be processed; wherein the historical video data is a historical operation process of loading steel bar bundles onto a ship using a loading machine; The target annotation module is used to annotate each of the to-be-processed images in sequence according to the annotated target list to obtain a plurality of pre-processed images; wherein the annotated target list includes ship loading machinery, steel bar bundles, and ship hulls, and the annotation information of the target annotation includes object position, name, width, height, number of image channels, and label category; A model building and training module, used to establish a target annotation data set for any target in the annotated target list according to the preprocessed image, and to establish and train a detection and recognition model corresponding to each target according to the target annotation data set; wherein the detection and recognition model is a target detection model based on deep learning, including a first detection and recognition model for identifying a ship loading machinery, a second detection and recognition model for identifying a steel bar bundle, and a third detection and recognition model for identifying a ship hull; The second acquisition module is used to acquire real-time video data of the loading area of ​​the smart port, and perform frame extraction processing on the real-time video data to obtain a real-time target image; wherein the real-time video data is a real-time operation process of loading steel bar bundles onto a ship using a loading machine; A detection and judgment module is used to detect and identify the real-time target images respectively using the detection and recognition models corresponding to each target, and judge the operation stage of the loading machinery for loading steel bar bundles at the current moment according to the position relationship between the target recognition frames at the current moment and the previous moment; wherein the operation stage includes: the loading machinery empty stage, the loading machinery picking up stage, the loading machinery delivery stage, the loading machinery loading stage, and the loading completion stage; The counting management module is used to add the count of the steel bar bundles detected and identified in the loading completion stage to the total number of loadings when the operation stage is judged to be the loading completion stage, so as to complete the cargo counting management of a loading cycle of the loading machinery; otherwise, the position relationship of each target recognition frame at the current moment and the previous moment is continuously detected in chronological order until the operation stage is judged to be the loading completion stage.

[0014] Furthermore, the model building and training module builds and trains the execution units of the detection and recognition models corresponding to each target respectively according to the target annotation data set, including: A building unit is used to build a network structure of a detection and recognition model based on a YOLOv8 network, including a backbone network, a neck network and a head module; wherein the backbone network includes a plurality of pixel adaptive extraction modules based on an attention mechanism, the pixel adaptive extraction modules are used to extract different weight features of an input image, and transmit them to the neck network respectively; the neck network includes a weight alignment module and a dense residual block, the weight alignment module and the dense residual block are used to effectively align target features in feature images of different scales; the head module is used to output feature images of three different sizes for final recognition; A division unit, used for dividing a target annotation data set of any target in the annotated target list into a training set, a validation set and a test set; The configuration unit is used to configure the loss function of the detection and recognition model. The Wise-IOUv1 loss function with dual attention characteristics is adopted, and based on the Wise-IOUv1 loss function, outlier degree β , monotone focusing coefficient of L-negative loss function of Wise-IOUv1 , construct Wise-IOUv3 loss function; A training unit, used for model training, adopts a knowledge transfer training method to train and optimize the detection and recognition model according to the training set and the validation set, and performs model testing in the test set to obtain a detection and recognition model corresponding to the target; The coordinates of a set point on the recognition prediction box of the detection recognition model are defined as ( x , y ), and its corresponding coordinates on the real frame are ( x t , y t ), then the calculation formula of Wise-IOUv1 loss function and Wise-IOUv3 loss function is expressed as: ; ; ; ; ; in, w c , h c Indicates the width and height of the minimum bounding rectangle of the prediction box and the real box. b ,b t Respectively represent the center points of the predicted box and the real box, represents the scale factor, represents the smoothing parameter and α represents the focusing coefficient.

[0015] In the third aspect, an electronic device is proposed, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the electronic device implements the smart port cargo management method based on deep learning as described above.

[0016] In a fourth aspect, a computer-readable storage medium is proposed, which is used to store a computer program. When the computer program runs on a computer, the computer executes the smart port cargo management method based on deep learning as described above.

[0017] It can be seen from the above technical solutions that the technical solutions of the present invention have the following beneficial effects: The present invention discloses a smart port cargo management method and system based on deep learning, and the method includes: obtaining historical and real-time video data of the smart port loading area, performing frame extraction processing on the historical and real-time video data to obtain a to-be-processed image and a real-time target image; wherein the historical and real-time video data are the historical and real-time operation process of using a loading machine to load steel bar bundles; performing target annotation on the to-be-processed image according to a marked target list to obtain a pre-processed image, and after establishing a target annotation data set for each target, establishing and training a detection and recognition model corresponding to each target, wherein the detection and recognition model is a target detection model based on deep learning; using the detection and recognition model corresponding to each target to detect and recognize the real-time target image respectively, and judging the operation stage of the loading machine for loading steel bar bundles at the current moment according to the positional relationship between each target recognition frame at the current moment and the previous moment; and when the operation stage judged is the loading completion stage, the count of the identified steel bar bundles is accumulated to the total number of loading; otherwise, continuously detecting the positional relationship between each target recognition frame at the current moment and the previous moment in chronological order until the operation stage judged is the loading completion stage. On the one hand, the present invention adopts the fusion of knowledge transfer and improved YOLOv8 network training detection and recognition model, which significantly improves the recognition speed and accuracy of targets in the smart port cargo management scenario, thereby significantly improving the accuracy and efficiency of the cargo statistical management process; on the other hand, by comparing and analyzing the positional relationship of the target recognition frame recognized at the current moment and the previous moment, the operation stage can be determined more accurately, thereby more accurately managing the cargo counting of the entire cargo loading operation.

[0018] In addition, the present invention is implemented by relying on a computer vision system, which does not require a large amount of manual participation during the operation of the system, realizes unmanned intelligent management, and can adapt to all-weather operation in different weather conditions; it not only reduces labor cost investment and reduces personnel safety hazards, but can also be invested in the management of different types of goods to realize intelligent data management and data traceability.

[0019] It should be appreciated that all combinations of the foregoing concepts, as well as additional concepts described in greater detail below, may be considered to be part of the inventive subject matter of the present disclosure, provided such concepts are not mutually inconsistent.

[0020] The foregoing and other aspects, embodiments and features of the present invention can be more fully understood from the following description in conjunction with the accompanying drawings. Other additional aspects of the present invention, such as the features and / or beneficial effects of the exemplary embodiments, will be apparent from the following description or learned from the practice of the specific embodiments according to the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings are not intended to be drawn to scale. In the accompanying drawings, each identical or nearly identical component shown in various figures may be represented by the same reference numeral. For clarity, not every component is labeled in each figure. Embodiments of various aspects of the present invention will now be described by way of example and with reference to the accompanying drawings, in which: Figure 1 This is a flow chart of the smart port cargo management method based on deep learning disclosed in the present invention; Figure 2 A flow chart of establishing and training a detection and recognition model corresponding to each target disclosed in the present invention; Figure 3 A schematic diagram of the network structure of the detection and recognition model disclosed in the present invention; Figure 4 It is a schematic diagram of a pixel adaptive extraction module in the detection and recognition model network disclosed in the present invention; Figure 5 A schematic diagram of a weight alignment module in a detection and recognition model network disclosed in the present invention; Figure 6 It is a flow chart of judging the current stage of steel bar bundle loading operation of the loading machinery disclosed by the present invention; Figure 7 This is a schematic diagram of the framework of the smart port cargo management system based on deep learning disclosed in the present invention; Figure 8 It is a schematic diagram of an electronic device disclosed in the present invention. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the embodiment of the present invention clearer, the technical solution of the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings of the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all of the embodiments. Based on the described embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. Unless otherwise defined, the technical terms or scientific terms used herein should be the common meaning understood by people with general skills in the field to which the present invention belongs.

[0023] The words "first", "second" and similar words used in the patent application specification and claims of the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. Similarly, unless the context clearly indicates otherwise, the singular form of "a", "an" or "the" and other similar words do not indicate a quantitative limitation, but indicate the existence of at least one. "Include" or "comprises" and other similar words mean that the elements or objects appearing before "include" or "comprises" include the features, wholes, steps, operations, elements and / or components listed after "include" or "comprises", and do not exclude the existence or addition of one or more other features, wholes, steps, operations, elements, components and / or their collections.

[0024] In the traditional port operation mode, manual statistics are usually used to obtain accurate values ​​when managing the entry and exit of goods. However, manual statistics have the following problems: it requires more manpower costs, there are safety hazards in the loading and unloading process, and there may be omissions or errors in the counting. The introduction of a computer vision system can solve the above problems in a targeted manner, but the effect of the system is greatly affected by the environment when it is used in an actual industrial environment. Therefore, the present invention aims to propose an improved computer vision system based on deep learning that can be used in command ports to improve the accuracy and stability of port cargo management.

[0025] The following is a further detailed introduction to the deep learning-based smart port cargo management method and system disclosed in the present invention in conjunction with the specific embodiments shown in the accompanying drawings.

[0026] Combination Figure 1 As shown, the smart port cargo management method based on deep learning disclosed in the embodiment includes the following steps: Step S102, obtaining historical video data of the loading area of ​​the smart port, and performing frame extraction processing on the historical video data to obtain an image to be processed; wherein the historical video data is a historical operation process of loading steel bar bundles onto a ship using a loading machine; The frequency of the frame extraction process to obtain the image to be processed is determined according to the moving speed of the loading machinery. The obtained image to be processed presents the loading process in an interval display manner. If the working speed of the loading machinery is too fast and the frame extraction interval is long, part of the loading process will be missing. Therefore, the appropriate frame extraction interval is determined according to the working speed of the loading machinery to ensure that the image to be processed can truly and coherently reflect the loading process. In the embodiment, historical video data is obtained by monitoring cameras installed in the smart port. There are multiple monitoring cameras, so the video angles are different. The angle video that can distinguish the target is selected as the historical video data; for example, for the loading of steel bar bundles, the video data corresponding to the radial section angle of the steel bar bundle is selected.

[0027] Step S104, for each of the images to be processed, target annotation is performed in sequence according to the annotation target list to obtain a number of pre-processed images; wherein the annotation target list includes loading machinery, steel bar bundles, and hulls, and the annotation information of the target annotations includes object position, name, width, height, number of image channels, and label category; each target is separately annotated so that the target features can be clearly identified when the target features are subsequently learned.

[0028] Step S106, establishing a target annotation data set for any target in the annotated target list according to the preprocessed image, and establishing and training a detection and recognition model corresponding to each target according to the target annotation data set; wherein the detection and recognition model is a target detection model based on deep learning, including a first detection and recognition model for identifying a ship loading machinery, a second detection and recognition model for identifying a steel bar bundle, and a third detection and recognition model for identifying a ship hull; The present invention deepens the structure of the detection and recognition model, designs a pixel adaptive extraction module, a weight alignment module and a dense residual block, automatically extracts key features in the image and outputs multi-scale feature images. This not only provides a more comprehensive understanding of the image content and improves the recognition accuracy, but also enhances the generalization ability of the network and makes it more stable in industrial applications.

[0029] Specific as Figure 2 As shown, the process of establishing and training the detection and recognition model corresponding to each target according to the target annotation data set includes: Step S202: Based on the YOLOv8 network, a network structure of a detection and recognition model is constructed, including a backbone network, a neck network, and a head module. Figure 3As shown; wherein, the backbone network includes several pixel adaptive extraction modules (PEAMs) based on the attention mechanism, and the pixel adaptive extraction modules are used to extract different weight features of the input image and transmit them to the neck network respectively to reduce the difficulty of subsequent recognition; the neck network includes a weight alignment module (WAM) and a dense residual block (R-DB), and the weight alignment module and the dense residual block are used to effectively align the target features in feature images of different scales, and at the same time reconfirm and strengthen the features of the target to minimize recognition errors; the head module is used to output feature images of three different sizes of 80×80×256, 40×40×512 and 20×20×521 for final recognition; like Figure 3 As shown in the figure, pixel adaptive extraction modules with widths of 5, 3, and 1 are set at different positions of the backbone network, aiming to fully capture the weight information of the input image, ensure the integrity of the pixel features, and strengthen the correlation between pixels in the neighborhood, ultimately achieving the technical effect of effectively improving the recognition efficiency of the model and enabling the model to more accurately identify and parse key information in the image.

[0030] Combination Figure 4 As shown, the pixel adaptive extraction module includes a 1×1 convolution unit, a horizontal weight extraction module, a vertical weight extraction module, a trunk weight extraction module and a fully connected unit; when applied, the 1×1 convolution unit is used to improve the input feature dimension, and the high-dimensional features output by the convolution unit are equally divided into three parts through a segmentation operation and respectively input into the horizontal weight extraction module, the vertical weight extraction module and the trunk weight extraction module, and after obtaining the horizontal weight and the vertical weight output by the horizontal weight extraction module and the vertical weight extraction module, the Sigmoid activation function is used to activate them by element multiplication, and the two activated weights are fused with the trunk weight output by the trunk weight extraction module in the fully connected unit to achieve weight fine-tuning; The expression of the Sigmoid activation formula is: ; The formula for the fully connected unit to output three types of weight information after weight splicing is: ; in, represents the output of the pixel adaptive extraction module, W C represents the horizontal weight, W V represents the vertical weight, W T represents the backbone weight, Concat For full connection operation, f Represents the input features.

[0031] The weight alignment module and dense residual block are designed in the neck network from the perspective of extracted weights to solve the pixel offset and network difficulty in fitting that may be caused by the network segmenting and resizing the feature image in the actual process, and to ensure the stability and accuracy of the target features in the complex processing process; The weight alignment module is as follows Figure 5 As shown in the figure, the core of this module is that it fuses the inputs of two parallel branches, namely the horizontal weight or the vertical weight, and the trunk weight; when processing, the horizontal weight or the vertical weight input by one branch is first Gaussian smoothed to remove the interference noise in the feature map and the distribution area of ​​the smoothed weight, thereby indirectly improving the accuracy and stability of the alignment processing; then, based on the structural measurement index, the wide-area weight similarity ratio is set, and based on the wide-area weight similarity ratio, the neighborhood close to the trunk weight is screened to ensure that the stability of the feature alignment is further improved; then, based on the screened neighborhood, the alignment weight space of the horizontal weight or the vertical weight and the trunk weight is output; The calculation formula of the Gaussian smoothing process is as follows: ; in, represents the point coordinates of the input image, Indicates the standard deviation of the mean value in the neighborhood of the corresponding subscript; And, the calculation formula of the wide area weighted similarity ratio is: ; in, Indicates the wide-area weight similarity ratio between the horizontal weight or vertical weight and the trunk weight, H and W Represent the height and width of the feature map respectively, represents the average value of the corresponding neighborhood, N (3 i , 3) indicates the The 3×3 pixel neighborhood of the recognition prediction box on the feature map, N (3 j , 3) indicates the A 3×3 neighborhood of the recognition prediction box, represents the standard deviation of the mean value in the neighborhood of the corresponding subscript, C 1. C 2 is a constant to avoid the denominator being zero.

[0032] In order to solve the problems of the surge in parameters, reduced feature processing efficiency and high learning complexity in the above-mentioned weight alignment process, a dense residual block is further used to reduce parameters and suppress the risk of network overfitting; that is, a dense residual block structure is used to fuse network features of the input alignment weight space so that the head module outputs feature images of different sizes; wherein the formula of the dense residual block structure is: ; ; in, Indicates dense connection. F Residual represents the desired output of the network, H (·) represents the internal nonlinear transformation module Conv-ReLU operation, x i Represents the input features of the corresponding layer number, Represents the total number of input features.

[0033] Step S204, dividing the target annotation data set of any target in the annotated target list into a training set, a validation set and a test set, for example, in a ratio of 7:2:1.

[0034] Step S206, configuring the loss function of the detection and recognition model, using the Wise-IOUv1 loss function with dual attention characteristics, and based on the Wise-IOUv1 loss function, outlier degree β , monotone focusing coefficient of L-negative loss function of Wise-IOUv1 , construct the Wise-IOUv3 loss function to avoid harmful gradients caused by low quality datasets; The coordinates of a set point on the recognition prediction box of the detection recognition model are defined as ( x , y ), and its corresponding coordinates on the real frame are ( x t , y t ), then the calculation formula of Wise-IOUv1 loss function and Wise-IOUv3 loss function is expressed as: ; ; ; ; ; in, w c , hc Indicates the width and height of the minimum bounding rectangle of the prediction box and the real box. b , b t Respectively represent the center points of the predicted box and the real box, represents the scale factor, represents the smoothing parameter and α represents the focusing coefficient.

[0035] When the detection and recognition models of each target trained according to the above steps are applied, it mainly includes three core stages in sequence: target feature extraction stage, target feature alignment stage and anchor frame positioning stage; the target feature extraction stage uses the powerful feature learning ability of the deep learning network to accurately extract the target's shape, texture, color and other key features from the input image, so as to improve the accuracy and stability of subsequent recognition; entering the target feature alignment stage, in this stage, the algorithm will perform detailed alignment processing on the extracted features to ensure the consistency of these features in space and scale, and provide an accurate basis for subsequent classification and positioning; finally, entering the anchor frame positioning stage, the algorithm will accurately mark the position of the corresponding target in the real-time target image according to the previously extracted and aligned features, thus completing the entire recognition process.

[0036] Step S208, model training, adopting the knowledge transfer training method to train and optimize the detection and recognition model according to the training set and the verification set, and perform model testing in the test set to obtain the detection and recognition model of the corresponding target; when training this scheme, the target annotation data set corresponding to the above-mentioned target is used as the optimization data set for standby, first the above-mentioned network structure is trained in the existing nut detection data set using the back propagation algorithm to obtain a pre-trained model, and then further knowledge transfer operations are performed on the training set of the target annotation data set, and finally the verification set is used to further optimize and fine-tune the obtained training model, and finally ensure that the trained model has stable target recognition capabilities; the first, second and third detection and recognition models in this scheme are all trained using the above-mentioned model training method.

[0037] Step S108, acquiring real-time video data of the loading area of ​​the smart port, and performing frame extraction processing on the real-time video data to obtain a real-time target image; wherein the real-time video data is a real-time operation process of loading steel bar bundles onto a ship using a loading machine; When obtaining a real-time target image through frame extraction processing, a smaller frame extraction interval can be used than when obtaining the image to be processed, so as to ensure more accurate judgment in the subsequent operation stage.

[0038] Step S110, using the detection and recognition model corresponding to each target to respectively detect and recognize the real-time target image, and judging the operation stage of the loading machinery for loading steel bar bundles at the current moment according to the position relationship between the target recognition frames at the current moment and the previous moment; wherein the operation stage includes: the loading machinery no-load stage, the loading machinery picking-up stage, the loading machinery delivery stage, the loading machinery loading stage, and the loading completion stage; Step S112, when the operation stage is judged to be the loading completion stage, the count of the steel bar bundles detected and identified in the loading completion stage is accumulated to the total loading number, completing the cargo counting management of one loading cycle of the loading machinery; otherwise, the positional relationship of each target recognition frame at the current moment and the previous moment is continuously detected in chronological order until the operation stage is judged to be the loading completion stage.

[0039] In this embodiment, the process of determining the operation phase of the steel bar bundle loading machine at the current moment according to the position relationship of each target recognition frame at the current moment and the previous moment in step S110 is as follows: Figure 6As shown, it includes: step S1101, when the first target recognition frame in the real-time target image at the current moment and the real-time target image at the previous moment does not contain the second target recognition frame, and the first target recognition frame gradually moves away from the third target recognition frame, it is determined that the operation stage of the loading machinery for loading steel bar bundles on the ship at the current moment is the unloaded stage of the loading machinery; wherein the first target recognition frame is the target recognition frame of the loading machinery, the second target recognition frame is the target recognition frame of the steel bar bundle, and the third target recognition frame is the target recognition frame of the hull; step S1102, when the first target recognition frame in the real-time target image at the current moment and the real-time target image at the previous moment contains the second target recognition frame, and the number of the second target recognition frames in the first target recognition frame gradually increases, it is determined that the operation stage of the loading machinery for loading steel bar bundles on the ship at the current moment is the cargo picking stage of the loading machinery; step S1103, when the real-time target image at the current moment and the real-time target image at the previous moment are both identified. In the real-time target image at the current moment, the first target recognition frame contains an equal number of second target recognition frames, and the first target recognition frame gradually approaches the third target recognition frame, and it is determined that the current moment of the operation stage of the loading machinery loading steel bundles onto the ship is the loading machinery delivery stage; step S1104, when the first target recognition frame and the third target recognition frame in the real-time target image at the current moment and the real-time target image at the previous moment partially overlap, and the number of second target recognition frames in the first target recognition frame gradually decreases, it is determined that the current moment of the operation stage of the loading machinery loading steel bundles onto the ship is the loading machinery loading stage; step S1105, when the overlap between the first target recognition frame and the third target recognition frame in the real-time target image at the current moment and the real-time target image at the previous moment reaches a set ratio, and the number of second target recognition frames in the first target recognition frame reaches a set value, it is determined that the current moment of the operation stage of the loading machinery loading steel bundles onto the ship is the loading completion stage.

[0040] Optionally, the loading machinery picking stage also includes identifying that the first target identification frame in the real-time target image at the current moment and the real-time target image at the previous moment both contain an equal number of second target identification frames, and the distance between the first target identification frame and the third target identification frame remains unchanged; the loading machinery loading stage also includes: identifying that the first target identification frame and the third target identification frame in the real-time target image at the current moment and the real-time target image at the previous moment partially overlap, and the number of second target identification frames in the first target identification frame remains unchanged.

[0041] The smart port cargo management method based on deep learning disclosed in the above embodiment, on the one hand, improves the target detection and recognition model, adopts network training that integrates knowledge transfer and improved YOLOv8 algorithm, and combines pixel adaptive extraction module, weight alignment module and dense residual block to realize real-time recognition of target cargo in smart ports, and the recognition accuracy and stability are significantly improved compared with the prior art; on the other hand, when determining the cargo loading status, by comparing and analyzing the positional relationship of the target recognition frame recognized at the current moment and the previous moment, the operation stage of the current loading machinery is more accurately determined, and then the cargo counting management of the entire cargo loading operation is more accurately performed; compared with the prior art, this solution not only has high cargo management accuracy, but also high management efficiency and low safety hazards.

[0042] As an optional implementation, the smart port cargo management method based on deep learning also includes: The real-time target images corresponding to the loading stage and the loading completion stage of the loading machinery, as well as the real-time video data of the time period corresponding to the real-time target images are intercepted and saved to the cargo management system of the smart port for data traceability.

[0043] As another optional implementation, the smart port cargo management method based on deep learning also includes: recording the total number of times that the current moment's operation stage of loading steel bar bundles onto the ship is the loading machinery picking-up stage based on the positional relationship of each target identification frame between the current moment and the previous moment, and the total number of times that the current moment's operation stage of loading steel bar bundles onto the ship is the loading machinery loading stage based on the positional relationship of each target identification frame between the current moment and the previous moment; calculating the first loading cycle number of the loading machinery corresponding to the loading machinery picking-up stage and the second loading cycle number of the loading machinery corresponding to the loading machinery loading stage based on the total time it takes for the loading machinery to complete a loading cycle of steel bar bundles onto the ship, the single picking-up time of the loading machinery, and the single loading time of the loading machinery; and judging the reliability of the smart port cargo area management based on the matching results of the first loading cycle number and the second loading cycle number.

[0044] Based on the same inventive concept as the above method embodiment, the present application embodiment also provides a smart port cargo management system based on deep learning, such as Figure 7 The figure shows a framework diagram of a smart port cargo management system based on deep learning. Figure 7It can be seen that the smart port management system includes: a first acquisition module, which is used to acquire historical video data of the loading area of ​​the smart port, and perform frame extraction processing on the historical video data to obtain an image to be processed; wherein the historical video data is a historical operation process of loading steel bar bundles on a ship using a loading machinery; a target labeling module, which is used to label each of the images to be processed in turn according to a labeling target list to obtain a number of preprocessed images; wherein the labeling target list includes loading machinery, steel bar bundles, and a hull, and the labeling information of the target labeling includes object position, name, width, height, number of image channels, and label category; a model establishment and training module, which is used to establish a target labeling data set for any target in the labeling target list according to the preprocessed image, and to establish and train a detection and recognition model corresponding to each target based on the target labeling data set; wherein the detection and recognition model is a target detection model based on deep learning, including a first detection and recognition model for identifying loading machinery, a second detection and recognition model for identifying steel bar bundles, and a third detection and recognition model for identifying the hull. a recognition model; a second acquisition module, used to acquire real-time video data of the loading area of ​​the smart port, and perform frame extraction processing on the real-time video data to obtain a real-time target image; wherein the real-time video data is a real-time operation process of loading steel bar bundles onto a ship by a loading machine; a detection and judgment module, used to detect and recognize the real-time target image respectively by using the detection and recognition model corresponding to each target, and judge the operation stage of the loading machine for loading steel bar bundles onto a ship at the current moment according to the positional relationship between each target recognition frame at the current moment and the previous moment; wherein the operation stage includes: the loading machine no-load stage, the loading machine picking-up stage, the loading machine delivery stage, the loading machine loading stage, and the loading completion stage; a counting management module, used to add the count of the steel bar bundles detected and recognized at the loading completion stage to the total number of loadings when the operation stage is judged to be the loading completion stage, so as to complete the cargo counting management of a loading cycle of the loading machine; otherwise, the positional relationship between each target recognition frame at the current moment and the previous moment is continuously detected in chronological order until the operation stage is judged to be the loading completion stage.

[0045] The system is used to implement the steps of the deep learning-based smart port cargo management method disclosed in the above embodiment, which have been described and will not be repeated here.

[0046] For example, the model building and training module builds and trains the execution units of the detection and recognition models corresponding to each target according to the target annotation data set, including: A building unit is used to build a network structure of a detection and recognition model based on a YOLOv8 network, including a backbone network, a neck network and a head module; wherein the backbone network includes a plurality of pixel adaptive extraction modules based on an attention mechanism, the pixel adaptive extraction modules are used to extract different weight features of an input image, and transmit them to the neck network respectively; the neck network includes a weight alignment module and a dense residual block, the weight alignment module and the dense residual block are used to effectively align target features in feature images of different scales; the head module is used to output feature images of three different sizes for final recognition; a division unit is used to divide the target annotation data set of any target in the annotated target list into a training set, a validation set and a test set; a configuration unit is used to configure the loss function of the detection and recognition model, adopting a Wise-IOUv1 loss function with dual attention characteristics, and based on the Wise-IOUv1 loss function, the outlier degree β , monotone focusing coefficient of L-negative loss function of Wise-IOUv1 , constructing a Wise-IOUv3 loss function; a training unit, used for model training, adopting a knowledge transfer training method to train and optimize the detection and recognition model according to the training set and the validation set, and performing a model test in the test set to obtain a detection and recognition model for the corresponding target; Among them, the pixel adaptive extraction module structure of the backbone network designed by the building unit is as follows: including a 1×1 convolution unit, a horizontal weight extraction module, a vertical weight extraction module, a backbone weight extraction module and a fully connected unit; The 1×1 convolution unit is used to increase the input feature dimension. The high-dimensional features output by the convolution unit are equally divided into three parts through a segmentation operation and are respectively input into a horizontal weight extraction module, a vertical weight extraction module and a trunk weight extraction module. After obtaining the horizontal weight and the vertical weight output by the horizontal weight extraction module and the vertical weight extraction module, the sigmoid activation function is used to activate the horizontal weight and the vertical weight by element multiplication. The two activated weights are fused with the trunk weight output by the trunk weight extraction module in the fully connected unit to achieve weight fine-tuning. The expression of the Sigmoid activation formula is: ; The formula for the fully connected unit to output three types of weight information after weight splicing is: ; in, represents the output of the pixel adaptive extraction module, W C represents the horizontal weight, W V represents the vertical weight, W Trepresents the backbone weight, Concat For full connection operation, f Represents the input features.

[0047] The weight alignment module and dense residual block designed in the neck network built by the building unit are used to effectively align the target features in feature images of different scales: The input horizontal weight or vertical weight is subjected to Gaussian smoothing to remove the interference noise in the feature map and smooth the distribution area of ​​the weight; wherein the calculation formula of the Gaussian smoothing is: ; in, represents the point coordinates of the input image, Indicates the standard deviation of the mean value in the neighborhood of the corresponding subscript; Based on the structural measurement index, a wide-area weighted similarity ratio is set, and based on the wide-area weighted similarity ratio, a neighborhood with a weight close to the trunk is screened; wherein the calculation formula of the wide-area weighted similarity ratio is: ; in, Indicates the wide-area weight similarity ratio between the horizontal weight or vertical weight and the trunk weight, H and W Represent the height and width of the feature map respectively, represents the average value of the corresponding neighborhood, N (3 i , 3) indicates the The 3×3 pixel neighborhood of the recognition prediction box on the feature map, N (3 j , 3) indicates the A 3×3 neighborhood of the recognition prediction box, represents the standard deviation of the mean value in the neighborhood of the corresponding subscript, C 1. C 2 is a constant to avoid the denominator being zero.

[0048] Based on the neighborhood obtained by screening, outputting an alignment weight space of the horizontal weight or the vertical weight and the trunk weight; A dense residual block structure is used to perform network feature fusion on the input alignment weight space so that the head module outputs feature images of different sizes; wherein the formula of the dense residual block structure is: ; ; in, Indicates dense connection. F Residual represents the desired output of the network, H(·) represents the internal nonlinear transformation module Conv-ReLU operation, x i Represents the input features of the corresponding layer.

[0049] The Wise-IOUv3 loss function of the configuration unit is as follows: The coordinates of a set point on the recognition prediction box of the detection recognition model are defined as ( x , y ), and its corresponding coordinates on the real frame are ( x t , y t ), then the calculation formula of Wise-IOUv1 loss function and Wise-IOUv3 loss function is expressed as: ; ; ; ; ; in, w c , h c Indicates the width and height of the minimum bounding rectangle of the prediction box and the real box. b , b t Respectively represent the center points of the predicted box and the real box, represents the scale factor, represents the smoothing parameter and α represents the focusing coefficient.

[0050] When the system is implemented specifically, the process in which its detection and judgment module determines the operation stage of the loading machinery for loading steel bundles on the ship at the current moment according to the positional relationship between the target recognition frames at the current moment and the previous moment is as follows: when the first target recognition frame in the real-time target image at the current moment and the real-time target image at the previous moment does not contain the second target recognition frame, and the first target recognition frame gradually moves away from the third target recognition frame, it is determined that the operation stage of the loading machinery for loading steel bundles on the ship at the current moment is the no-load stage of the loading machinery; wherein, the first target recognition frame is the target recognition frame of the loading machinery, the second target recognition frame is the target recognition frame of the steel bundle, and the third target recognition frame is the target recognition frame of the hull; when the first target recognition frame in the real-time target image at the current moment and the real-time target image at the previous moment contains the second target recognition frame, and the number of second target recognition frames in the first target recognition frame gradually increases, it is determined that the operation stage of the loading machinery for loading steel bundles on the ship at the current moment is the loading machinery loading stage. cargo stage; when the first target recognition frame in the real-time target image at the current moment and the real-time target image at the previous moment contains an equal number of second target recognition frames, and the first target recognition frame gradually approaches the third target recognition frame, it is determined that the current moment of the operation stage of the loading machinery loading steel bundles onto the ship is the loading machinery delivery stage; when the first target recognition frame and the third target recognition frame in the real-time target image at the current moment and the real-time target image at the previous moment partially overlap, and the number of second target recognition frames in the first target recognition frame gradually decreases, it is determined that the current moment of the operation stage of the loading machinery loading steel bundles onto the ship is the loading machinery loading stage; when the overlap between the first target recognition frame and the third target recognition frame in the real-time target image at the current moment and the real-time target image at the previous moment reaches a set ratio, and the number of second target recognition frames in the first target recognition frame reaches a set value, it is determined that the current moment of the operation stage of the loading machinery loading steel bundles onto the ship is the loading completion stage.

[0051] As an optional implementation, the smart port cargo management system also includes: a storage module for intercepting and saving the real-time target images corresponding to the loading stage and the loading completion stage of the loading machinery, and the real-time video data of the time period corresponding to the real-time target image to the cargo management system of the smart port, so that later staff can review it from the electronic work platform of the smart port for data traceability.

[0052] In summary, the smart port cargo management method and system based on deep learning disclosed in the present invention are applied to the smart port cargo management scenario, and have the characteristics of high statistical efficiency, high degree of automation, high accuracy, safety, 24-hour uninterrupted operation, and unattended liberation of labor, which greatly reduces the labor intensity of workers, reduces safety hazards, and reduces labor costs. In addition, the technical solution of the present invention is not limited to the public loading machinery for steel bar bundle loading operations, and other management processes such as cargo loading, cargo unloading and transshipment, and cargo area archiving in smart ports can also be adopted.

[0053] Based on the same inventive concept as the above-mentioned method embodiment, an electronic device is also provided in an embodiment of the present application, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, the electronic device implements the control method in the above-mentioned embodiment.

[0054] In one embodiment, the electronic device may be a server. In this embodiment, the structure of the electronic device may be as follows: Figure 8 As shown, it includes a memory 201 , a communication module 203 and one or more processors 202 .

[0055] The memory 201 is used to store computer programs executed by the processor 202. The memory 201 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and programs required for running the instant messaging function, etc.; the data storage area may store various instant messaging information and operation instruction sets, etc.

[0056] The memory 201 may be a volatile memory, such as a random-access memory (RAM); the memory 201 may also be a non-volatile memory, such as a read-only memory, a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD); or the memory 201 may be any other medium that can be used to carry or store a desired computer program in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 201 may be a combination of the above memories.

[0057] The processor 202 may include one or more central processing units (CPU) or a digital processing unit, etc. The processor 202 is configured to implement the above-mentioned audio data processing method when calling the computer program stored in the memory 201 .

[0058] The communication module 203 is used to communicate with terminal devices and other servers.

[0059] The specific connection medium between the memory 201, the communication module 203 and the processor 202 is not limited in the embodiment of the present application. Figure 8 In the embodiment, the memory 201 and the processor 202 are connected via a bus 204. The bus 204 is connected to the processor 202 via a bus 204. Figure 8 The arrows in the figure are used to describe the connection between other components, which is only for illustrative purposes and is not intended to be limiting. Bus 204 can be divided into an address bus, a data bus, a control bus, etc. For ease of description, Figure 8 The diagram only uses one arrow, but does not describe only one bus or one type of bus.

[0060] Based on the same inventive concept as the above method embodiment, an embodiment of the present invention also provides a computer-readable storage medium, which is used to store a computer program. When the computer program is run on a computer, the electronic device implements the control method in the above embodiment. The computer-readable storage medium can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a magnetic disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0061] Based on the same inventive concept as the above-mentioned method embodiment, an embodiment of the present invention further provides a computer program product, the computer program product including a computer program. When the program product is run on an electronic device, the computer program is used to enable the electronic device to execute the steps of the control method according to various exemplary embodiments of the present application described above in this specification. The program product may adopt any combination of one or more readable media. These computer program commands may be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the commands executed by the processor of the computer or other programmable data processing device generate commands for implementing the steps in the process Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0062] Although the present invention has been disclosed as above with preferred embodiments, it is not intended to limit the present invention. A person with ordinary knowledge in the technical field to which the present invention belongs may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, the protection scope of the present invention shall be determined by the definition of the claims.

Claims

1. A smart port cargo management method based on deep learning, characterized in that: include: Acquire historical video data of the loading area of ​​the smart port, and perform frame extraction processing on the historical video data to obtain an image to be processed; wherein the historical video data is a historical operation process of loading steel bar bundles onto a ship using a loading machine; For each of the images to be processed, target annotation is performed in sequence according to the annotation target list to obtain a number of pre-processed images; wherein the annotation target list includes loading machinery, steel bar bundles, and ship hulls, and the annotation information of the target annotation includes object position, name, width, height, number of image channels, and label category; According to the preprocessed image, a target annotation data set is established for any target in the annotated target list, and a detection and recognition model corresponding to each target is established and trained according to the target annotation data set; wherein the detection and recognition model is a target detection model based on deep learning, including a first detection and recognition model for identifying a ship loading machinery, a second detection and recognition model for identifying a steel bar bundle, and a third detection and recognition model for identifying a ship hull; Acquire real-time video data of the loading area of ​​the smart port, and perform frame extraction processing on the real-time video data to obtain a real-time target image; wherein the real-time video data is the real-time operation process of loading steel bar bundles onto a ship using a loading machine; The detection and recognition models corresponding to each target are used to detect and recognize the real-time target images respectively, and the operation stage of the loading machinery for loading steel bar bundles is judged at the current moment according to the position relationship between the target recognition frames at the current moment and the previous moment; wherein the operation stage includes: the loading machinery no-load stage, the loading machinery picking-up stage, the loading machinery delivery stage, the loading machinery loading stage, and the loading completion stage; When the operation stage is judged to be the loading completion stage, the count of the steel bar bundles detected and identified in the loading completion stage is added to the total number of loadings, completing the cargo counting management of one loading cycle of the loading machinery; otherwise, the positional relationship of each target recognition frame at the current moment and the previous moment is continuously detected in chronological order until the operation stage is judged to be the loading completion stage.

2. According to claim 1, the smart port cargo management method based on deep learning is characterized in that: The process of establishing and training the detection and recognition models corresponding to each target based on the target annotation data set is as follows: Based on the YOLOv8 network, a network structure of the detection and recognition model is constructed, including a backbone network, a neck network and a head module; wherein the backbone network includes several pixel adaptive extraction modules based on the attention mechanism, and the pixel adaptive extraction modules are used to extract different weight features of the input image and transmit them to the neck network respectively; the neck network includes a weight alignment module and a dense residual block, and the weight alignment module and the dense residual block are used to effectively align the target features in feature images of different scales; the head module is used to output feature images of three different sizes for final recognition; For a target annotation data set of any target in the annotated target list, divide it into a training set, a validation set and a test set; Configure the loss function of the detection and recognition model, use the Wise-IOUv1 loss function with dual attention characteristics, and based on the Wise-IOUv1 loss function, outlier β , monotone focusing coefficient of L-negative loss function of Wise-IOUv1 , construct Wise-IOUv3 loss function; Model training, using a knowledge transfer training method to train and optimize the detection and recognition model according to the training set and the validation set, and performing model testing in the test set to obtain a detection and recognition model for the corresponding target; Among them, the coordinates of a set point on the recognition prediction box of the detection recognition model are defined as ( x , y ), and its corresponding coordinates on the real frame are ( x t , y t ), then the calculation formula of Wise-IOUv1 loss function and Wise-IOUv3 loss function is expressed as: ； ； ； ； ； in, w c , h c Indicates the width and height of the minimum bounding rectangle of the prediction box and the real box. b , b t Respectively represent the center points of the predicted box and the real box, represents the scale factor, represents the smoothing parameter and α represents the focusing coefficient.

3. The smart port cargo management method based on deep learning according to claim 2 is characterized in that: The pixel adaptive extraction module includes a 1×1 convolution unit, a horizontal weight extraction module, a vertical weight extraction module, a trunk weight extraction module and a fully connected unit; The 1×1 convolution unit is used to increase the input feature dimension. The high-dimensional features output by the convolution unit are equally divided into three parts through a segmentation operation and are respectively input into a horizontal weight extraction module, a vertical weight extraction module and a trunk weight extraction module. After obtaining the horizontal weight and the vertical weight output by the horizontal weight extraction module and the vertical weight extraction module, the sigmoid activation function is used to activate the horizontal weight and the vertical weight by element multiplication. The two activated weights are fused with the trunk weight output by the trunk weight extraction module in the fully connected unit to achieve weight fine-tuning. The expression of the Sigmoid activation formula is: ; The formula for the fully connected unit to output three types of weight information after weight splicing is: ； in, represents the output of the pixel adaptive extraction module, W C represents the horizontal weight, W V represents the vertical weight, W T represents the backbone weight, Concat For full connection operation, f Represents the input features.

4. The deep learning-based smart port cargo management method according to claim 3 is characterized in that: The process in which the weight alignment module and the dense residual block in the neck network are used to effectively align target features in feature images of different scales is as follows: The input horizontal weight or vertical weight is subjected to Gaussian smoothing to remove the interference noise in the feature map and smooth the distribution area of ​​the weight; wherein the calculation formula of the Gaussian smoothing is: ; in, represents the point coordinates of the input image, Indicates the standard deviation of the mean value in the neighborhood of the corresponding subscript; Based on the structural measurement index, a wide-area weighted similarity ratio is set, and based on the wide-area weighted similarity ratio, a neighborhood with a weight close to the trunk is screened; wherein the calculation formula of the wide-area weighted similarity ratio is: ； in, It represents the similarity ratio of the wide-area weight between the horizontal weight or vertical weight and the trunk weight. H and W Represent the height and width of the feature map respectively, represents the average value of the corresponding neighborhood, N (3 i , 3) indicates the The 3×3 pixel neighborhood of the recognition prediction box on the feature map, N (3 j , 3) indicates the A 3×3 neighborhood of the recognition prediction box, represents the standard deviation of the mean value in the neighborhood of the corresponding subscript, C 1. C 2 is a constant to avoid the denominator being zero; Based on the neighborhood obtained by screening, outputting an alignment weight space of the horizontal weight or the vertical weight and the trunk weight; A dense residual block structure is used to perform network feature fusion on the input alignment weight space so that the head module outputs feature images of different sizes; wherein the formula of the dense residual block structure is: ； ； in, Indicates dense connection. F Residual represents the desired output of the network, H (·) represents the internal nonlinear transformation module Conv-ReLU operation, x i Represents the input features of the corresponding layer.

5. The smart port cargo management method based on deep learning according to claim 1 is characterized in that: The process of judging the operation stage of the loading machinery for loading steel bar bundles on the ship at the current moment according to the position relationship of each target recognition frame at the current moment and the previous moment is as follows: When the first target recognition frame in the real-time target image at the current moment and the real-time target image at the previous moment does not contain the second target recognition frame, and the first target recognition frame gradually moves away from the third target recognition frame, it is determined that the operation stage of the ship loading machinery loading steel bar bundles at the current moment is the no-load stage of the ship loading machinery; wherein the first target recognition frame is the target recognition frame of the ship loading machinery, the second target recognition frame is the target recognition frame of the steel bar bundle, and the third target recognition frame is the target recognition frame of the hull; When the first target recognition frame in the real-time target image at the current moment and the real-time target image at the previous moment both contain the second target recognition frame, and the number of the second target recognition frames in the first target recognition frame gradually increases, it is determined that the operation stage of the loading machinery for loading steel bar bundles is the loading machinery picking-up stage at the current moment; When the first target recognition frame in the real-time target image at the current moment and the real-time target image at the previous moment both contain an equal number of second target recognition frames, and the first target recognition frame gradually approaches the third target recognition frame, it is determined that the operation stage of the loading machinery for loading steel bar bundles is the loading machinery delivery stage at the current moment; When the first target recognition frame and the third target recognition frame in the real-time target image at the current moment and the real-time target image at the previous moment are partially overlapped, and the number of the second target recognition frames in the first target recognition frame gradually decreases, it is determined that the operation stage of the loading machinery for loading steel bar bundles at the current moment is the loading machinery loading stage; When the overlap between the first target identification frame and the third target identification frame in the real-time target image at the current moment and the real-time target image at the previous moment reaches a set ratio, and the number of second target identification frames in the first target identification frame reaches a set value, it is determined that the operation stage of the loading machinery for loading steel bar bundles at the current moment is the loading completion stage.

6. The deep learning-based smart port cargo management method according to claim 1 is characterized in that: Also includes: The real-time target images corresponding to the loading stage and the loading completion stage of the loading machinery, as well as the real-time video data of the time period corresponding to the real-time target images are intercepted and saved to the cargo management system of the smart port for data traceability.

7. A smart port cargo management system based on deep learning, characterized in that: include: The first acquisition module is used to acquire historical video data of the loading area of ​​the smart port, and perform frame extraction processing on the historical video data to obtain an image to be processed; wherein the historical video data is a historical operation process of loading steel bar bundles onto a ship using a loading machine; The target annotation module is used to annotate each of the to-be-processed images in sequence according to the annotated target list to obtain a plurality of pre-processed images; wherein the annotated target list includes ship loading machinery, steel bar bundles, and ship hulls, and the annotation information of the target annotation includes object position, name, width, height, number of image channels, and label category; A model building and training module, used to establish a target annotation data set for any target in the annotated target list according to the preprocessed image, and to establish and train a detection and recognition model corresponding to each target according to the target annotation data set; wherein the detection and recognition model is a target detection model based on deep learning, including a first detection and recognition model for identifying a ship loading machinery, a second detection and recognition model for identifying a steel bar bundle, and a third detection and recognition model for identifying a ship hull; The second acquisition module is used to acquire real-time video data of the loading area of ​​the smart port, and perform frame extraction processing on the real-time video data to obtain a real-time target image; wherein the real-time video data is a real-time operation process of loading steel bar bundles onto a ship using a loading machine; A detection and judgment module is used to detect and identify the real-time target images respectively using the detection and recognition models corresponding to each target, and judge the operation stage of the loading machinery for loading steel bar bundles at the current moment according to the position relationship between the target recognition frames at the current moment and the previous moment; wherein the operation stage includes: the loading machinery empty stage, the loading machinery picking up stage, the loading machinery delivery stage, the loading machinery loading stage, and the loading completion stage; The counting management module is used to add the count of the steel bar bundles detected and identified in the loading completion stage to the total number of loadings when the operation stage is judged to be the loading completion stage, so as to complete the cargo counting management of a loading cycle of the loading machinery; otherwise, the position relationship of each target recognition frame at the current moment and the previous moment is continuously detected in chronological order until the operation stage is judged to be the loading completion stage.

8. The deep learning-based smart port cargo management system according to claim 7 is characterized in that: The model building and training module builds and trains the execution units of the detection and recognition models corresponding to each target according to the target annotation data set, including: A building unit is used to build a network structure of a detection and recognition model based on a YOLOv8 network, including a backbone network, a neck network and a head module; wherein the backbone network includes a plurality of pixel adaptive extraction modules based on an attention mechanism, the pixel adaptive extraction modules are used to extract different weight features of an input image, and transmit them to the neck network respectively; the neck network includes a weight alignment module and a dense residual block, the weight alignment module and the dense residual block are used to effectively align target features in feature images of different scales; the head module is used to output feature images of three different sizes for final recognition; A division unit, used for dividing a target annotation data set of any target in the annotated target list into a training set, a validation set and a test set; The configuration unit is used to configure the loss function of the detection and recognition model. The Wise-IOUv1 loss function with dual attention characteristics is adopted, and based on the Wise-IOUv1 loss function, outlier degree β , monotone focusing coefficient of L-negative loss function of Wise-IOUv1 , construct Wise-IOUv3 loss function; A training unit, used for model training, adopts a knowledge transfer training method to train and optimize the detection and recognition model according to the training set and the validation set, and performs model testing in the test set to obtain a detection and recognition model corresponding to the target; Among them, the coordinates of a set point on the recognition prediction box of the detection recognition model are defined as ( x , y ), and its corresponding coordinates on the real frame are ( x t , y t ), then the calculation formula of Wise-IOUv1 loss function and Wise-IOUv3 loss function is expressed as: ； ； ； ； ； in, w c , h c Indicates the width and height of the minimum bounding rectangle of the prediction box and the real box. b , b t Respectively represent the center points of the predicted box and the real box, represents the scale factor, represents the smoothing parameter and α represents the focusing coefficient.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the electronic device implements the smart port cargo management method based on deep learning as described in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store a computer program. When the computer program is run on a computer, the computer executes the deep learning-based smart port cargo management method as described in any one of claims 1 to 6.

Images