A ship plate detection positioning method, a storage medium and an electronic device

Abstract

The application relates to a ship plate detection positioning method, a storage medium and electronic equipment, which comprises the following steps: labeling a ship picture shot by a port wharf, establishing a ship plate detection positioning model, updating the ship plate detection positioning model through a non-local neighborhood calculation alignment algorithm and a channel fusion enhancement algorithm; using the above data set, training and adjusting the updated ship plate detection positioning model to obtain a trained ship plate detection positioning model; shooting a monitoring video stream through a camera arranged at the port wharf, inputting the monitoring video stream into the trained ship plate detection positioning model, detecting a ship plate on the port wharf, and calculating a positioning coordinate of the ship plate appearing in the video stream. Differing from the prior art, the application performs non-local alignment on multi-level features in a feature pyramid through a multi-scale non-local algorithm, fuses channel features between feature maps before and after alignment, and further improves the detection precision of the algorithm.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of ship plate detection, and in particular relates to a ship plate detection positioning method, a storage medium and an electronic device. BACKGROUND

[0002] The identity management of a ship is currently lack of unified specifications, and using a traditional method to manage the identity of a ship often requires a large amount of manpower and resources for identification and recording. With the rapid development of artificial intelligence and deep learning, using a target detection method to detect and position a ship plate has become a method that can greatly improve efficiency.

[0003] When using a target detection technology to detect a ship at present, the detection difficulty is relatively large due to insufficient shooting hardware conditions, poor shooting environment and light conditions, interference to the ship plate, and a small target.

[0004] More information related to the above technical solutions can be found in the following documents:

[0005] In the invention patent with the publication number CN112633277A, a channel ship plate detection, positioning and identification method based on deep learning is disclosed, which uses a semantic segmentation method to perform pixel-level classification on a ship plate image based on an AdvancedEAST algorithm, so as to obtain an accurate and reliable ship plate text line boundary box, and combines a CRNN character recognition algorithm to output accurate ship plate character information. However, the above method does not align the ship plate, resulting in misalignment of ship plate recognition, poor recognition accuracy and inaccurate recognition.

[0006] In the invention patent with the publication number CN111461116A, a ship plate text recognition model is disclosed, which includes a ship plate text feature extraction module for receiving a ship plate region standard image and extracting a convolution feature map therefrom; a ship plate text cycle module for making sequence prediction on the convolution feature map; a ship plate text translation module for comprehensively integrating all sequence predictions and decoding output text information against a character library; the ship plate text feature extraction module, the ship plate text cycle module and the ship plate text translation module are connected in sequence, and the ship plate text translation module is further connected with the ship plate text feature extraction module.

[0007] In the process of implementing the present application, the inventors found the following problems in the prior art:

[0008] In the prior art, the feature map is not aligned during upsampling and downsampling in the ship plate detection process, which affects the ship plate detection accuracy and results in poor detection accuracy. SUMMARY

[0009] In view of the above problems, the present application provides a ship plate detection positioning method, a storage medium and an electronic device, which are used to solve the technical problem that the feature map is not aligned during upsampling and downsampling in the process of ship plate detection in the prior art, thereby affecting the ship plate detection accuracy and reducing the detection accuracy.

[0010] To achieve the above-mentioned object, in a first aspect, the present application provides a ship plate detection positioning method, comprising the following steps:

[0011] A camera is arranged at a port terminal, the camera is in communication connection with a server, and a photographed ship picture is transmitted to the server;

[0012] The ship picture photographed at the port terminal is labeled, and the ship plate appearing in the ship picture is marked as a data set for training and testing, which is ready for use;

[0013] A ship plate detection positioning model is established, a deformable convolution alignment algorithm and a non-local neighborhood calculation alignment algorithm are respectively used to align the local and global features of the ship picture, and a channel fusion enhancement algorithm is further used to update the ship plate detection positioning model;

[0014] The updated ship plate detection positioning model is trained and parameterized using the above-mentioned data set, and a trained ship plate detection positioning model is obtained;

[0015] A monitoring video stream is photographed by the camera arranged at the port terminal, the monitoring video stream is input into the trained ship plate detection positioning model, a ship plate on a ship at the port terminal is detected, and a positioning coordinate of the ship plate appearing in the video stream is calculated;

[0016] The ship plate positioning coordinate calculated is used to frame and display the position of the ship plate in the image signal collected by the current camera in real time.

[0017] Different from the prior art, the above technical solution uses a deformable convolution alignment algorithm and a non-local neighborhood calculation alignment algorithm to respectively align the local and global features of the ship picture, further uses a channel fusion enhancement algorithm to update the ship plate detection positioning model, uses a real data set to train and parameterize the updated ship plate detection positioning model, obtains a new ship plate detection positioning model, and uses the ship plate positioning coordinate calculated to frame and display the position of the ship plate in the image signal collected by the current camera in real time. In this way, a multi-scale non-local algorithm is used to perform non-local alignment on the multi-level features in the feature pyramid, so as to improve the robustness of the algorithm and reduce the calculation amount. In addition, in order to make up for the lack of correlation between channels in the non-local algorithm, the channel features between the feature maps before and after alignment are fused, so as to further improve the detection accuracy of the algorithm.

[0018] As an embodiment of the present application, the non-local neighborhood computing alignment algorithm is specifically a non-local algorithm using a multi-scale feature alignment through global correlation computing using a feature pyramid.

[0019] In this way, by introducing context information in the computation, the influence of noise is reduced, and the ability of the pixel to compute relevant dependent information is improved.

[0020] As an embodiment of the present application, the calculation formula of the non-local neighborhood computing alignment algorithm is as follows:

[0021]

[0022] where i is the position index on the feature map that needs to be aligned, j is the position index on the feature map that is the alignment ruler, z j is the feature mapping matrix of the feature map that is the alignment ruler, the function φ is a long-range dependent relationship of the i, j positions on the two feature maps, the function θ is a mapping of the z j feature map, the new feature mapping of the representation z j , and the output response y i is obtained by summing the display measurements of the two feature maps, and the data is normalized by the scalar function C(x, z).

[0023] In this way, the robustness of the algorithm can be improved, the amount of computation can be reduced, and the influence of noise can be reduced.

[0024] As an embodiment of the present application, a neighborhood computing algorithm is added to the non-local neighborhood computing alignment algorithm, and the calculation formula after adding the neighborhood computing algorithm is as follows:

[0025]

[0026] where δ(r) represents a neighborhood centered on the input index position, and determines the size of the neighborhood block of the picture segmentation in the neighborhood computing. The calculation formula after adding the neighborhood computing algorithm changes the original single pixel computation to neighborhood computation centered on the index pixel.

[0027] In this way, neighborhood similarity computation is added, and the single pixel computation is extended to neighborhood pixel similarity computation. Neighborhood computation brings stronger constraints to alignment, makes the alignment parts highly relevant, and suppresses irrelevant neighborhoods.

[0028] As an embodiment of the present application, after the non-local algorithm aligns the global features, a deformable convolution alignment algorithm with a local receptive field is used to further supplement the preliminarily aligned features.

[0029] Thus, the features preliminarily aligned can be further supplemented by the deformable convolution alignment method, and the image is optimized.

[0030] As an embodiment of the present application, the calculation formula of the deformable convolution alignment method is as follows:

[0031]

[0032]

[0033] wherein, is the feature map after alignment, Δ i is the offset between the two feature maps, T i is the feature map to be aligned, is the reference feature map as the alignment standard. is and T i is the concatenation of the two feature maps, f align is the alignment function for aligning the feature map i with the offset Δ is the alignment function for aligning the feature map offset is the function for calculating the offset between the two feature maps, both of which are implemented by using the deformable convolution.

[0034] Thus, the accuracy of the image is further improved by the deformable convolution alignment method.

[0035] As an embodiment of the present application, the channel fusion enhancement algorithm specifically comprises performing the ratio-invariant adaptive average pooling and the maximum pooling on the feature maps of different scales respectively, and fusing and calculating, so as to aggregate the original feature mapping space information and the fused feature mapping information.

[0036] Thus, by compressing and aggregating the information on the spatial scale of each feature map, and cooperating with the multi-scale alignment module, the problems of "what" and "how much" are respectively concerned, and the problem of lack of channel information in the non-local alignment operation is made up.

[0037] As an embodiment of the present application, the calculation formula of the channel fusion enhancement algorithm is as follows:

[0038]

[0039]

[0040]

[0041] Mc(F)

[0042] =σ(MLP(Mc(F1);Mc(F2);Mc(F3)))

[0043] =σ(W1(F cat ))

[0044] wherein F c avg and F c max are adaptive average pooling and adaptive max pooling operation respectively, MLP represents calculation using multi-layer perceptron, F1 and F2 are feature maps of upper layer and feature maps of current layer, F3 is feature map after feature fusion, σ represents sigmoid activation function, W0 and W1 are weights of multi-layer perceptron, which are shared for F1 and F2.

[0045] In this way, the detection accuracy can be further improved in combination with the multi-scale alignment algorithm.

[0046] To achieve the above object, in a second aspect, the inventor provides a storage medium storing a computer program, which, when executed by a processor, implements the method steps of any of the above.

[0047] To achieve the above object, in a third aspect, the inventor provides an electronic device comprising a processor and a storage medium, wherein the storage medium is as described above.

[0048] The processor is configured to execute the computer program stored in the storage medium to implement the method steps of any of the above.

[0049] Compared with the prior art, the technical solution of the present application uses a multi-scale non-local algorithm to perform non-local alignment on multi-level features in a feature pyramid, so as to improve the robustness of the algorithm and reduce the amount of calculation. In addition, in order to make up for the lack of correlation between channels in the non-local algorithm, the channel features between the feature maps before and after alignment are fused, so as to further improve the detection accuracy of the algorithm.

[0050] The above summary is only a summary of the technical solution of the present application. In order to enable those skilled in the art to more clearly understand the technical solution of the present application, and then implement the content described in the specification and drawings, and in order to enable the above and other objects, features and advantages of the present application to be more easily understood, the following will be described in combination with the specific embodiments of the present application and the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0051] The accompanying drawings are only used to show the principles, implementation manners, applications, characteristics and effects of the specific embodiments of the present application and other related contents, and cannot be considered as limitations of the present application.

[0052] In the drawings of the specification:

[0053] Figure 1 Flow chart of a ship plate detection and positioning method according to an embodiment of the present application;

[0054] Figure 2 Module schematic diagram of an electronic device according to an embodiment of the present application;

[0055] Figure 3 Anti-interference performance comparison result diagram of final display according to an embodiment of the present application;

[0056] Figure 4 Comparison result diagram of improved missed detection of final display according to an embodiment of the present application;

[0057] Figure 5 Comparison result diagram of improved detection precision of final display according to an embodiment of the present application;

[0058] Figure 6 Schematic diagram of a prediction box according to an embodiment of the present application;

[0059] Figure 7 Schematic diagram of an intersection-over-union according to an embodiment of the present application.

[0060] The reference signs involved in the above-mentioned figures are explained as follows:

[0061] 10, electronic device;

[0062] 101, processor;

[0063] 102, storage medium. DETAILED DESCRIPTION

[0064] In order to describe the possible application scenarios, technical principles, specific implementable schemes, and the purposes and effects achieved by the present application in detail, the following will be described in detail in combination with the specific embodiments listed and the accompanying drawings. The embodiments described in the present text are only used to more clearly illustrate the technical schemes of the present application, and therefore cannot be used to limit the protection scope of the present application.

[0065] In the present text, the term “embodiment” means that the specific features, structures or characteristics described in combination with the embodiments can be contained in at least one embodiment of the present application. The term “embodiment” appearing at various positions in the specification does not necessarily refer to the same embodiment, and does not particularly limit the independence or association between other embodiments. In principle, in the present application, as long as there is no technical contradiction or conflict, the technical features mentioned in each embodiment can be combined in any way to form a corresponding implementable technical scheme.

[0066] Unless otherwise defined, the meanings of technical terms used in the present application are the same as commonly understood by one of ordinary skill in the art to which the present application belongs; the use of related terms in the present application is only for the purpose of describing specific embodiments and is not intended to limit the present application.

[0067] In the description of the present application, the phrase "and / or" is a description of a logical relationship between objects, which means that there can be three relationships, for example, A and / or B, which means that there are three cases: A exists, B exists, and A and B exist at the same time. In addition, the character " / " in the present application generally represents that the associated objects before and after are a kind of "or" logical relationship.

[0068] In the present application, phrases such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual quantity, primary and secondary or order relationship between the entities or operations.

[0069] In the present application, the phrases "include", "contain", "have" or other similar expressions used in the statements are intended to cover non-exclusive inclusion, and these expressions do not exclude the presence of other elements in the process, method or product including the described elements, so that the process, method or product including a series of elements can not only include those limited elements, but also include other elements not explicitly listed, or also include elements inherent to such process, method or product.

[0070] As the same as the understanding in the "Guidelines for Examination", in the present application, the expressions "greater than", "less than", "exceed" and the like are understood as not including the number; the expressions "above", "below", "within" and the like are understood as including the number. In addition, in the description of the embodiments of the present application, the meaning of "multiple" is more than two (including two), and similar expressions related to "multiple" are also understood in this way, for example, "multiple groups", "multiple times" and the like, unless otherwise explicitly limited.

[0071] In the prior art, during the detection of ship plates, the feature map will not be aligned during upsampling and downsampling, which affects the detection accuracy of ship plates and results in poor detection accuracy.

[0072] The applicant finds that optimizing the ship plate positioning algorithm in the space and channel dimensions, first, through a multi-scale non-local algorithm, the multi-level features in the feature pyramid are aligned non-locally, and a neighborhood calculation algorithm is added, which brings stronger constraints to the feature map alignment, makes the alignment part highly relevant, suppresses irrelevant neighborhoods, improves the robustness of the algorithm, and reduces the calculation amount. At the same time, in order to make up for the lack of correlation between channels in the non-local algorithm, a new channel fusion enhancement module is designed, which further improves the detection accuracy of the algorithm by fusing the channel features between the pre-alignment and post-alignment feature maps. It can be easily embedded into various current mainstream target detection algorithms using feature pyramids. When using the application to position and detect ship plates, detection errors caused by interference of ship plates can be effectively avoided, and detection accuracy can be improved.

[0073] The ship plate detection and positioning method, storage medium and electronic device involved in the embodiment can be applied to various application scenarios requiring ship plate detection and positioning.

[0074] According to some embodiments of the present application, referring to Figures 1 to 6 The embodiment relates to a ship plate detection and positioning method, comprising the following steps:

[0075] A camera is arranged at a port terminal, the camera is in communication connection with a server, and a ship picture captured by the camera is transmitted to the server;

[0076] The ship picture captured by the port terminal is labeled, and a ship plate appearing in the ship picture is marked as a data set for training and testing, for standby use;

[0077] A ship plate detection and positioning model is established, a deformable convolution alignment algorithm and a non-local neighborhood calculation alignment algorithm are respectively used to align local and global features of the ship picture, and a channel fusion enhancement algorithm is further added to update the ship plate detection and positioning model;

[0078] The updated ship plate detection and positioning model is trained and parameter-adjusted using the above data set, and a trained ship plate detection and positioning model is obtained;

[0079] A monitoring video stream is captured by the camera arranged at the port terminal, the monitoring video stream is input into the trained ship plate detection and positioning model, a ship plate on a ship at the port terminal is detected, and a ship plate positioning coordinate appearing in the video stream is calculated;

[0080] The ship plate positioning coordinate calculated is used to frame and display the position of the ship plate in an image signal currently collected by the camera in real time.

[0081] Distinguish from prior art, the above technical scheme is through the deformable convolution alignment algorithm and the non-local neighborhood calculation alignment algorithm respectively aligning the local and global features of the ship picture, then increasing the channel fusion enhancement algorithm to update the ship plate detection positioning model, using the real data set to train and parameterize the updated ship plate detection positioning model, obtaining a new ship plate detection positioning model, the ship plate positioning coordinates can be calculated, the position of the ship plate in the image signal collected by the current camera is framed and displayed in real time. In this way, through the multi-scale non-local algorithm, the multi-level features in the feature pyramid are non-locally aligned to improve the robustness of the algorithm and reduce the calculation amount. At the same time, in order to make up for the lack of correlation between channels in the non-local algorithm, the channel features between the feature maps before and after alignment are fused to further improve the detection accuracy of the algorithm.

[0082] According to some embodiments of the application, optionally, the non-local neighborhood calculation alignment algorithm is specifically using a non-local algorithm to perform multi-scale feature alignment through global correlation calculation using a feature pyramid.

[0083] In this way, by introducing context information in the calculation, the influence of noise is reduced, and the ability of pixel calculation related dependent information is improved.

[0084] According to some embodiments of the application, optionally, the calculation formula of the non-local neighborhood calculation alignment algorithm is as follows:

[0085]

[0086] Wherein, i is the position index on the feature map that needs to be aligned, j is the position index on the feature map that needs to be aligned, z j is the feature mapping matrix of the feature map that needs to be aligned, the function φ is a long-distance dependent relationship between i and j positions on two feature maps, the function θ is a mapping of the feature map z j , the new feature mapping of z j , the output response y i is obtained by summing the display measurements of the two feature maps, and the data is normalized by the scalar function C(x,z).

[0087] In this way, the robustness of the algorithm can be improved, the calculation amount can be reduced, and the influence of noise can be reduced.

[0088] According to some embodiments of the application, optionally, a neighborhood calculation algorithm is added to the non-local neighborhood calculation alignment algorithm, and the calculation formula after adding the neighborhood calculation algorithm is as follows:

[0089]

[0090] Wherein, δ(r) represents a neighborhood centered on the input index position, determines the size of the neighborhood block of picture segmentation in the neighborhood calculation, and the calculation formula after adding the neighborhood calculation algorithm changes the original single pixel calculation to neighborhood calculation centered on the index pixel.

[0091] In this way, the neighborhood similarity calculation is increased, the single pixel calculation is expanded to the similarity calculation with neighborhood pixels, the neighborhood calculation brings stronger constraint to the alignment, makes the alignment position highly relevant, and suppresses irrelevant neighborhoods.

[0092] According to some embodiments of the present application, optionally, after the global feature is aligned by the non-local algorithm, a deformable convolution alignment algorithm with a local receptive field is used to further supplement the preliminarily aligned feature.

[0093] In this way, the preliminarily aligned feature can be further supplemented by the deformable convolution alignment algorithm, and the image is optimized.

[0094] According to some embodiments of the present application, optionally, the calculation formula of the deformable convolution alignment method is as follows:

[0095]

[0096]

[0097] Wherein, is the feature map after alignment, Δ i is the offset between the two feature maps, T i is the feature map to be aligned, is the feature map to be aligned. is and T i are the concatenation of the two feature maps, f align is the alignment function of aligning the feature map i with the offset Δ is the alignment function of aligning the feature map offset is a function of calculating the offset between the two feature maps, both of which are implemented by using deformable convolution.

[0098] In this way, the accuracy of the image is further improved by the deformable convolution alignment method. According to some embodiments of the present application, optionally, the channel fusion enhancement algorithm is specifically to respectively perform the ratio-invariant adaptive average pooling and the maximum pooling on the feature maps of different scales, and to fuse and calculate, so as to aggregate the original feature mapping space information and the fused feature mapping information.

[0099] In this way, by compressing the information on each feature map space scale and cooperating with the multi-scale alignment module, the problems of "what" and "how much" are respectively focused on, and the problem of lack of channel information in the non-local alignment operation is compensated.

[0100] According to some embodiments of the present application, optionally, the calculation formula of the channel fusion enhancement algorithm is as follows:

[0101]

[0102]

[0103]

[0104] Mc(F)

[0105] =σ(MLP(Mc(F1);Mc(F2);Mc(F3)))

[0106] =σ(W1(F cat ))

[0107] Wherein, F c avg and F c max are adaptive average pooling and adaptive maximum pooling operations respectively, MLP represents calculation using a multi-layer perceptron, F1 and F2 are the feature mapping of the upper layer and the feature mapping of the current layer, F3 is the feature mapping after feature fusion, σ represents a sigmoid activation function, and W0 and W1 are weights of the multi-layer perceptron, which are shared for the two inputs F1 and F2.

[0108] In this way, the detection accuracy can be further improved in combination with the multi-scale alignment algorithm.

[0109] According to some embodiments of the present application, optionally, after updating the ship plate detection positioning model by the non-local neighborhood calculation alignment algorithm and the channel fusion enhancement algorithm, in the step of training and parameter adjusting the updated ship plate detection positioning model, the optimized algorithm is used to learn and extract three different size features to adapt to different size targets, and the inference of target recognition and positioning is performed through the detection head of the model. From the regression vector of the detection head, the offset t x , t y , and the scaling value t w , t h of the frame scale of the anchor frame relative to the upper left corner coordinate position can be obtained. The calculation formula is as follows:

[0110] b x =2σ(t x )-0.5+C x

[0111] b y = 2σ(t y )-0.5 + C y

[0112]

[0113]

[0114] where p w ,p h are the width and height of the prior box, b x ,b y are the coordinates of the center point of the predicted box, and b w ,b h are the width and height of the predicted box. Thus, by the above formula, the predicted box can be calculated using the four values t x ,t y ,t w ,t h , as shown in FIG. 32. Figure 6

[0115] At this time, a large number of predicted boxes will be obtained, but not every predicted box is valid. Therefore, scoring and non-maximum suppression screening are required. Scoring screening is to screen the predicted box whose score meets the confidence, and non-maximum suppression is to screen the box with the largest score in a certain area belonging to the same category. Confidence represents the confidence that the predicted target box contains the target and the accuracy of the prediction of this target box, and the formula is as follows:

[0116]

[0117] where the first term takes 1 if the target center falls in a grid, otherwise it takes 0. The second term is the IoU value between the predicted target box and the actual label.

[0118] During the training process, the model needs to be punished to force the model to learn in the desired direction. Therefore, a special loss function is designed to correct the prediction bias. The loss function includes three categories: classification loss, positioning loss, and confidence loss. The total loss is the weighted sum of the three, and the attention of the model to the three is adjusted by the weight value.

[0119] In the classification loss, the binary cross-entropy loss is calculated for each category, and the formula is as follows:

[0120]

[0121]

[0122] In the above formula, N represents the total number of categories, x​i y is the predicted value of the current class i y is the probability of the current class after the activation of the number of rows * i L is the true value of the current class (0 or 1) class The classification loss is calculated.

[0123] In the calculation of the bounding box loss, if the squared difference loss is simply used to calculate the distance between the predicted coordinates and the length and width of the predicted box and the true value, some problems will be caused. Since the bounding box prediction needs to focus on the overlapping area of the box and the true box, and the ratio of the area of the overlapping area to the area of the union of the two is better, but the use of squared loss cannot well measure this. Therefore, we use the IoU-based loss calculation method.

[0124] The intersection over union (IoU) is an important concept in object detection, which is the ratio of the intersection of the target box and the real box to the union, as shown in Figure 7 The ratio of the intersection of the two blocks to their union is the intersection over union.

[0125] The formula of the intersection over union is as follows:

[0126]

[0127] We use CIoU as an upgraded version of IoU to make the predicted box converge faster to the center point of the real box, and consider the proportion of the long box, and add a balance factor to balance the loss caused by the proportion of length and width and the loss caused by IoU. Its formula is as follows:

[0128]

[0129]

[0130]

[0131]

[0132]

[0133]

[0134]

[0135]

[0136]

[0137] The confidence loss we use is the cross-entropy loss. The formula is as follows:

[0138]

[0139] wherein, o i represents the IoU value of the predicted target bounding box and the real target bounding box, c is the predicted value, is the predicted confidence of c through the Sigmoid activation function.

[0140] Therefore, the loss function of the present application is:

[0141] Loss total = λ1Loss loc + λ2Loss cla + λ3Loss conf

[0142] According to some embodiments of the present application, optionally, the training and testing data sets are obtained by labeling with a picture labeling tool, and the data sets include various types, sizes, angles of fishing boats, and different clarity, different fonts, printed or handwritten ship plates.

[0143] In this way, by using the unique ship picture data and using the picture labeling tool to obtain the labeling, various interferences encountered in the ship plate detection task can be coped with, thereby improving the detection accuracy,

[0144] According to some embodiments of the present application, optionally, various target detection models can be updated and optimized through a non-local neighborhood calculation alignment algorithm and a channel fusion enhancement algorithm.

[0145] In this way, the present mainstream various target detection models can be easily inserted, and the user can independently select the required target detection model according to the use requirements.

[0146] In the second aspect, the present application further provides a storage medium, the storage medium stores a computer program, and the computer program is executed by a processor to realize the method steps of the first aspect of the present application.

[0147] In the third aspect, please refer to Figure 2 The present application further provides an electronic device 10, which comprises a processor 101 and a storage medium 102, the storage medium 102 is the storage medium of the second aspect; the processor 101 is used to execute the computer program stored in the storage medium 102 to realize the method steps of the first aspect.

[0148] Different from the prior art, the technical scheme of the application aligns the multi-level features in the feature pyramid through a multi-scale non-local algorithm, so as to improve the robustness of the algorithm and reduce the calculation amount. Meanwhile, in order to make up for the lack of correlation between channels in the non-local algorithm, the channel features between the feature maps before and after alignment are fused, so as to further improve the detection accuracy of the algorithm.

[0149] Those skilled in the art should understand that each of the above embodiments can be provided as a method, device or computer program product. These embodiments can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Each of the above embodiments involves all or part of the steps in the method, which can be completed by instructing the relevant hardware by a program. The program can be stored in a storage medium readable by a computer device, and is used to execute all or part of the steps of the above embodiments. The computer device includes but is not limited to: personal computer, server, general-purpose computer, special-purpose computer, network device, embedded device, programmable device, smart mobile terminal, smart home device, wearable smart device, vehicle-mounted smart device, etc.; the storage medium includes but is not limited to: RAM, ROM, magnetic disk, magnetic tape, optical disk, flash memory, U disk, mobile hard disk, memory card, memory stick, network server storage, network cloud storage, etc.

[0150] The above embodiments are described with reference to flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be realized by computer program instructions. These computer program instructions can be provided to the processor of the computer device to produce a machine, so that the instructions executed by the processor of the computer device produce a process for implementing the specified function in each flow or multiple flows and / or blocks. Figure 1 The device for implementing the function specified in each flow or multiple flows and / or blocks. Figure 1 The device for implementing the function specified in each flow or multiple flows and / or blocks.

[0151] These computer program instructions can also be stored in the computer device readable storage medium, which can guide the computer device to work in a specific way, so that the instructions stored in the computer device readable storage medium produce a product including instruction devices, which implement the specified function in each flow or multiple flows and / or blocks. Figure 1 The device for implementing the function specified in each flow or multiple flows and / or blocks. Figure 1 The device for implementing the function specified in each flow or multiple flows and / or blocks.

[0152] These computer program instructions can also be loaded into the computer device, so that a series of operation steps are executed on the computer device to produce a computer implemented process, so that the instructions executed on the computer device provide a process for implementing the specified function in each flow or multiple flows and / or blocks. Figure 1one or more processes and / or blocks Figure 1 steps of the functions specified in the one or more blocks.

[0153] It should be noted that, although the above-mentioned embodiments have been described herein, the patent protection scope of the present application is not limited thereby. Therefore, based on the innovative idea of the present application, the changes and modifications made to the embodiments described herein, or the equivalent structures or equivalent process transformations made using the content of the present application specification and drawings, directly or indirectly apply the above technical solutions to other related technical fields, are all included in the patent protection scope of the present application.

Claims

1. A method for detecting and locating ship license plates, characterized in that, Includes the following steps: Cameras are installed at the port and dock, and the cameras are connected to a server to transmit the images of ships they capture to the server. The images of ships taken at the port and docks are labeled, and the ship license plates appearing in the images are marked. This data will be used as a training and testing dataset for future reference. A vessel license plate detection and localization model is established. The local and global features of the vessel image are aligned using a deformable convolution alignment algorithm and a non-local neighborhood calculation alignment algorithm, respectively. Then, a channel fusion enhancement algorithm is added to update the vessel license plate detection and localization model. Using the above dataset, the updated boat license plate detection and localization model is trained and its parameters are tuned to obtain the trained boat license plate detection and localization model. The surveillance video stream is captured by cameras installed at the port terminal. The surveillance video stream is then input into the trained vessel license plate detection and positioning model to detect the vessel license plates on the port terminal and calculate the positioning coordinates of the vessel license plates appearing in the video stream. By calculating the vessel license plate's location coordinates, the position of the vessel license plate in the image signal currently captured by the camera is selected and displayed in real time. The nonlocal neighborhood computation alignment algorithm specifically uses a nonlocal algorithm to perform multi-scale feature alignment through global correlation calculation and feature pyramid. A neighborhood calculation algorithm is added to the nonlocal neighborhood calculation alignment algorithm. The calculation formula after adding the neighborhood calculation algorithm is as follows: Where δ(r) represents the neighborhood centered on the input index position, which determines the size of the neighborhood block in the image segmentation in the neighborhood calculation. The calculation formula after adding the neighborhood calculation algorithm changes the original calculation of a single pixel to the neighborhood calculation centered on the index pixel. The channel fusion enhancement algorithm specifically involves performing adaptive average pooling and max pooling with constant ratios on feature maps of different scales, and then fusing them to aggregate the original feature map spatial information and the fused feature map information.

2. The method for detecting and locating ship license plates according to claim 1, characterized in that, The calculation formula for the nonlocal neighborhood alignment algorithm is as follows: Where i is the position index on the feature map that needs to be aligned, j is the position index on the feature map that serves as the alignment scale, and z... j It is the feature mapping matrix of the feature maps aligned to the scale. The function φ calculates a long-range dependency between positions i and j on the two feature maps, and the function θ is a feature mapping between z and j. j A mapping of the feature map, the new representation z j Feature mapping, output response y i By summing the explicit measurements of the two feature maps, the aligned features at the two scales are obtained, and the data is normalized using the scalar function C(x,z).

3. The method for detecting and locating ship license plates according to claim 1, characterized in that, After aligning global features using a nonlocal algorithm, a deformable convolution alignment algorithm with a local receptive field is used to further supplement the initially aligned features.

4. The method for detecting and locating ship license plates according to claim 3, characterized in that, The calculation formula for the deformable convolution alignment algorithm is as follows: in, It is the aligned feature map, Δ i It is the offset between two feature maps, T i These are feature maps that need to be aligned. It is a feature map used as a reference for alignment standards. yes and T i Cascade of two feature maps, f align It uses the offset Δ i For feature maps The alignment function, f, performs alignment. offset It is a function that calculates the offset between two feature maps, both of which are implemented using deformable convolution.

5. The method for detecting and locating ship license plates according to claim 1, characterized in that, The calculation formula for the channel fusion enhancement algorithm is as follows: Among them, F c avg and F c max These are adaptive average pooling and adaptive max pooling operations, respectively. MLP stands for Multilayer Perceptron for computation. F1 and F2 are the feature maps of the upper and current layers, respectively. F3 is the feature map after feature fusion. σ represents the sigmoid activation function. W0 and W1 are the weights of the Multilayer Perceptron, which are shared for the two inputs F1 and F2.

6. A storage medium, characterized in that, The storage medium stores a computer program that, when executed by a processor, implements the steps of the method as described in any one of claims 1 to 5.

7. An electronic device, characterized in that, Includes a processor and a storage medium, wherein the storage medium is the storage medium as described in claim 6; The processor is used to execute a computer program stored in the storage medium to implement the steps of the method as described in any one of claims 1 to 5.

Images