Container quantification reference processing and damage positioning method, system and equipment based on image segmentation, and medium

Abstract

The invention provides a container quantitative reference processing and damage positioning method, system and device based on image segmentation and a medium, which are applied to the technical field of wharf container detection, and are characterized in that each key part, such as a corner fitting, a top surface, a side surface and the like, of a container is identified through image segmentation, and relative positions and spatial layouts of the parts are determined; according to the method, a standard size and a segmentation result of the container are combined to construct a calculation reference of each part of the container in the image, so that through the constructed calculation reference, key information of states such as attitude and position relation of the container in the image is reflected, and a standardized size reference can be established through a known actual size; subsequently, basic conditions required for quantification can be provided for container damage positioning and size quantification calculation based on the calculation benchmark.

Description

Technical Field

[0001] The present application relates to the technical field of damage detection of container at a terminal, and in particular to a method, system, device and medium for quantitative benchmark processing and damage positioning of containers based on image segmentation. Background Art

[0002] Containers are susceptible to various forms of damage during long-distance transportation, loading and unloading, and use, such as deformation or damage to the door and body structure. Therefore, accurate identification and quantification of damaged parts of containers is an important part of ensuring the safe use of containers.

[0003] In recent years, computer vision detection technology, especially image segmentation and target detection methods in deep learning, has gradually made significant progress in the field of automated detection. These solutions generally improve recognition effects by segmenting images and effectively distinguishing targets from backgrounds, thereby accurately identifying various types of targets in images.

[0004] Regarding the detection and identification of container damage, existing solutions have also begun to use computer vision detection technology, such as CN112819793A, CN115187535A, CN115222697A, etc. Although these technical solutions are based on image processing and deep learning for container damage detection, these technical solutions use models to directly predict the category and location information of container damage from two-dimensional images, so the container damage identification error in actual application is relatively large, and when the size of the damage defect needs to be quantified later, these solutions cannot provide useful quantitative basic data. Other existing technologies, such as CN110992337A and CN115965885A, etc., provide methods for detecting container damage based on images and point cloud data. Although they have initially achieved relatively accurate identification and quantification of damage, they need to rely on laser radar scanning to obtain point cloud data, which increases the hardware and computing costs, maintenance difficulty and system complexity of the container damage detection system, and it is difficult to provide laser radar related deployment conditions in most container application scenarios. Summary of the invention

[0005] In view of this, the present invention provides a method, system, device and medium for container quantitative benchmark processing and damage positioning based on image segmentation, which identifies the key parts of the container (such as corner pieces, top surface, side surface, etc.) through image segmentation, and determines the relative position and spatial layout between these key parts, constructs a container working state model, and then combines the standard size of the container and the segmentation result to construct the size calculation benchmark of each area of ​​the container, and obtains the state model corresponding to the container in the actual scene. Because the state model not only reflects the direction and position relationship of the container in the image, but also establishes a standardized benchmark through the known actual size, which can be used for subsequent damage size quantification calculation.

[0006] The present invention provides the following technical solutions: The present invention provides a container quantitative benchmark processing method based on image segmentation, comprising: Based on the pre-trained segmentation model, each component of the container in the two-dimensional image of the container is segmented and identified to form a segmentation result, wherein the component includes one or more of the following container parts that can reflect the posture and position of the container: front door, rear front, top surface, side surface, top corner fittings, side corner fittings, and slings; According to the corner fittings positions, segmentation results and known combination relationships between anchors, the container placement is inferred based on the segmentation results. The state model of each placement is distinguished by two parts: the identification sequence and the corner fittings features. According to the standard size of the standard container type corresponding to the placement posture, a calculation basis for each component of the container in the two-dimensional container image is constructed, wherein the standard container type is the standard container corresponding to the container in the two-dimensional container image.

[0007] Compared with the prior art, the beneficial effects that can be achieved by at least one of the above technical solutions adopted by the present invention include at least: 1. The present invention uses instance segmentation technology to accurately segment and identify the key parts of the container, including the door, front, top, side, and corner fittings, and combines the positions of these parts with the relative relationship between the corner fittings to infer the placement state (i.e., the working condition) of the container based on the image; 2. By combining the standard size of the container with the segmentation results, a calculation benchmark (such as a geometric model) is established for each part of the container in the image, the damaged location is preliminarily determined, and the image coordinates are mapped to the actual size coordinates. This geometric model provides a geometric benchmark for the subsequent damage size quantification, making the subsequent damage size calculation more accurate and standardized; 3. The present invention realizes container status modeling and damage location only through two-dimensional images and image segmentation technology, avoiding expensive lidar equipment and complex data processing procedures, and reducing the hardware cost, maintenance difficulty and overall complexity of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0009] Figure 1 It is an overall schematic diagram of the container state modeling solution based on image segmentation in this application; Figure 2 is a flow chart of a container quantization benchmark processing method based on image segmentation in this application; Figure 3 It is a flow chart of the container damage positioning method based on image segmentation in this application; Figure 4 It is a flow chart of the container state modeling and damage location method based on image segmentation in this application; Figure 5 It is a schematic diagram of the deployment of cameras at collection points in this application; Figure 6a , 6b is a schematic diagram of the result of container image acquisition in this application; Figure 7a , 7b It is a schematic diagram of the result of container image segmentation in this application; Figure 8 It is a flow chart of the method for inferring the actual placement posture of the container in this application; Figure 9a , 9b 9c is a schematic diagram of the working conditions of the box top door angle and the box side in this application; Fig.10 It is a flow chart of the method for constructing the container geometric model and dimension benchmark in this application; Fig.11 is a schematic diagram of perspective transformation of a quadrilateral affected by perspective effect into a standard-sized rectangle in this application; Fig.12 It is a schematic diagram of a perspective matrix obtained by perspective transformation of different parts of a container in this application; Fig.13 It is a schematic diagram of the container positioning dimension diagram in this application; Fig.14It is a structural diagram of a container quantization benchmark processing system based on image segmentation in this application; Fig.15 It is a structural schematic diagram of a container damage positioning system based on image segmentation in this application; Fig.16 It is a schematic diagram of the structure of the electronic device in this application. DETAILED DESCRIPTION

[0010] The embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0011] The following describes the implementation methods of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The present application can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that, in the absence of conflict, the following embodiments and the features in the embodiments can be combined with each other. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work belong to the scope of protection of the present application.

[0012] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein can be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present application, it should be understood by those skilled in the art that an aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number and aspect described herein can be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein can be used to implement this device and / or practice this method.

[0013] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present application. The drawings only show components related to the present application rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0014] Additionally, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, it will be understood by those skilled in the art that the examples can be practiced without these specific details.

[0015] Regarding the existing container detection solutions, such as the prior art presented in the above background technology, through analyzing the problems and exploring solutions, it is found that: On the one hand, the existing solutions are only based on images. Due to perspective distortion and occlusion in the presentation of containers in images under different shooting angles, the spatial placement of the containers (i.e., specific working conditions) is not considered, which leads to errors in the position judgment of the damage frame, making it impossible to accurately determine the damage and lacking a geometric basis for damage quantification. Furthermore, image-based damage recognition is limited to predicting the type and approximate location of the damage, without considering the segmentation and positioning of various components of the container (such as corner fittings, doors, top surfaces, sides, etc.), and cannot provide a geometric basis for the subsequent quantification of damage sizes.

[0016] On the other hand, although some existing solutions can initially achieve quantitative work of container damage detection through images and point cloud data, they rely on the use of lidar scanning to obtain point cloud data, which requires a significant increase in the system's hardware and computing costs, maintenance difficulty, and system complexity. In addition, most application scenarios do not have the conditions for deploying scanning radars.

[0017] In view of this, the present invention proposes a container state modeling and damage location solution based on image segmentation: Figure 1 As shown in the figure, the key parts of the container, such as corner pieces, top surface, side surface, etc., are identified through image segmentation, and the relative positions and spatial layout of these parts are determined. Combined with the standard size of the container and the segmentation results, a geometric model of the container in the image is constructed. The constructed geometric model not only reflects the key information such as the posture and position relationship of the container in the image, but also can establish a standardized benchmark through the known actual size, so that based on the benchmark, it can provide basic data in the subsequent quantitative calculation of the damaged size.

[0018] Therefore, for container inspection, although the spatial position of the container is uncertain in actual applications, corresponding inspections are performed after considering the state of the container. That is, image segmentation technology can be used to identify key components of the container, such as corner fittings, doors, sides, top surfaces, etc., and then determine the relative positions and spatial distribution of these parts. Finally, a container state model is established to facilitate the subsequent preliminary positioning of the damage location, and also lay the foundation for the subsequent quantification of the damage size.

[0019] The technical solutions provided by various embodiments of the present application are described below in conjunction with the accompanying drawings.

[0020] like Figure 2 As shown, the present invention provides a container quantitative benchmark processing method based on image segmentation, which may include: Step S202: Segment and identify various components of the container in the two-dimensional image of the container based on the pre-trained segmentation model to form a segmentation result.

[0021] It should be noted that the segmented components may be parts that can reflect the posture, position, etc. of the container, for example, one or more parts of the container door, front, top, side, corner top, corner side, sling, etc.

[0022] The container image is pre-segmented through the segmentation model to obtain the segmentation results corresponding to each component of the container, so that these segmentation results can be used to build the model later.

[0023] It should be noted that the two-dimensional image may be a container image provided by a collection device. For a schematic diagram of the collection method and results, please refer to the examples below.

[0024] Step S204: infer the placement posture of the container according to the segmentation result based on the corner piece position, the segmentation result and the known combination relationship between the anchors.

[0025] As analyzed above, the existing solution only processes the 2D images of containers, but the results are not ideal because the placement of the containers is not considered. In addition, the 2D images collected in the application scenario are limited by the site, the collection angle of the shooting equipment, etc., so the various sides of the container in the image have different proportions. Under these factors, the collected images cannot be directly used for quantification.

[0026] Therefore, the present invention adds data related to the placement posture (i.e., specific working conditions) during processing. Specifically, it is a segmentation result obtained by combining prior knowledge (such as corner fitting positions, anchors, etc.) with actual segmentation. Since there is a combination relationship between these data on the container as a whole, these relationships can be used to determine the placement posture, which facilitates the addition of these placement postures to subsequent models.

[0027] The state model of each placement posture can use two parts, an identification sequence and corner piece features, as distinguishing marks, where the identification sequence can be a string (a pure numeric string, a non-numeric string, or a string consisting of numbers and non-numeric characters) to serve as a unique identification sequence; if the corner piece features are the same under the same sequence, they are used as the identification of a state model; and if the sequences are the same but the corner piece features are different, they are used as the identification of two different state models.

[0028] Step S206: construct a calculation basis for each component of the container in the two-dimensional container image according to the standard size of the corresponding standard container type under the placement posture. It should be noted that the standard container type here refers to the standard container corresponding to the container in the two-dimensional container image. For example, if the container type in the image is a dry cargo container, the standard container type can be the corresponding container type of the dry cargo container.

[0029] By combining the placement posture with the standard box size, the corresponding geometric model and calculation benchmark (i.e., the new scale) are established for each component segmented from the two-dimensional image of the container actually taken. Therefore, based on the geometric model, it is possible to determine which box surface in the corresponding component is damaged, and for the damage that occurs, relevant basic data for dimensional quantification can be provided based on the benchmark. That is, the positioning model can be used to obtain quantitative data such as the presence or absence of damage, location, size, etc.

[0030] During implementation, when the placement posture is known, if the container surface is not damaged, the corresponding geometric model should be the same as the geometric model of the surface of the standard-sized box (or the difference is very small); otherwise, there should be a difference, so that whether there is damage can be determined based on the difference. Similarly, the purpose of the benchmark is similar. When there is damage, the specific location of the damage and the size of the damage can be determined based on the geometric model and the benchmark, thereby providing basic quantitative conditions for the quantification work.

[0031] In summary, by incorporating the placement posture into the positioning model, even the two-dimensional images of the container obtained from different shooting angles, which previously showed problems such as perspective distortion and occlusion in the image, can reduce the impact of these problems and improve the ability to provide quantitative conditions, which is very beneficial for determining the position of the damaged frame and reducing errors. It also enables the positioning model to have the basic conditions for providing quantitative data of the damage, making up for the deficiency of the existing solutions that lack quantitative basic data.

[0032] In some embodiments, in determining the placement posture, the number of anchors can be used as the basis for the first judgment. Specifically, the number of anchors is first determined, and then the number of anchors is combined in a preset order as an identification sequence, and then different postures can correspond to the identification sequence.

[0033] For the case of having the same identification sequence, the spacing between the corner pieces and / or the inclination of the center line are further used to distinguish them into different placement postures.

[0034] It should be noted that in other implementations, the corner pieces (such as spacing, inclination of the center line, etc.) may be used as the first judgment to make an inference, which will not be elaborated here.

[0035] In some implementations, when constructing the geometric model and the benchmark, a mapping relationship between coordinates may be constructed to form data corresponding to the geometric model and the benchmark.

[0036] For example, first use corner point detection to find the coordinates of each vertex of the box door, box front, top surface and side surfaces that appear in the image, sort their coordinates counterclockwise starting from the upper left corner, and combine the positions of the corner pieces to determine their relative positions, and obtain the coordinates and corresponding positions of the box door, box front, top surface and side surfaces, and then establish a mapping relationship from the coordinates in the image to the coordinates of the actual size based on the standard sizes of the box door, box front and corner pieces.

[0037] After the mapping relationship is established, the geometric model and benchmark of each component can be determined based on the mapping data.

[0038] In some implementations, after taking the placement posture into consideration, a more accurate geometric model and benchmark can be obtained based on the perspective relationship in image processing, so as to further improve the effect of image distortion and occlusion caused by the shooting angle on the performance.

[0039] The mapping relationship can be constructed as follows:

[0040] in, , are the coordinates of the midpoint of the image, , is the standard size value, H is the perspective transformation matrix between two faces, i=1,2,3,4.

[0041] After considering the placement posture, the mapping relationship between the coordinates of the midpoint of the image and the standard size value can be completed through the perspective transformation matrix. Among them, the perspective transformation matrix can be a priori knowledge (a priori data) or a posteriori knowledge (a posteriori data) determined according to actual conditions, and is not specifically limited here.

[0042] In some implementations, non-quantitative targets may be eliminated first.

[0043] For example, based on the segmentation results and the segmentation position of the spreader, the recognition results of non-quantitative targets in the image are first eliminated, which makes it easier to identify the anchors and their quantity.

[0044] In some embodiments, the segmentation model of the present invention may be a trained model, that is, the model may be an existing container segmentation model, or a segmentation model obtained through the training scheme provided below in the present invention.

[0045] During implementation, the pre-trained segmentation model is trained through the following steps: constructing a data set required for instance segmentation based on a container two-dimensional image data set, and training a preset segmentation model based on the data set; wherein the data set includes annotation data for annotating key parts of the container in the container two-dimensional image.

[0046] The labeled data with key parts annotated constitute the instance segmentation training data set, so that the trained segmentation model is more efficient and accurate in segmenting the various components of the container.

[0047] It should be noted that the two-dimensional container image used in the present invention can be container two-dimensional image data from an existing database, or it can be a container two-dimensional image collected by a camera device installed at various positions of the gantry crane at the wharf and shooting at different angles.

[0048] Based on the same inventive concept, the present application also provides a container damage positioning method based on image segmentation, that is, after the geometric model and size reference of the container are established in the image, the initial position of the damage is positioned and detected.

[0049] refer to Figure 3 Schematically, a container damage location method based on image segmentation includes: Step 402: construct a calculation benchmark for each component of the container in the image. The container quantization benchmark processing method based on image segmentation described in any example of this application can be used to obtain the geometric model and size benchmark of each component of the container in the image to form the calculation benchmark required for quantization.

[0050] Step 404: Determine the location of the damaged container based on the calculation reference established in the image.

[0051] By establishing the geometric model and size benchmark of the container in the image, the calculation benchmark data required for the quantitative basis can be obtained, and the actual state and actual proportion of the container in the image can be obtained, so that the position of the damage on the container can be quantified, thereby improving the accuracy of damage positioning.

[0052] In some embodiments, in terms of using a geometric model to determine which container surface is damaged, or even to determine a preliminary location of the damage, the following steps may be included: first determine whether the damage is located on a certain container surface; then, based on the standard dimensions of each part of the container and the relative position of the damage, use a perspective transformation to map the damage coordinates to the actual space, thereby completing a preliminary determination of the location of the container damage.

[0053] The damage coordinates are mapped to the actual space through the standard size of each part of the container and the relative position of the damage, which is conducive to improving the accuracy of the quantitative data.

[0054] In some implementations, with the aid of damage box annotation, the damage situation can be determined more effectively. Therefore, when determining whether the damage is located on a certain container surface, the following determination method can also be used: first obtain the coordinates of the four vertices of each part of the container from the segmentation result, and then combine the center coordinates of the damage box and use the point-in-polygon algorithm to preliminarily determine whether the damage is located on a certain container surface.

[0055] Another example is given below, which is a schematic illustration formed by combining the above-mentioned examples.

[0056] refer to Figure 4 In the method for modeling a damage location model based on container status based on image segmentation proposed in the present invention, the method mainly includes the following steps: Step 1: Collect 2D images of the container using cameras at different angles and positions of the bridge crane, annotate key parts, construct an instance segmentation dataset, and train the instance segmentation model.

[0057] The data set in step 1 can be a data set with annotated components (i.e., combined parts), which can include the box door, box front, top surface, side surface, corner top surface, corner side surface, and hanger. Therefore, based on the annotated data set, the instance segmentation model can be trained with annotated features.

[0058] In addition, you can refer to the container 2D image acquisition example Figure 5 Illustration: Due to the actual scene conditions, the schematic diagram of the camera deployment at the collection point. Cameras A and B can collect images from both ends of the container, and camera C can collect images from the side of the container. That is, cameras A and B for video can be deployed on the bridge crane in the direction of the container's travel, and camera C can be deployed on the side space of the scene. At this time, cameras A and B collect images of the top surface of the container from the upper space, while camera C on the side collects images of the side of the container. The collection results can be seen in Figure 6a , 6b It should be noted that those skilled in the art should understand that Figure 6a , 6b The mosaic area on the container is used to cover the original commercial logo of the container, which does not affect the understanding of the present application and is not sufficient to constitute a relevant limitation of the present application. Similar illustrations involved in the following text will not be explained one by one.

[0059] Step 2: Based on the trained instance segmentation model, segment and recognize the container 2D image; the collected 2D container image is segmented to form the segmentation results of each key part of the container (such as corner fittings, box surface, etc.). Figure 7a, for the two-dimensional image of the container collected from the bridge crane camera A or B, segmentation can be performed to obtain the top container surface segmentation result (see the wireframe area where the top surface is located), the corner fitting segmentation result (see the white dot area where the top surface corner fitting is located), the container door surface segmentation result (see the wireframe area where the container door surface is located), etc.; referring to 7b, for the two-dimensional image of the container collected from the side camera C, segmentation can be performed to obtain the side container surface segmentation result (see the wireframe area where the side surface is located), the corner fitting segmentation result (see the white dot area where the top surface corner fitting is located), etc.

[0060] Step 3: Infer the placement of the container (i.e., the spatial state of the container, also refers to the specific working condition of the container) based on the position of the corner fittings, the segmentation results, and the combination of the anchors. It should be noted that the anchors can be used as one of the key parts of the container, so the spatial state of the container can be quickly and accurately inferred based on the known combination of corner fittings, box surfaces, anchors and other components.

[0061] refer to Figure 8 As shown in the figure, in the reasoning of the container state (i.e., placement posture), the recognition results of non-quantitative targets in the image are first eliminated based on the segmentation recognition results and the segmentation position of the spreader, and then the number of each part, i.e., the anchors, is determined, and their numbers are combined in a certain order into an identification sequence. For cases with the same identification sequence, the spacing of the corner pieces and the inclination of the center line are used to further distinguish them, that is, when the sequences are the same, the corner pieces are further combined for secondary classification. Therefore, by making the state model of each working condition include the two parts of the identification sequence and the corner piece feature, the state models under different working conditions can be further distinguished.

[0062] Specifically, the result of the box image segmentation is subjected to corner point detection to obtain the vertex coordinates of the segmented box surface, and the center of the corner piece surface is taken as the center of the segmentation result to obtain the corner piece center coordinates. Figure 8 As shown, the recognition results of non-quantitative targets in the image are eliminated based on the results of segmentation and recognition and the segmentation position of the spreader, and then the number of each part, i.e., the anchors, is determined, and their numbers are combined in a certain order into an identification sequence. For the cases with the same identification sequence, the spacing of the corner pieces and the inclination of the center line are used to further distinguish them. That is, the state model of each working condition includes two parts: the identification sequence and the corner piece characteristics.

[0063] As attached Figure 9a The figure shows the state diagram corresponding to the working condition named "top-door", where ABCD are the four vertices of the box door, EFHG are the vertices of the box top segmentation area, and ab is the center of the corner piece. The corresponding working condition sequence can be determined as 101020. This sequence of the working condition is unique among all states, and its spatial state can be uniquely determined.

[0064] As attached Figure 9b , 9c Shown are schematic diagrams of the working conditions of "side2" and "side2-1", respectively. The sequences of the two working conditions are both 000102, which are distinguished based on the position characteristics of the corner fittings, specifically by the inclination of the lines ab and a´b´ connecting the centers of the corner fittings.

[0065] Step 4: Construct a calculation benchmark for each part of the container in the image based on the standard dimensions of different container types; refer to Fig.10 and Fig.11 As shown in the figure, in the process of building the geometric model and benchmark of the container, the corner point detection is first used to find the coordinates of each vertex of the door, front, top and side in the image, and the coordinates are sorted counterclockwise from the upper left corner. Combined with the position of the corner fittings, their relative positions are determined, and the coordinates of the door, front, top and side in the image and their positions are obtained. Then, according to the standard sizes of the door, front and corner fittings, the perspective mapping relationship from the coordinates in the image to the actual size coordinates is established, and the benchmarks of each part are built. The mapping relationship can be expressed as:

[0066] in, , are the coordinates of the midpoint of the image, , is the standard size value, H is the perspective transformation matrix between two faces, i=1,2,3,4.

[0067] like Fig.11 As shown, by combining the standard sizes of different box types, the perspective mapping relationship between the coordinates of the box door, box front, side and top in the image and the actual size coordinates is established respectively, where points 1, 2, 3, and 4 are the four vertices of the box door, box front, side or top in the image, points 1', 2', 3', and 4' are the vertices of the transformed standard rectangle, and L and D are their standard size values. Specifically, for the box door or box front, L is the box width of the box type, in mm, D is the box height H in mm, for the box top, L is the box width of the top surface, in mm, D is the length value of the actual segmented part of the top surface along the box length direction, in mm, for the side, L is the length value of the actual segmented part of the side along the box length direction, in mm, D is the box height of the box type, in mm, and H is the transformation matrix.

[0068] It should be noted that the schematic diagram of perspective transformation from quadrilaterals with different parts affected by perspective effect to standard-sized rectangles can be referred to Fig.12Illustration: (a) For the front or door of the box, the corresponding transformation matrix H1 is obtained through the above transformation; (b) For the top surface of the box, the corresponding transformation matrix H2 is obtained through the above transformation; (c) For the side surface of the box, the corresponding transformation matrix H3 is obtained through the above transformation.

[0069] As attached Fig.13 The figure shows the positioning dimension diagram of the container, where C1 and C2 are structural dimensions related to corner fittings, H, L, W are the external height, length and width of the container, which are related to the container type, and P and S are the center distances of the corner fitting holes along the width and length of the container.

[0070] As attached Figure 9a The working condition shown includes two parts: the box door and the box top. The calculation basis is the perspective transformation matrix within its surface. The vertex ABCD of the box door corresponds to an ordered list of image coordinates. The box corresponds to a box door of standard size. Suppose its actual height is H mm and width is W mm. The coordinates of ABCD after transformation can be set to [[0, H], [0, 0], [W, 0], [W, H]]. The transformation matrix within its surface can be obtained from the coordinates before and after the transformation. For the box top, the actual length of its EF and HG sides is estimated. The estimation method is that the size of aF and bG is a standard length determined by the box type. The projection length on EF and HG is used as the unit length of EF and HG. The average of the actual lengths of EF and HG is calculated and set to L. The width is W. Then the transformed coordinates of EFGH are [[0, L], [0, 0], [W, 0], [W, L]]. The transformation matrix within its surface can be obtained from the coordinates before and after the transformation.

[0071] Step 5: Preliminarily determine the location of the container damage based on the benchmarks established in the image, in preparation for quantifying the size of the damage; In the determination of damage, after obtaining the coordinates of the four vertices of each part of the container through instance segmentation, the coordinates are sorted counterclockwise from the upper left corner, and combined with the center coordinates of the damage box, the point-in-polygon algorithm is used to preliminarily determine whether the damage is located on a certain container surface. Then, based on the standard size of each part of the container and the relative position of the damage, the damage coordinates are mapped to the actual space using perspective transformation to complete the preliminary judgment of the damage position of the container.

[0072] Based on the same inventive concept, the present application also provides a container quantitative benchmark processing system based on image segmentation, and a container quantitative benchmark processing system based on image segmentation formed based on the above-mentioned various method examples to automatically, real-time and accurately establish a damage positioning model.

[0073] refer to Fig.14 Schematically, a container quantitative benchmark processing system 100 based on image segmentation may include: Segmentation model 101: segmenting and identifying various components of the container in the two-dimensional image of the container based on the pre-trained segmentation model to form a segmentation result; Reasoning model 103: inferring the placement posture of the container according to the segmentation result based on the corner piece position, the segmentation result and the known combination relationship between the anchors; Quantitative benchmark model 105: constructs a calculation benchmark for each component of the container in the two-dimensional image of the container according to the standard size of the corresponding standard container type under the placement posture.

[0074] It should be noted that the setting of the unit module functions and the number of modules in the container quantitative benchmark processing system based on image segmentation can be set accordingly according to the aforementioned method embodiment, and will not be further described here.

[0075] Based on the same inventive concept, the present application also provides a container damage positioning system based on image segmentation.

[0076] refer to Fig.15 Schematically, a container damage positioning system 300 based on image segmentation includes: State module 301: using the container quantization benchmark processing method based on image segmentation described in any example of the present application to establish a calculation benchmark for each component of the container in the image; Positioning module 303: Determine the location of the damaged container based on the geometric model and size reference established in the image.

[0077] It should be noted that the settings of the unit module functions and the number of modules in the container damage positioning system based on image segmentation can be set accordingly according to the aforementioned method embodiments, and will not be further described here.

[0078] Based on the same inventive concept, the present application also provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute: a container quantitative benchmark processing method based on image segmentation as described in any embodiment of the present application, or a container damage positioning method based on image segmentation as described in any embodiment of the present application.

[0079] like Fig.16 As shown, the present application also provides a schematic diagram of the structure of an electronic device, which shows the structure of the electronic device 500. The electronic device 500 is merely an example and should not limit the functions and scope of use of the embodiments of the present invention.

[0080] The electronic device 500 may include: at least one processor 510; and a memory 520 communicatively connected to the at least one processor; wherein the memory 520 stores instructions executable by the at least one processor 510, and the instructions are executed by the at least one processor 510 so that the at least one processor 510 can execute: the container quantitative benchmark processing method based on image segmentation as described in any embodiment of the present application, or the container damage positioning method based on image segmentation as described in any embodiment of the present application.

[0081] It should be noted that the electronic device 500 may be in the form of a general-purpose computing device, for example, it may be a server device.

[0082] In implementation, the components of the electronic device 500 may include but are not limited to: the above-mentioned at least one processor 510, the above-mentioned at least one memory 520, and a bus 530 connecting different system components (including the memory 520 and the processor 510), wherein the bus 530 may include a data bus, an address bus, and a control bus.

[0083] In implementation, the memory 520 may include a volatile memory, such as a random access memory (RAM) 5201 and / or a cache memory 5202 , and may further include a read-only memory (ROM) 5203 .

[0084] The memory 520 may also include a program tool 5205 having a set (at least one) of program modules 5204, such program modules 5204 including but not limited to: an operating system, one or more application programs, other program modules and program data, each of which or some combination may include the implementation of a network environment.

[0085] The processor 510 executes various functional applications and data processing by running computer programs stored in the memory 520 .

[0086] The electronic device 500 may also communicate with one or more external devices 540 (e.g., keyboards, pointing devices, etc.). Such communication may be performed via an input / output (I / O) interface 550. In addition, the electronic device 500 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 560, which communicates with other modules in the electronic device 500 via a bus 530. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 500, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID (disk array) systems, tape drives, and data backup storage systems, etc.

[0087] In this specification, the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the embodiments described later, the description is relatively simple, and the relevant parts can be referred to the partial description of the previous embodiments.

[0088] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A container quantitative benchmark processing method based on image segmentation, characterized in that: include: Based on the pre-trained segmentation model, each component of the container in the two-dimensional image of the container is segmented and identified to form a segmentation result, wherein the component includes one or more of the following container parts that can reflect the posture and position of the container: front door, rear front, top surface, side surface, top corner fittings, side corner fittings, and slings; According to the corner fittings positions, segmentation results and known combination relationships between anchors, the container placement is inferred based on the segmentation results. The state model of each placement is distinguished by two parts: the identification sequence and the corner fittings features. According to the standard size of the standard container type corresponding to the placement posture, a calculation basis for each component of the container in the two-dimensional container image is constructed, wherein the standard container type is the standard container corresponding to the container in the two-dimensional container image.

2. The container quantitative benchmark processing method based on image segmentation according to claim 1 is characterized in that: Inferring the placement posture of the container based on the segmentation result includes: determining the number of anchors, combining the number of anchors in a preset order as an identification sequence, and further distinguishing the placement posture by identifying the spacing of the corner pieces and / or the inclination of the center line for the same identification sequence; And / or, constructing the calculation reference of each part of the container includes the following steps: using corner point detection to find the coordinates of each vertex of the door, front, top and side of the container that appear in the image, sorting the coordinates in a counterclockwise order from the upper left corner, combining the position of the corner fittings, judging their relative positions, obtaining the coordinates and corresponding positions of the door, front, top and side, and then establishing a mapping relationship from the coordinates in the image to the actual size coordinates according to the standard sizes of the door, front and corner fittings; wherein the mapping relationship is constructed as follows: in, , are the coordinates of the midpoint of the image, , is the standard size value, H is the perspective transformation matrix between two faces, i=1,2,3,4; And / or, before determining the number of anchors, the recognition results of non-quantified targets in the image are eliminated based on the segmentation results and the segmented positions of the spreaders.

3. The container quantitative benchmark processing method based on image segmentation according to any one of claims 1-2, characterized in that: The pre-trained segmentation model is trained by the following steps: constructing a data set required for instance segmentation based on a container two-dimensional image data set, and training a preset segmentation model based on the data set; wherein the data set includes annotation data for annotating key parts of the container in the container two-dimensional image; And / or, the two-dimensional image of the container is a two-dimensional image of the container captured by a camera device at different angles at various positions of the bridge crane.

4. A container damage location method based on image segmentation, characterized in that: include: Using the container quantitative benchmark processing method based on image segmentation as claimed in any one of claims 1 to 3, the calculation benchmark of each component of the container in the image is obtained; Based on the calculated reference established in the image, the location of the container damage is determined.

5. The method for locating damaged containers based on image segmentation according to claim 4 is characterized in that: Determining the location of container damage includes: first determining whether the damage is located on a certain container surface; then, based on the standard size of each part of the container and the relative position of the damage, using perspective transformation to map the damage coordinates to the actual space, thereby completing a preliminary determination of the location of the container damage.

6. The method for locating damaged containers based on image segmentation according to claim 5, characterized in that: When judging whether the damage is located on a certain container surface, the coordinates of the four vertices of each part of the container are first obtained from the segmentation result, and then combined with the center coordinates of the damage box, the point-in-polygon algorithm is used to preliminarily judge whether the damage is located on a certain container surface.

7. A container quantitative benchmark processing system based on image segmentation, characterized in that: include: Segmentation model: Based on the pre-trained segmentation model, the various components of the container in the two-dimensional image of the container are segmented and identified to form a segmentation result. The components include one or more of the following container parts that can reflect the posture and position of the container: front door, rear front, top surface, side surface, top corner fittings, side corner fittings, and slings; Reasoning model: Based on the corner fittings position, segmentation results and known combination relationships between anchors, the container placement is inferred based on the segmentation results. The state model of each placement posture uses the identification sequence and corner fittings features as distinguishing marks. Quantitative benchmark model: constructing a calculation benchmark for each component of the container in the two-dimensional container image according to the standard size of the standard container type corresponding to the placement posture, wherein the standard container type is the standard container corresponding to the container in the two-dimensional container image.

8. A container damage positioning system based on image segmentation, characterized in that: include: State module: using the container quantitative benchmark processing method based on image segmentation as described in any one of claims 1 to 3 to establish a calculation benchmark for each component of the container in the image; Positioning module: Determines the location of damaged containers based on the calculated reference established in the image.

9. An electronic device, characterized in that: include: at least one processor; And, a memory communicatively connected to the at least one processor; wherein the memory stores instructions executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute: a container quantitative benchmark processing method based on image segmentation as described in any one of claims 1 to 3, or a container damage positioning method based on image segmentation as described in any one of claims 4 to 6.

10. A computer storage medium, characterized in that: The computer storage medium stores computer executable instructions, and when the computer executable instructions are executed by the processor, they perform: a container quantitative benchmark processing method based on image segmentation as described in any one of claims 1-3, or a container damage positioning method based on image segmentation as described in any one of claims 4-6.

Images