Container Quantification Benchmark Processing and Damage Location Method, System, Equipment and Medium Based on Image Segmentation

Abstract

The present application provides a method, system, device and medium for container quantization benchmark processing and damage location based on image segmentation, which is applied to the technical field of terminal container detection. Among them, each key part of the container, such as corner fittings, top surface, side surface, etc., is recognized through image segmentation, and the relative positions and spatial layouts of these parts are determined. Combining the standard dimensions of the container and the segmentation results, a calculation benchmark for each part of the container in the image is constructed. Thus, through the constructed calculation benchmark, not only the key information such as the attitude and position relationship of the container in the image is reflected, but also a standardized dimension benchmark can be established based on the known actual dimensions. Subsequently, based on the calculation benchmark, the basic conditions required for quantization can be provided for container damage location and dimension quantization calculation.

Description

Technical Field

[0001] The present application relates to the technical field of terminal container damage detection, and particularly relates to a container quantization benchmark processing and damage location method, system, device, and medium based on image segmentation. Background Art

[0002] Containers are prone to various forms of damage during long-distance transportation, loading and unloading, and use, such as deformation or damage to the container door and the container body structure. Therefore, accurately identifying and quantifying the damaged parts of the container is an important link to ensure the safe use of the container.

[0003] In recent years, computer vision detection technologies, especially image segmentation and object detection methods in deep learning, have gradually made remarkable progress in the field of automated detection. These solutions generally improve the recognition effect by performing image segmentation on the image and effectively distinguishing the target from the background, so as to accurately identify various targets in the image.

[0004] Regarding container damage detection and recognition, existing solutions have also started to adopt computer vision detection technologies, such as CN112819793A, CN115187535A, CN115222697A, etc. Although these technical solutions are based on image processing and deep learning for container damage detection, these technical solutions directly predict the category and location information of container damage from two-dimensional images using the model. Therefore, the container damage recognition error in actual applications is relatively large, and when it is necessary to quantify the size of the damage defect later, these solutions cannot provide useful quantitative basic data. Some other existing technologies, such as CN110992337A and CN115965885A, etc., provide methods for detecting container damage based on images and point cloud data. Although they initially achieve more accurate recognition and quantification of damage, they rely on lidar scanning to obtain point cloud data, which increases the hardware and computational costs, maintenance difficulty, and system complexity of the container damage detection system, and it is difficult to provide lidar-related deployment conditions in most container application scenarios. Summary of the Invention

[0005] In view of this, the present invention provides a method, system, device and medium for container quantization benchmark processing and damage location based on image segmentation. By image segmentation, each key part of the container (such as corner fittings, top surface, side surface, etc.) is recognized, and the relative positions and spatial layouts between these key parts are determined to construct a container working condition state model. Then, combined with the standard dimensions of the container and the segmentation result, a dimension calculation benchmark for each area of the container is constructed to obtain the corresponding state model of the container in the actual scenario. Since this state model not only reflects the orientation and position relationship of the container in the image, but also establishes a standardized benchmark through known actual dimensions, which can be used for subsequent damaged dimension quantization calculations.

[0006] The present invention provides the following technical solutions:

[0007] The present invention provides a method for container quantization benchmark processing based on image segmentation, including:

[0008] Segment and recognize each component part of the container in the two-dimensional container image based on a pre-trained segmentation model to form a segmentation result. The component parts include one or more of the following container parts that can reflect the attitude and position of the container: front door of the container, rear front of the container, top surface of the container, side surface of the container, top corner fittings, side corner fittings, spreader.

[0009] According to the position of the corner fittings, the segmentation result, and the known combination relationship between the anchor objects, infer the placement attitude of the container based on the segmentation result. The state model of each placement attitude is distinguished by two parts: an identification sequence and a corner fitting feature.

[0010] According to the standard dimensions of the corresponding standard container type in the placement attitude, construct a calculation benchmark for each component part of the container in the two-dimensional container image, where the standard container type is the standard container corresponding to the container in the two-dimensional container image.

[0011] Compared with the prior art, the beneficial effects that can be achieved by at least one of the above technical solutions adopted by the present invention at least include:

[0012] 1. The present invention accurately segments and recognizes each key part of the container, including the door, front of the container, top surface, side surface, and corner fittings, through instance segmentation technology, and combines the positions of these parts and the relative relationship with the corner fittings, so as to infer the placement state (i.e., the working condition) of the container based on the image.

[0013] 2. By combining the standard dimensions of the container and the segmentation result, a calculation benchmark (such as a geometric model) for each part of the container in the image is established, the damage position is initially determined, and the image coordinates are mapped to the actual dimension coordinates. This geometric model provides a geometric benchmark for subsequent damaged dimension quantization, making the subsequent damaged dimension calculation more accurate and standardized.

[0014] 3. The present invention realizes the container status modeling and damage location only through two-dimensional images and image segmentation technology, avoiding expensive lidar equipment and complex data processing procedures, and reducing the system's hardware cost, maintenance difficulty, and overall complexity. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] To more clearly illustrate the technical solutions of the embodiments of the present application, the accompanying drawings required for the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other accompanying drawings can be obtained based on these drawings without creative efforts.

[0016] Figure 1 is the overall schematic diagram of the container status modeling solution based on image segmentation in the present application;

[0017] Figure 2 is the flowchart of the container quantization benchmark processing method based on image segmentation in the present application;

[0018] Figure 3 is the flowchart of the container damage location method based on image segmentation in the present application;

[0019] Figure 4 is the flowchart of the container status modeling and damage location method based on image segmentation in the present application;

[0020] Figure 5 is the schematic diagram of the camera deployment at the acquisition point in the present application;

[0021] Figure 6a and 6b is the schematic diagram of the result of container image acquisition in the present application;

[0022] Figure 7a and 7b is the schematic diagram of the result of container image segmentation in the present application;

[0023] Figure 8 is the flowchart of the method for inferring the actual placement attitude of the container in the present application;

[0024] Figure 9a and 9b 9c is the schematic diagram of the conditions of the top and side doors and the side of the container in the present application;

[0025] Figure 10 is the flowchart of the method for constructing the container geometric model and dimensional reference in the present application;

[0026] Figure 11It is a schematic diagram of the perspective transformation of a quadrilateral affected by the perspective effect into a standard-size rectangle in this application;

[0027] Figure 12 It is a schematic diagram of obtaining a perspective matrix through perspective transformation of different parts of a container in this application;

[0028] Figure 13 It is a schematic diagram of the positioning dimension diagram of a container in this application;

[0029] Figure 14 It is a schematic diagram of the structure of a container quantization benchmark processing system based on image segmentation in this application;

[0030] Figure 15 It is a schematic diagram of the structure of a container damage positioning system based on image segmentation in this application;

[0031] Figure 16 It is a schematic diagram of the structure of an electronic device in this application. Detailed implementation manners

[0032] The embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0033] The following illustrates the implementation manners of this application through specific specific examples. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of this application, rather than all the embodiments. This application can also be implemented or applied through other different specific implementation manners. Various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, without conflict, the following embodiments and the features in the embodiments can be combined with each other. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in this application without making creative efforts belong to the scope of protection of this application.

[0034] It should be noted that the following describes various aspects of the embodiments within the scope of the appended claims. It should be obvious that the aspects described herein can be embodied in a wide variety of forms, and any specific structure and / or function described herein is illustrative only. Based on this application, those skilled in the art should understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number and aspects described herein can be used to implement the device and / or practice the method. In addition, this device can be implemented and this method can be practiced using other structures and / or functions in addition to one or more of the aspects described herein.

[0035] It should also be noted that the illustrations provided in the following embodiments only schematically illustrate the basic concept of the present application. The diagrams only show the components related to the present application, rather than being drawn according to the number, shape, and size of the components in actual implementation. The type, quantity, and ratio of each component in actual implementation can be arbitrarily changed, and the component layout type may also be more complex.

[0036] In addition, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, those skilled in the art will understand that the examples can be practiced without these specific details.

[0037] Regarding the existing container detection solutions, such as the prior art submitted in the foregoing background art, through the analysis of problems and the exploration of solutions, it is found that:

[0038] On the one hand, the existing solutions are only based on images. Due to problems such as perspective distortion and occlusion in the presentation of the container in the image at different shooting angles, the spatial placement state of the container (i.e., the specific working conditions) is not considered, which leads to errors in the judgment of the damaged frame position, thus making it impossible to accurately determine the damage and lacking the geometric basis for damage quantification. Moreover, damage recognition based on image recognition is limited to predicting the category and approximate position of the damage, without considering the segmentation and positioning of each component of the container (such as corner fittings, doors, top surfaces, side surfaces, etc.), and it is also impossible to provide a geometric basis for subsequent damage size quantification.

[0039] On the other hand, although some existing solutions can initially achieve the quantification work of container damage detection through images and point cloud data, they rely on using lidar scanning to obtain point cloud data, which significantly increases the hardware and computing costs, maintenance difficulty, and system complexity of the system, and most application scenarios do not have the deployment conditions for scanning radars.

[0040] In view of this, the present invention proposes a container status modeling and damage positioning solution based on image segmentation: as Figure 1 shown, by segmenting the image to identify each key part of the container, such as corner fittings, top surfaces, side surfaces, etc., and determining the relative positions and spatial layouts of these parts, combined with the standard dimensions of the container and the segmentation results, a geometric model of the container in the image is constructed. Thus, through the constructed geometric model, not only the key information such as the attitude and position relationship of the container in the image is reflected, but also a standardized benchmark can be established based on the known actual dimensions, so as to provide basic data for subsequent damage size quantification calculations based on the benchmark.

[0041] Therefore, for container inspection, although the spatial position of the container is uncertain in actual applications, corresponding inspections can be carried out after considering the container status. That is, image segmentation technology can be used to identify key components of the container, such as corner fittings, doors, sides, tops, etc., and then determine the relative positions and spatial distributions of these parts. Finally, a container status model is established to facilitate the preliminary positioning of the location of damage, and also lay a foundation for the subsequent quantification of the damage size.

[0042] The following will describe the technical solutions provided by the embodiments of the present application with reference to the accompanying drawings.

[0043] As Figure 2 shown, the present invention provides a method for processing a container quantification benchmark based on image segmentation, which may include:

[0044] Step S202: Segment and identify each component of the container in the two-dimensional container image based on a pre-trained segmentation model to form a segmentation result.

[0045] It should be noted that the segmented components can be related parts that can reflect the attitude, position, etc. of the container. For example, one or more parts such as the container door, the front of the container, the top surface, the side surface, the top surface of the corner fitting, the side surface of the corner fitting, and the spreader.

[0046] The container image is pre-segmented by the segmentation model in advance to obtain the segmentation results corresponding to each component of the container, so that these segmentation results can be used to construct a model subsequently.

[0047] It should be noted that the two-dimensional image can be a container acquisition image provided by an acquisition device. For the acquisition method and result illustration, reference can be made to the examples later.

[0048] Step S204: Infer the placement attitude of the container based on the position of the corner fittings, the segmentation result, and the known combination relationship between the anchor objects according to the segmentation result.

[0049] As analyzed above, the existing solutions only start from the processing of two-dimensional container images. Since the placement attitude of the container is not considered, the results are not ideal. In addition, due to the two-dimensional images collected in the application scenario, due to limitations such as the site and the acquisition angle of the shooting device, the proportions of the various surfaces of the container body in the image are different, and the collected images cannot be directly used for quantification under these factors.

[0050] Therefore, the present invention adds data related to the placement attitude (i.e., specific working conditions) in the processing. Specifically, by combining prior knowledge (such as the position of the corner fittings, the anchor objects, etc.) with the segmentation results obtained by actual segmentation, since there is a combination relationship among these data on the overall container, these relationships can be used to determine the placement attitude, which is convenient for adding these placement attitudes to the subsequent model.

[0051] The state model of each placement posture can use two parts, namely the identification sequence and the corner fitting feature, as distinguishing marks. Among them, the identification sequence can be a string (a pure digital string, a non-digital string, or a string composed of numbers and non-numbers), which is used as a unique identification sequence; if the corner fitting features are the same under the same sequence, it is used as the identification of a state model; while if the sequences are the same but the corner fitting features are different, they are used as the identifications of two different state models.

[0052] Step S206: Construct a calculation reference for each component of the container in the two-dimensional image of the container according to the standard dimensions of the corresponding standard container type in the placement posture. It should be noted that the standard container type here refers to the standard container corresponding to the container in the two-dimensional image of the container. For example, if the container type in the image is a dry cargo container, the standard container type can be the corresponding container type of the dry cargo container.

[0053] By combining the placement posture with the standard container type dimensions, geometric models and calculation references (i.e., new scales) corresponding to each component segmented from the two-dimensional image of the container obtained from actual shooting are established. Thus, based on the geometric models, it can be determined which container surface in the corresponding component is damaged, and for the emerging damage, relevant basic data for dimensional quantification can be provided based on the reference. That is, subsequently, quantification data such as the presence or absence, position, and size of the damage can be obtained using the positioning model.

[0054] In implementation, when the placement posture is known, if the container surface is not damaged, the corresponding geometric model should be the same as (or have very little difference from) the geometric model of this surface of the standard-sized container. Conversely, there should be a difference. Thus, it can be determined whether there is damage based on the difference situation. Similarly, the use of the reference is similar. When there is damage, the specific position and size of the damage can be determined based on the geometric model and the reference, thereby providing basic quantification conditions for the quantification work.

[0055] In summary, by incorporating the placement posture into the positioning model, even for two-dimensional images of containers obtained from different shooting angles, which previously had problems such as perspective distortion and occlusion in the images, after introducing the placement posture, the influence brought by these problems can be reduced, the ability to provide quantification conditions is improved, which is very beneficial for judging the position of the damaged frame and reducing errors, and it can enable the positioning model to have the basic conditions for providing quantification data of the damage, making up for the deficiency that the existing solutions lack quantification basic data and cannot provide it.

[0056] In some implementation manners, when determining the placement posture, the number of anchor objects can be used as the basis for the first judgment. Specifically, first determine the number of anchor objects, and then combine the number of anchor objects in a preset order as the identification sequence. Furthermore, different postures can correspond to different identification sequences.

[0057] For cases with the same identification sequence, the spacing of the corner fittings and / or the inclination of the center connection line are further utilized to distinguish different placement postures.

[0058] It should be noted that in other embodiments, the corner fittings (such as spacing, inclination of the center connection line, etc.) can also be used as the first judgment for inference, which will not be elaborated here.

[0059] In some embodiments, when constructing the geometric model and the reference, it can be to construct the mapping relationship between coordinates to form the data corresponding to the geometric model and the reference.

[0060] For example, first use corner detection to find the vertex coordinates of the box door, box front, top surface, and side surfaces that appear in the image, sort their coordinates counterclockwise starting from the upper left corner, combine the positions of the corner fittings, judge their relative positions, obtain the coordinates and corresponding positions of the box door, box front, top surface, and side surfaces, and then establish the mapping relationship from the coordinates in the image to the actual size coordinates according to the standard sizes of the box door, box front, and corner fittings.

[0061] After establishing the mapping relationship, the geometric models and references of each component can be determined based on the mapping data.

[0062] In some embodiments, after considering the placement posture, in image processing, based on the perspective relationship, more accurate geometric models and references can be obtained, further improving the influence of distortion and occlusion caused by the shooting angle on the performance of the image.

[0063] The mapping relationship can be constructed as follows:

[0064]

[0065] Among them, , is the coordinate of the point in the image, , is the standard size value, H is the perspective transformation matrix between two surfaces, and i = 1, 2, 3, 4.

[0066] After considering the placement posture, the mapping relationship between the coordinates of the points in the image and the standard size values can be completed through the perspective transformation matrix. Among them, the perspective transformation matrix can be prior knowledge (prior data), or posterior knowledge (posterior data) determined according to the actual situation, which is not specifically limited here.

[0067] In some embodiments, for non-quantized targets, they can also be removed first.

[0068] For example, first, based on the segmentation result and the segmentation position of the spreader, the recognition results of non-quantized objects in the image are eliminated, which is more convenient for identifying the anchor and its quantity.

[0069] In some embodiments, the segmentation model of the present invention can be a trained model, that is, the model can be an existing container segmentation model or a segmentation model obtained through the training scheme provided below in the present invention.

[0070] In implementation, the pre-trained segmentation model is trained through the following steps: According to the container two-dimensional image dataset, a dataset required for instance segmentation is constructed, and a preset segmentation model is trained based on the dataset; wherein, the dataset includes annotation data for annotating key parts of the container in the container two-dimensional image.

[0071] The annotation data annotated with key parts constitutes a dataset for instance segmentation training, so that the trained segmentation model is more efficient and accurate in segmenting each component of the container.

[0072] It should be noted that the container two-dimensional images used in the present invention can be container two-dimensional image data from an existing database or container two-dimensional images collected for containers by camera devices set at various positions of the quay crane on the dock and taken at different angles.

[0073] Based on the same inventive concept, the present application also provides a method for locating container damage based on image segmentation, that is, after establishing a geometric model and a size reference of the container in the image, a preliminary position of the damage is located and detected.

[0074] Reference Figure 3 Schematic, a method for locating container damage based on image segmentation includes:

[0075] Step 402, construct a calculation reference for each component of the container in the image. Among them, the geometric model and size reference of each component of the container in the image can be obtained by using the method for processing the container quantization reference based on image segmentation described in any one of the examples in the present application, etc., to form the calculation reference required for quantization.

[0076] Step 404, based on the calculation reference established in the image, determine the position of the container damage.

[0077] By establishing calculation reference data such as geometric models and size references for the container in the image, the actual state and actual ratio of the container in the image are obtained, and then the damage on the container body can be quantitatively located, improving the accuracy of damage location.

[0078] In some embodiments, in determining which container surface has damage and even in initially determining the location of the damage, the following steps may be included: First, determine whether the damage is located on a certain container surface; then, based on the standard dimensions of each part of the container and the relative position of the damage, use perspective transformation to map the damage coordinates to the actual space to complete the initial determination of the location of the container damage.

[0079] Obtaining the mapping of the damage coordinates to the actual space through the standard dimensions of each part of the container and the relative position of the damage is beneficial to improving the accuracy of the quantitative data.

[0080] In some embodiments, with the assistance of damage box annotation, the damage situation can be determined more effectively. Therefore, when determining whether the damage is located on a certain container surface, the following determination method can also be adopted: First, obtain the four vertex coordinates of each part of the container from the segmentation result, and then, in combination with the center coordinates of the damage box, use the algorithm of a point inside a polygon to initially determine whether the damage is located on a certain container surface.

[0081] Next, another example is listed. This example is a schematic illustration formed by combining the foregoing multiple examples.

[0082] Reference Figure 4 Schematically, in the method for building a damage location model for the container status based on image segmentation proposed by the present invention, the following main steps are included:

[0083] Step 1: Collect two-dimensional images of the container through camera devices at different angles at each position of the gantry crane, label the key parts, construct a dataset for instance segmentation, and train the instance segmentation model.

[0084] In the dataset in Step 1, it can be a dataset in which each part (i.e., the combined part) is labeled. The parts may include the container door, the front of the container, the top surface, the side surface, the top surface of the corner fitting, the side surface of the corner fitting, and the spreader. Therefore, based on the labeled dataset, the instance segmentation model can be trained with labeled features.

[0085] In addition, for an example of collecting two-dimensional images of the container, reference can be made to Figure 5 Schematically: Limited by the actual scene conditions, cameras A and B deployed at the collection points can collect images at both ends of the container, and camera C can collect images of the side of the container. That is, cameras A and B for imaging can be deployed on the gantry crane in the traveling direction of the container, and camera C can be deployed in the side space of the scene. At this time, cameras A and B collect images of the top surface of the container from the upper space, and the side camera C collects images of the side of the container. The collection results can be seen in Figure 6a 、 6b Schematically. It should be noted that those skilled in the art should understand,Figure 6a , 6b The mosaic area on the container is used to block the original commercial logo of the container, which does not affect the understanding of this application and does not constitute relevant limitations on this application. Similar illustrations involved in the following text will not be elaborated one by one.

[0086] Step 2: Based on the trained instance segmentation model, perform segmentation and recognition on the two-dimensional container image; among them, segment the collected two-dimensional container image to form segmentation results of each key part of the container (such as corner fittings, container surfaces, etc.). Refer to Figure 7a , for the two-dimensional container image collected from the gantry crane camera A or B, segmentation can be performed to obtain the top container surface segmentation result (see the wireframe area indication at the position of the top surface), the corner fitting segmentation result (see the area indication of the white dots at the position of the top surface corner fittings), the container door surface segmentation result (see the wireframe area indication at the position of the container door surface), etc.; refer to Figure 7b for illustration, for the two-dimensional container image collected from the side camera C, segmentation can be performed to obtain the side container surface segmentation result (see the wireframe area indication at the position of the side surface), the corner fitting segmentation result (see the area indication of the white dots at the position of the top surface corner fittings), etc.

[0087] Step 3: Infer the placement posture of the container (i.e., the spatial state of the container, also referring to the specific working conditions of the container) according to the combination method among the corner fitting positions, segmentation results, and anchoring objects; it should be noted that the anchoring object can be regarded as one of the key parts of the container, so that according to the known combination methods of components such as corner fittings, container surfaces, and anchoring objects, the spatial state of the container can be inferred quickly and accurately.

[0088] Refer to Figure 8 For illustration, in the inference of the container state (i.e., the placement posture), first, based on the segmentation and recognition results and the segmentation position of the spreader, the recognition results of non-quantized objects in the image are eliminated, and then the number of each part, that is, the number of anchoring objects, is determined. Their numbers are combined in a certain order to form an identification sequence. For cases with the same identification sequence, the distance between the recognized corner fittings and the inclination of the center connection line are further used for differentiation, that is, when the sequences are the same, secondary classification of the corner fittings is further combined. Therefore, by including two parts, the identification sequence and the corner fitting features, in the state model of each working condition, the state models under different working conditions can be further distinguished.

[0089] Specifically, after corner point detection of the image segmentation result of the box body, the vertex coordinates of the segmented box body surface are obtained, and the center of the segmentation result of the corner fitting surface is taken as the corner fitting center coordinates. As shown in the appendix Figure 8Eliminate the recognition results of non - quantified objects in the image based on the results of segmentation recognition and the segmentation positions of the spreaders, then determine the quantity of each part, i.e., the number of anchors, and combine these quantities in a certain order to form an identification sequence. For cases with the same identification sequence, further distinguish them by using the spacing between the recognized corner fittings and the inclination of the center connection line. That is, the state model of each working condition consists of two parts: the identification sequence and the corner fitting features.

[0090] As shown in the Figure 9a figure, it is the schematic diagram of the state corresponding to the working condition named "top - door". ABCD are the four vertices of the box door, EFHG are the vertices of the box top segmentation area, ab is the center of the corner fitting, and the corresponding working condition sequence can be determined as 101020. This sequence of this working condition is unique among all states and can uniquely determine its spatial state.

[0091] As shown in the Figure 9b , 9c figure are the schematic diagrams of the working conditions of "side2" and "side2 - 1" respectively. The sequences of both working conditions are 000102, and they are distinguished according to the position characteristics of the corner fittings. Specifically, it is judged by the inclination of the connection lines ab and a´b´ passing through the centers of the corner fittings.

[0092] Step 4: Combine the standard dimensions of different box types to construct the calculation benchmarks for each part of the container in the image;

[0093] Refer to Figure 10 and Figure 11 the schematic diagrams. In the construction process of the container geometric model and benchmarks, first use corner point detection to find the vertex coordinates of the box door, box front, top surface, and side surfaces that appear in the image. Sort these coordinates counter - clockwise starting from the upper left corner, and combine the positions of the corner fittings to judge their relative positions, then obtain the coordinates and positions of the box door, box front, top surface, and side surfaces in the image. Then, based on the standard dimensions of the box door, box front, and corner fittings, establish the perspective mapping relationship from the coordinates in the image to the actual - size coordinates, that is, complete the construction of the benchmarks for each part. The mapping relationship can be expressed as:

[0094]

[0095] Among them, , are the coordinates of the points in the image, , are the standard size values, H is the perspective transformation matrix between two surfaces, and i = 1, 2, 3, 4.

[0096] As shown in Figure 11As shown, by combining the standard dimensions of different box types, the perspective mapping relationships between the coordinates of the box door, box front, side, and top in the image and the actual dimension coordinates are established respectively. Among them, points 1, 2, 3, and 4 are the four vertices of the box door, box front, side, or top in the image, points 1’, 2’, 3’, and 4’ are the vertices of the transformed standard rectangle, and L and D are their standard dimension values. Specifically, for the box door or box front, L is the box width value of this box type, with the unit of mm, and D is the box height H with the unit of mm. For the box top, L is the box width of this top surface, with the unit of mm, and D is the length value of the actual divided part of this top surface along the box length direction, with the unit of mm. For the side, L is the length value of the actual divided part of this side along the box length direction, with the unit of mm, and D is the box height of this box type, with the unit of mm. H is the transformation matrix.

[0097] It should be noted that the perspective transformation schematic diagrams of the quadrilaterals affected by the perspective effect at different parts to the standard dimension rectangles can be referred to Figure 12 Illustration: (a) For the box front or box door surface, the corresponding transformation matrix H1 is obtained through the aforementioned transformation; (b) For the box top surface, the corresponding transformation matrix H2 is obtained through the aforementioned transformation; (c) For the box side, the corresponding transformation matrix H3 is obtained through the aforementioned transformation.

[0098] As shown in the appendix Figure 13 The following is the positioning dimension diagram of the container, where C1 and C2 are the structural dimensions related to the corner fittings, and H, L, and W are the external height, length, and width of the container, which are specifically related to the box type. P and S are the distances between the center points of the corner fitting holes along the width and length directions of the container body.

[0099] As shown in the appendix Figure 9a The working condition shown in the appendix includes two parts: the box door and the box top. The calculation basis is the perspective transformation matrix within their surfaces. For the vertices A, B, C, and D of the box door, they correspond to an ordered list of image coordinates. This box body corresponds to a box door with standard dimensions. Let its actual height be H mm and width be W mm. The transformed coordinates of A, B, C, and D can be set as [[0, H], [0, 0], [W, 0], [W, H]]. The transformation matrix within its surface can be obtained from the coordinates before and after the transformation. For the box top, the actual lengths of its EF and HG sides are estimated. The estimation method is that the dimensions of aF and bG are a standard length determined by the box type, and their projected lengths on EF and HG are used as the unit lengths of EF and HG. The average value of the actual lengths of EF and HG is calculated and set as L, and the width is W. Then the transformed coordinates of EFGH are [[0, L], [0, 0], [W, 0], [W, L]]. The transformation matrix within its surface can be obtained from the coordinates before and after the transformation.

[0100] Step 5: Based on the benchmark established in the image, preliminarily judge the position of the damage to the container to prepare for the quantification of the damage size;

[0101] In damage determination, after obtaining the four vertex coordinates of each part of the container through instance segmentation, the coordinates are sorted counterclockwise starting from the upper left corner. Combining with the center coordinates of the damage box, the point-in-polygon algorithm is used to preliminarily determine whether the damage is located on a certain container surface. Then, based on the standard dimensions of each part of the container and the relative position of the damage, perspective transformation is used to map the damage coordinates to the actual space to complete the preliminary determination of the damage position of the container.

[0102] Based on the same inventive concept, the present application also provides a container quantification benchmark processing system based on image segmentation, and a container quantification benchmark processing system formed based on each of the foregoing method examples to automatically, real-time and accurately establish a damage positioning model.

[0103] Reference Figure 14 As shown in the schematic diagram, a container quantification benchmark processing system 100 based on image segmentation may include:

[0104] Segmentation model 101: Segment and identify each component part of the container in the two-dimensional container image based on a pre-trained segmentation model to form a segmentation result;

[0105] Inference model 103: Infer the placement posture of the container according to the corner fitting position, the segmentation result, and the known combination relationship between the anchors based on the segmentation result;

[0106] Quantification benchmark model 105: Construct a calculation benchmark for each component part of the container in the two-dimensional container image according to the standard dimensions of the corresponding standard container type in the placement posture.

[0107] It should be noted that the function settings of the unit modules and the number of modules in the container quantification benchmark processing system based on image segmentation can be set accordingly according to the foregoing method embodiments, and will not be elaborated here.

[0108] Based on the same inventive concept, the present application also provides a container damage positioning system based on image segmentation.

[0109] Reference Figure 15 As shown in the schematic diagram, a container damage positioning system 300 based on image segmentation includes:

[0110] Status module 301: Establish a calculation benchmark for each component part of the container in the image by using the container quantification benchmark processing method based on image segmentation described in any one of the examples in the present application;

[0111] Positioning module 303: Determine the position of the container damage based on the geometric model and dimension benchmark established in the image.

[0112] It should be noted that the functions of the unit modules and the settings of the number of modules in the container damage location system based on image segmentation can be set accordingly according to the foregoing method embodiments, and will not be elaborated here.

[0113] Based on the same inventive concept, the present application also provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein, the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute: the container quantization benchmark processing method based on image segmentation as described in any one of the embodiments of the present application, or the container damage location method based on image segmentation as described in any one of the embodiments of the present application.

[0114] As Figure 16 shown, the present application also provides a schematic structural diagram of an electronic device. The structure of the electronic device 500 is shown in the figure. Here, the electronic device 500 is only an example and should not limit the functions and usage scope of the embodiments of the present invention.

[0115] In the electronic device 500, it may include: at least one processor 510; and a memory 520 communicatively connected to the at least one processor; wherein, the memory 520 stores instructions executable by the at least one processor 510, and the instructions are executed by the at least one processor 510 to enable the at least one processor 510 to execute: the container quantization benchmark processing method based on image segmentation as described in any one of the embodiments of the present application, or the container damage location method based on image segmentation as described in any one of the embodiments of the present application.

[0116] It should be noted that the electronic device 500 may be presented in the form of a general computing device, for example, it may be a server device.

[0117] In implementation, the components of the electronic device 500 may include but are not limited to: the above-mentioned at least one processor 510, the above-mentioned at least one memory 520, and a bus 530 connecting different system components (including the memory 520 and the processor 510), wherein the bus 530 may include a data bus, an address bus, and a control bus.

[0118] In implementation, the memory 520 may include volatile memory, such as random access memory (RAM) 5201 and / or cache memory 5202, and may further include read-only memory (ROM) 5203.

[0119] The memory 520 may also include a program tool 5205 having a set (at least one) of program modules 5204. Such program modules 5204 include, but are not limited to: an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment.

[0120] The processor 510 performs various functional applications and data processing by running computer programs stored in the memory 520.

[0121] The electronic device 500 may also communicate with one or more external devices 540 (such as a keyboard, a pointing device, etc.). Such communication may be carried out through the input / output (I / O) interface 550. Moreover, the electronic device 500 may also communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through the network adapter 560. The network adapter 560 communicates with other modules in the electronic device 500 through the bus 530. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 500, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID (redundant array of independent disks) systems, tape drives, and data backup storage systems, etc.

[0122] In this specification, for the same or similar parts among the various embodiments, reference can be made to each other. Each embodiment focuses on the differences from other embodiments. In particular, for the embodiments described later, the description is relatively simple, and for the relevant parts, reference can be made to the partial description of the foregoing embodiments.

[0123] As described above, the above is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any change or replacement that can be easily thought of by those skilled in the art within the technical scope disclosed by the present application should be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A container quantitative benchmark processing method based on image segmentation, characterized in that: include: Based on the pre-trained segmentation model, each component of the container in the two-dimensional image of the container is segmented and identified to form a segmentation result, wherein the component includes one or more of the following container parts that can reflect the posture and position of the container: front door, rear front, top surface, side surface, top corner fittings, side corner fittings, and slings; According to the corner fittings positions, segmentation results and known combination relationships between anchors, the container placement is inferred based on the segmentation results. The state model of each placement is distinguished by two parts: the identification sequence and the corner fittings features. According to the standard size of the standard container type corresponding to the placement posture, a calculation basis of each component of the container in the two-dimensional container image is constructed, the damaged position is preliminarily determined, and the image coordinates are mapped to the actual size coordinates; wherein the standard container type is the standard container corresponding to the container in the two-dimensional container image.

2. The container quantitative benchmark processing method based on image segmentation according to claim 1 is characterized in that: Inferring the placement posture of the container based on the segmentation result includes: determining the number of anchors, combining the number of anchors in a preset order as an identification sequence, and further distinguishing the placement posture by identifying the spacing of the corner pieces and / or the inclination of the center line for the same identification sequence; And / or, constructing the calculation reference of each part of the container includes the following steps: using corner point detection to find the coordinates of each vertex of the door, front, top and side of the container that appear in the image, sorting the coordinates in a counterclockwise order from the upper left corner, combining the position of the corner fittings, judging their relative positions, obtaining the coordinates and corresponding positions of the door, front, top and side, and then establishing a mapping relationship from the coordinates in the image to the actual size coordinates according to the standard sizes of the door, front and corner fittings; wherein the mapping relationship is constructed as follows: in, , are the coordinates of the midpoint of the image, , is the standard size value, H is the perspective transformation matrix between two faces, i=1,2,3,4; And / or, before determining the number of anchors, the recognition results of non-quantified targets in the image are eliminated based on the segmentation results and the segmented positions of the spreaders.

3. The container quantitative benchmark processing method based on image segmentation according to any one of claims 1-2, characterized in that: The pre-trained segmentation model is trained by the following steps: constructing a data set required for instance segmentation based on a container two-dimensional image data set, and training a preset segmentation model based on the data set; wherein the data set includes annotation data for annotating key parts of the container in the container two-dimensional image; And / or, the two-dimensional image of the container is a two-dimensional image of the container captured by a camera device at different angles at various positions of the bridge crane.

4. A container damage location method based on image segmentation, characterized in that: include: Using the container quantitative benchmark processing method based on image segmentation as claimed in any one of claims 1 to 3, the calculation benchmark of each component of the container in the image is obtained; Based on the calculated reference established in the image, the location of the container damage is determined.

5. The method for locating damaged containers based on image segmentation according to claim 4 is characterized in that: Determining the location of container damage includes: first determining whether the damage is located on a certain container surface; then, based on the standard size of each part of the container and the relative position of the damage, using perspective transformation to map the damage coordinates to the actual space, thereby completing a preliminary determination of the location of the container damage.

6. The method for locating damaged containers based on image segmentation according to claim 5, characterized in that: When judging whether the damage is located on a certain container surface, the coordinates of the four vertices of each part of the container are first obtained from the segmentation result, and then combined with the center coordinates of the damage box, the point-in-polygon algorithm is used to preliminarily judge whether the damage is located on a certain container surface.

7. A container quantitative benchmark processing system based on image segmentation, characterized in that: include: Segmentation model: Based on the pre-trained segmentation model, the various components of the container in the two-dimensional image of the container are segmented and identified to form a segmentation result. The components include one or more of the following container parts that can reflect the posture and position of the container: front door, rear front, top surface, side surface, top corner fittings, side corner fittings, and slings; Reasoning model: Based on the corner fittings position, segmentation results and known combination relationships between anchors, the container placement is inferred based on the segmentation results. The state model of each placement posture uses the identification sequence and corner fittings features as distinguishing marks. Quantitative benchmark model: constructing a calculation benchmark for each component of the container in the two-dimensional container image according to the standard size of the standard container type corresponding to the placement posture, wherein the standard container type is the standard container corresponding to the container in the two-dimensional container image.

8. A container damage positioning system based on image segmentation, characterized in that: include: State module: using the container quantitative benchmark processing method based on image segmentation as described in any one of claims 1 to 3 to establish a calculation benchmark for each component of the container in the image; Positioning module: Determines the location of damaged containers based on the calculated reference established in the image.

9. An electronic device, characterized in that: include: at least one processor; And, a memory communicatively connected to the at least one processor; wherein the memory stores instructions executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute: a container quantitative benchmark processing method based on image segmentation as described in any one of claims 1 to 3, or a container damage positioning method based on image segmentation as described in any one of claims 4 to 6.

10. A computer storage medium, characterized in that: The computer storage medium stores computer executable instructions, and when the computer executable instructions are executed by the processor, they perform: a container quantitative benchmark processing method based on image segmentation as described in any one of claims 1-3, or a container damage positioning method based on image segmentation as described in any one of claims 4-6.

Images