Deformation detection method, device, electronic equipment and storage medium for box-type structure

Abstract

The present application provides a deformation detection method, device, electronic device and computer-readable storage medium for a box-type structure; the method comprises: obtaining a depth image of the side of the box-type structure; based on the depth information of the depth image, identifying the areas corresponding to the multiple components included in the side from the depth image; determining a reference plane corresponding to at least one of the components, and determining the distance between the pixels included in each of the areas and the reference plane; based on the distance between the pixels included in each of the areas and the reference plane, determining the deformation area in each of the areas. Through the present application, the area where the box-type structure is deformed can be detected efficiently and accurately.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a deformation detection method, device, electronic device and computer-readable storage medium for a box-type structure. Background Art

[0002] Artificial Intelligence (AI) is a comprehensive technology in computer science. By studying the design principles and implementation methods of various intelligent machines, it enables machines to have the functions of perception, reasoning and decision-making. Artificial Intelligence technology is a comprehensive discipline that covers a wide range of fields, such as natural language processing technology and machine learning / deep learning. With the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0003] The related technology lacks an effective solution for detecting the deformation of box-type structures based on artificial intelligence. The box-type structure is mainly observed by human eyes, and the specific degree of deformation is determined by manual measurement using measuring instruments for the observed deformed areas. This leads to low efficiency and accuracy in deformation detection of box-type structures. Summary of the invention

[0004] The embodiments of the present application provide a deformation detection method, device, electronic device and computer-readable storage medium for a box-type structure, which can efficiently and accurately detect the area where the box-type structure has been deformed.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] The present application provides a method for detecting deformation of a box-type structure, including:

[0007] Acquire a depth image of the side of the box-like structure;

[0008] Based on the depth information of the depth image, identifying from the depth image the regions corresponding to the multiple components included in the side surface;

[0009] Determine a reference plane corresponding to at least one of the components, and determine a distance between a pixel point included in each of the regions and the reference plane;

[0010] Based on the distance between the pixel points included in each of the regions and the reference plane, a deformation region in each of the regions is determined.

[0011] In the above scheme, when there is a specific component whose intensity is greater than the intensity threshold among the multiple components, determining the reference plane corresponding to at least one of the components includes: determining the fifth depth mean of the pixels included in the area corresponding to the specific component; based on the fifth depth mean, determining the specific reference plane corresponding to the specific component; determining the deformation area in each of the areas based on the distance between the pixels included in each area and the reference plane includes: determining the distance between the pixels included in the specific component and the specific reference plane; based on the distance between the pixels included in the specific component and the specific reference plane, determining the deformation area in the area corresponding to the specific component.

[0012] In the above scheme, the obtaining of the two-dimensional image of the side of the box-type structure includes: obtaining three-dimensional point cloud data for the box-type structure; extracting the two-dimensional image of the side from the three-dimensional point cloud data; the obtaining of the depth image of the side of the box-type structure includes: extracting the depth image of the side from the three-dimensional point cloud data.

[0013] The present application provides a box-type deformation detection device, comprising:

[0014] An acquisition module, used for acquiring a depth image of the side of the box-type structure;

[0015] an identification module, configured to identify, from the depth image, regions corresponding to the plurality of components included in the side surface, respectively, based on the depth information of the depth image;

[0016] A determination module, used to determine a reference plane corresponding to at least one of the components, and to determine a distance between pixels included in each of the regions and the reference plane;

[0017] The determination module is further used to determine the deformation area in each of the regions based on the distance between the pixel points included in each of the regions and the reference plane.

[0018] In the above scheme, the determination module is also used to slide through the depth image with a window of a preset size to determine the jump amplitude of the depth value between different pixel points included in each window; the area where the window corresponding to the jump amplitude is greater than the amplitude threshold is located is determined as a noise point in the depth image; the device also includes a deletion module for deleting the noise point from the depth image.

[0019] In the above scheme, the recognition module is also used to identify the areas corresponding to the multiple components included in the side from the depth image based on the depth information of the depth image and the gradient difference between different pixel points included in the depth image.

[0020] In the above scheme, the determination module is also used to determine the pixel points in the depth image whose depth value is less than the depth threshold and whose gradient difference with the adjacent pixel points is greater than the gradient difference threshold; and the area in the depth image composed of pixel points whose depth value is less than the depth threshold and whose gradient difference with the adjacent pixel points is greater than the gradient difference threshold, and the corresponding spatial feature is a corner, is determined as the area corresponding to the corner piece included in the side.

[0021] In the above scheme, the determination module is also used to determine the first boundary of the reinforcing plate included in the side surface that coincides with the corner piece based on the area corresponding to the corner piece included in the side surface; taking at least one pixel point included in the first boundary as a starting point, traversing and determining the gradient of multiple subsequent pixel points; determining the second boundary of the reinforcing plate based on the pixel points corresponding to the position where the gradient difference is greater than the gradient difference threshold; and determining the area composed of the first boundary and the second boundary in the depth image as the area corresponding to the reinforcing plate included in the side surface.

[0022] In the above scheme, the determination module is also used to determine the left and right boundaries of the side beam included in the side based on the area corresponding to the reinforcing plate included in the side; take at least one pixel point included in the boundary of the depth image as the starting point, traverse and determine the gradient of subsequent multiple pixel points; determine the upper and lower boundaries of the side beam based on the pixel points corresponding to the positions where the gradient difference is greater than the gradient difference threshold; and determine the area in the depth image composed of the left and right boundaries of the side beam and the upper and lower boundaries of the side beam as the area corresponding to the side beam included in the side.

[0023] In the above scheme, the determination module is also used to determine the upper and lower boundaries of the beam included in the side surface based on the area corresponding to the corner piece included in the side surface; determine the interval composed of the minimum horizontal coordinate and the maximum horizontal coordinate corresponding to the corner piece, and traverse to determine the gradient of each pixel point in the interval; determine the left and right boundaries of the beam based on the pixel points corresponding to the position where the gradient difference is greater than the gradient difference threshold; determine the area composed of the upper and lower boundaries of the beam and the left and right boundaries of the beam in the depth image as the area corresponding to the beam included in the side surface.

[0024] In the above scheme, the determination module is also used to determine the boundary of the extension plate included in the side that coincides with the crossbeam based on the area corresponding to the crossbeam included in the side; determine the upper and lower boundaries of the extension plate based on the area corresponding to the reinforcing plate included in the side; take at least one pixel point included in the boundary that coincides with the crossbeam as the starting point, traverse and determine the gradient of subsequent multiple pixel points; determine the remaining boundary of the extension plate based on the pixel points corresponding to the position where the gradient difference is greater than the difference threshold; determine the area in the depth image composed of the boundary that coincides with the crossbeam, the upper and lower boundaries of the extension plate and the remaining boundary as the area corresponding to the extension plate included in the side; and determine the area in the side except for the corner piece, the reinforcing plate, the crossbeam, the side beam and the extension plate as the area corresponding to the side plate included in the side.

[0025] In the above scheme, the device also includes a selection module for selecting a target component of the largest size from the multiple components; the determination module is also used to determine the depth mean of the pixel points included in the area corresponding to the target component, and determine the reference plane corresponding to the target component based on the depth mean.

[0026] In the above scheme, the determination module is also used to determine the convex groove area and the concave groove area included in the side; determine the first depth mean of the pixels included in the convex groove area, and delete the pixels in the convex groove area whose distance from the first depth mean is greater than the depth mean threshold; determine the second depth mean of the remaining pixels in the convex groove area, and determine the first reference plane corresponding to the convex groove area based on the second depth mean; determine the third depth mean of the pixels included in the concave groove area, and delete the pixels in the concave groove area whose distance from the third depth mean is greater than the depth mean threshold; determine the fourth depth mean of the remaining pixels in the concave groove area, and determine the second reference plane corresponding to the concave groove area based on the fourth depth mean.

[0027] In the above scheme, the determination module is also used to perform the following processing for each of the regions: determining the pixel points in the region whose distance is greater than a distance threshold; determining the region composed of the pixel points corresponding to the distance greater than the distance threshold as a deformation region in the region; and determining the confidence of the deformation region based on the ratio between the number of noise points included in the deformation region and the total number of pixel points included in the deformation region.

[0028] In the above scheme, when there is a specific component whose intensity is greater than the intensity threshold among the multiple components, the determination module is also used to determine the fifth depth mean of the pixel points included in the area corresponding to the specific component; based on the fifth depth mean, determine the specific reference plane corresponding to the specific component; and determine the distance between the pixel points included in the specific component and the specific reference plane; based on the distance between the pixel points included in the specific component and the specific reference plane, determine the deformation area in the area corresponding to the specific component.

[0029] In the above scheme, the acquisition module is also used to acquire a two-dimensional image of the side of the box-type structure; the device also includes a labeling module, which is used to label the areas corresponding to the multiple components included in the side and the deformation areas in each of the areas in the two-dimensional image based on the correspondence between the depth image and the two-dimensional image.

[0030] In the above scheme, the acquisition module is also used to acquire three-dimensional point cloud data for the box-type structure; the device also includes an extraction module, which is used to extract the two-dimensional image of the side from the three-dimensional point cloud data; and is used to extract the depth image of the side from the three-dimensional point cloud data.

[0031] An embodiment of the present application provides an electronic device, including:

[0032] A memory for storing executable instructions;

[0033] The processor is used to implement the deformation detection method of the box-type structure provided in the embodiment of the present application when executing the executable instructions stored in the memory.

[0034] An embodiment of the present application provides a computer-readable storage medium storing executable instructions for causing a processor to execute and implement the deformation detection method of a box-type structure provided in the embodiment of the present application.

[0035] An embodiment of the present application provides a computer program product, which includes computer executable instructions for implementing the deformation detection method of the box-type structure provided in the embodiment of the present application when executed by a processor.

[0036] The embodiments of the present application have the following beneficial effects:

[0037] By acquiring a depth image of the side of the box-type structure, it is possible to identify the areas corresponding to the multiple components included in the side from the depth image based on the depth information of the depth image. Then, at least one component can be selected from the multiple components to determine the reference plane corresponding to the at least one component. Subsequently, the deformation area in each area can be determined based on the distance between the pixels included in each area and the reference plane. In this way, automatic detection of the deformation degree of the box-type structure is achieved, thereby improving detection efficiency and detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 1 is a schematic diagram of the architecture of a box-type deformation detection system 100 provided in an embodiment of the present application;

[0039] Figure 2 is a schematic diagram of the structure of the server 200 provided in an embodiment of the present application;

[0040] Figure 3 It is a schematic diagram of the flow of a deformation detection method of a box-type structure provided in an embodiment of the present application;

[0041] Figure 4 It is a schematic diagram of the flow of a deformation detection method of a box-type structure provided in an embodiment of the present application;

[0042] Figure 5 It is a schematic diagram of the flow of a deformation detection method of a box-type structure provided in an embodiment of the present application;

[0043] Fig. 6A is a line scan 2D schematic diagram of the top surface of a container provided in an embodiment of the present application;

[0044] Figure 6B is a depth image of the top surface of the container provided in an embodiment of the present application;

[0045] Figure 6C is a schematic diagram of deformation detection results of the top surface of a container provided in an embodiment of the present application;

[0046] Figure 7 It is a schematic diagram of the flow of a deformation detection method of a box-type structure provided in an embodiment of the present application;

[0047] Fig. 8A is a line scan 2D schematic diagram of the top surface of a container to be inspected provided in an embodiment of the present application;

[0048] Figure 8B is a depth image of the top surface of the container to be detected provided by an embodiment of the present application;

[0049] Fig. 9 is a schematic diagram of the regional segmentation result of the top surface of a container provided in an embodiment of the present application;

[0050] Fig.10 It is a flow chart of a container top surface deformation detection method provided by related technology;

[0051] Fig.11 It is a schematic diagram of the process of the container top surface deformation detection method provided in the embodiment of the present application;

[0052] Fig. 12A is a line scan 2D schematic diagram of the top surface of a container to be inspected provided in an embodiment of the present application;

[0053] Fig. 12B is a depth image of the top surface of the container to be detected provided by an embodiment of the present application;

[0054] Fig. 12C is a schematic diagram of deformation detection results of the top surface of a container provided in an embodiment of the present application;

[0055] Fig.13A is a line scan 2D schematic diagram of the top surface of a container to be inspected provided in an embodiment of the present application;

[0056] Fig. 13B is a depth image of the top surface of the container to be detected provided in an embodiment of the present application;

[0057] Fig. 13C It is a schematic diagram of the deformation detection result of the top surface of the container provided in an embodiment of the present application. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0059] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it can be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0060] In the following description, the terms "first\second" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0061] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0062] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0063] 1) Depth image, also known as distance image, refers to an image that uses the depth (distance) from the image collector (such as a camera) to each point in the scene as a pixel value. It directly reflects the geometric shape of the visible surface of the scene. For example, the grayscale value of each pixel in the depth image can be used to represent the distance of a certain point in the scene from the camera. The depth image can be calculated into point cloud data after coordinate transformation, and point cloud data with rules and necessary information can also be inversely calculated into depth image data.

[0064] 2) Point cloud data, also known as three-dimensional point cloud data, refers to a set of vectors in a three-dimensional coordinate system. These vectors are usually expressed in the form of X, Y, and Z three-dimensional coordinates, and are generally used to represent the outer surface shape of an object. In addition to the geometric position information represented by (X, Y, Z), point cloud data can also represent the RGB color, grayscale value, depth, segmentation results, etc. of a point.

[0065] 3) Linear scanning imaging refers to the process of imaging an object by scanning the image plane each time to form a line image, or even a two-dimensional image, and finally stitching them together along the direction of the object's movement to achieve a complete image, which is referred to as line scanning imaging.

[0066] 4) 2D image, also known as plane image, has only X-axis and Y-axis. 2D image and RGB image have the same meaning.

[0067] 5) 3D images refer to images with three-dimensional effects, with X-axis, Y-axis and Z-axis.

[0068] 6) Box-type structure, a three-dimensional structure with a storage space, such as containers, packaging boxes, modular houses, etc. Taking the top surface of a container as an example, the top surface of a container usually includes corner pieces (located at the four corners of the top surface), reinforcement plates (located at the four corners of the top surface, next to the corner pieces), extension plates (located at both ends of the top surface), top cross beams (located at the left and right ends of the top surface), top side beams (located at the upper and lower ends of the top surface) and top plates (located in the middle of the top surface).

[0069] Taking the container as an example, in order to detect the deformation degree of the container surface, the relevant technology provides the following solutions:

[0070] Solution A: Capture external images and / or internal images of the container, where the external images are used to detect damage to the outside of the container, while the internal images are used to detect damage to the inside of the container; store or transmit the captured images for subsequent calculation to obtain the inspection results and be able to flexibly capture container images in non-fixed scenarios.

[0071] Solution B: Provide a fixed-scene container intelligent inspection system, including a container truck position acquisition unit, an image acquisition and processing unit, a terminal processing unit, a data transmission unit, and a human-computer interaction unit. The system is deployed on the lane where the container truck passes through, and is used in fixed scenes including ports, customs, container yards, yards and other container entrances, for automatic image acquisition and intelligent inspection of containers on the container truck.

[0072] Solution C: The container is observed by the inspector's eyes, and the areas observed to be deformed are manually measured using a measuring instrument to determine the specific degree of deformation.

[0073] However, the applicant discovered during the implementation of this application that although Scheme A and Scheme B can detect the degree of deformation of the top surface of the container, both technical solutions are complex and only use 2D information. Among them, Scheme A only uses a 2D camera and cannot obtain specific deformation values. It is only suitable for a rough analysis of container losses. For container yards, it is necessary to accurately obtain the precise deformation values ​​of each area of ​​the container, which cannot be achieved with a 2D camera alone. Although the image acquisition device of Scheme B is a high-definition camera, it is also unable to obtain specific values ​​of regional deformation. In addition, if the container has turns in the above two solutions, the collected information itself will inevitably not be consistent with statistical laws. At this time, the entire system will fail and deformation detection cannot be performed at all. Scheme C relies heavily on the proficiency of the inspection personnel, is too inefficient, takes a long time to inspect, and often only focuses on areas with severe deformation.

[0074] In response to the above technical problems, the embodiments of the present application provide a deformation detection method, device, electronic device and computer-readable storage medium for a box-type structure, which can efficiently and accurately detect the area where the box-type structure is deformed. The following describes an exemplary application of the electronic device provided by the embodiments of the present application. The electronic device provided by the embodiments of the present application can be implemented as a terminal, or as a server, or implemented by a terminal and a server in collaboration. The following is an example of a deformation detection method for a box-type structure provided by an embodiment of the present application being implemented in collaboration with a terminal and a server.

[0075] See also Figure 1, Figure 1 It is a schematic diagram of the architecture of the box-type structure deformation detection system 100 provided in an embodiment of the present application. The terminal 400 is connected to the server 200 via the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0076] The terminal 400 (running a client 410, such as a deformation detection client of a box-type structure) can be used to obtain a deformation detection request for the box-type structure. For example, after the detection personnel input the depth image of the side of the box-type structure in the client 410, the terminal 400 automatically obtains the deformation detection request for the box-type structure. Then, the terminal 400 sends the deformation detection request for the box-type structure to the server 200 through the network 300. After receiving the deformation detection request for the box-type structure sent by the terminal 400, the server 200 extracts the depth image of the side of the carried box-type structure from the deformation detection request, and then, based on the depth information of the depth image, identifies from the depth image the areas corresponding to the multiple components included in the side (for example, when the box-type structure is a container, the corresponding multiple components may be corner pieces, reinforcement plates, extension plates, beams, etc.), and then selects at least one component from the multiple components, determines the reference plane corresponding to the at least one component, and determines the distance between the pixels included in each area and the reference plane. Finally, based on the distance between the pixels included in each area and the reference plane, determines the deformation area (i.e., the detection result) in each area. After determining the deformation area in each area, the server 200 can return the detection results carrying the determined deformation areas in each area to the terminal 400 through the network 300, so that the terminal 400 can mark the two-dimensional image corresponding to the depth image (for example, the inspection personnel can input the depth image of the side of the box-type structure in the client 410 and the two-dimensional image of the side of the box-type structure at the same time, wherein the depth image and the two-dimensional image are completely aligned, and the coordinates of the pixel points of each area are consistent in the two images) according to the detection results, and display the two-dimensional image marked with the detection results in the human-computer interaction interface of the client 410.

[0077] In some embodiments, the deformation detection method of the box-type structure provided in the embodiment of the present application may also be implemented by the terminal alone.

[0078] For example, Figure 1Taking the terminal 400 shown in as an example, the deformation detection plug-in of the box-type structure can be implanted in the client 410 running in the terminal 400, so as to implement the deformation detection method of the box-type structure locally on the client 410. For example, after obtaining the deformation detection request for the box-type structure, the terminal 400 calls the deformation detection plug-in of the box-type structure to implement the deformation detection method of the box-type structure, by extracting the depth image of the side of the box-type structure carried in the deformation detection request, based on the depth information of the depth image, identifying the areas corresponding to the multiple components included in the side from the depth image, then selecting at least one component from the multiple components, and determining the reference plane corresponding to the at least one component, then, after determining the reference plane, the distance between the pixel points included in each area and the reference plane can be determined, and based on the distance, the deformation area in each area is determined. In this way, the deformation degree of the box-type structure can be automatically detected, which improves the detection efficiency and accuracy.

[0079] In other embodiments, after obtaining the deformation detection request for the box-type structure, the terminal 400 can also call the deformation detection interface of the box-type structure provided by the server 200 (for example, it can be provided in the form of a cloud service, that is, a deformation detection service for the box-type structure). After receiving the deformation detection request, the server 200 first extracts the depth image of the side of the box-type structure to be detected from the deformation detection request, and then, based on the depth information of the depth image, identifies the areas corresponding to the multiple components included in the side from the depth image, and then selects at least one component from the multiple components, and determines the reference plane corresponding to the at least one component. Finally, after determining the reference plane, the distance between the pixels included in each area and the reference plane can be determined, and based on the distance, the deformation area in each area is determined. In this way, the deformation degree of the box-type structure can be automatically detected. For example, when it is detected that the deformation degree of a box-type structure exceeds the deformation degree threshold (that is, it cannot be used anymore), a reminder message can be sent to the inspection personnel to remind the inspection personnel to replace it.

[0080] In some embodiments, the deformation detection method of the box-type structure provided in the embodiments of the present application can be implemented in combination with cloud technology.

[0081] For example, Figure 1The server 200 shown in the figure can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal 400 can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited to this. The terminal 400 and the server 200 can be directly or indirectly connected by wired or wireless communication, which is not limited in the embodiments of the present application.

[0082] Below Figure 1 The structure of the server 200 shown in FIG. Figure 2 , Figure 2 is a schematic diagram of the structure of the server 200 provided in an embodiment of the present application, Figure 2 The server 200 shown includes: at least one processor 210, a memory 240, and at least one network interface 220. The various components in the server 200 are coupled together via a bus system 230. It is understood that the bus system 230 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 230 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, the bus system 230 is not described in detail. Figure 2 Various buses are labeled as bus system 230 .

[0083] The processor 210 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0084] The memory 240 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. The memory 240 may optionally include one or more storage devices that are physically remote from the processor 210.

[0085] The memory 240 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 240 described in the embodiments of the present application is intended to include any suitable type of memory.

[0086] In some embodiments, memory 240 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0087] Operating system 241, including system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0088] A network communication module 242, for reaching other computing devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 include: Bluetooth, Wireless Compatibility Authentication (WiFi), and Universal Serial Bus (USB);

[0089] In some embodiments, the box-type structure deformation detection device provided in the embodiments of the present application can be implemented in a software manner. Figure 2 The box-type structure deformation detection device 243 stored in the memory 240 is shown, which can be software in the form of programs and plug-ins, including the following software modules: acquisition module 2431, identification module 2432, determination module 2433, deletion module 2434, selection module 2435, marking module 2436 and extraction module 2437. These modules are logical, so they can be arbitrarily combined or further split according to the functions implemented. It should be pointed out that in Figure 2 For the sake of convenience, all the above modules are shown at once, but it should not be considered that the box-type structure deformation detection device 243 excludes the implementation of only including the acquisition module 2431, the identification module 2432 and the determination module 2433. The functions of each module will be explained below.

[0090] In other embodiments, the deformation detection device of the box-type structure provided in the embodiments of the present application can be implemented in hardware. As an example, the device provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the deformation detection method of the box-type structure provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs) or other electronic components.

[0091] As mentioned above, the deformation detection method of the box-type structure provided in the embodiment of the present application can be implemented by various types of electronic devices, for example, it can be implemented by a terminal or a server alone, or it can be implemented by a terminal and a server in collaboration. Figure 1 The terminal 400 and the server 200 shown in FIG. 1 cooperate to implement the deformation detection method of the box-type structure provided in the embodiment of the present application as an example for explanation. Figure 3 , Figure 3 is a flow chart of a deformation detection method for a box-type structure provided in an embodiment of the present application, which will be combined with Figure 3 The steps shown are explained.

[0092] In step S101 , a depth image of the side surface of the box-type structure is acquired.

[0093] In some embodiments, the box-type structure may include a container, a packing box, a modular house, etc., and the side of the box-type structure may include the top surface, the side surface, the ground, etc. of the box-type structure. For example, when the deformation degree of the top surface of the box-type structure needs to be detected, the depth image of the top surface of the box-type structure may be acquired; when the deformation degree of the side surface of the box-type structure needs to be detected, the depth image of the side surface of the box-type structure may be acquired.

[0094] For example, taking the acquisition of the depth image of the top surface of the container as an example, the methods for acquiring the depth image can be divided into two categories: passive ranging sensing and active ranging sensing, wherein the passive ranging sensing method is binocular stereo vision, where two images of the same scene are simultaneously acquired through two cameras at a certain distance apart (i.e., the same container to be detected is photographed at the same time through two cameras at a certain distance apart to obtain two images), and the corresponding pixel points in the two images are found through a stereo matching algorithm, and then the parallax information is calculated according to the triangulation principle, and the parallax information can be converted to obtain the depth information used to characterize the objects in the scene (i.e., the depth image of the top surface of the container is obtained, and the depth image includes the depth information of the top surface of the container to be detected). Based on the stereo matching algorithm, the depth image of the scene can also be obtained by shooting a group of images at different angles in the same scene (for example, the top surface of the container to be detected can be photographed from different angles, and the depth image of the top surface of the container can be obtained based on the group of images obtained by shooting). In addition, the depth information of the top surface of the container can be indirectly estimated by analyzing the photometric features, light and dark features and other features of the two-dimensional image of the top surface of the container to be detected.

[0095] For example, continuing from the above, compared with passive ranging sensors, the characteristics of active ranging sensors are: the image acquisition device itself needs to emit energy to complete the acquisition of depth information, which also ensures that the acquisition of depth images is independent of the acquisition of two-dimensional images. Active ranging sensors can include structured light, laser scanning and time of flight (TOF) technology, where structured light refers to light with a specific pattern, which has patterns such as points, lines, and surfaces. The principle of acquiring depth images based on structured light is: projecting structured light onto a scene (such as a container to be detected), and capturing the corresponding pattern with structured light by the image sensor. Since the pattern of structured light will be deformed due to the shape of the container, the depth information of each point in the scene can be obtained by calculating the position of the pattern image in the captured image and the degree of deformation using the triangulation principle (that is, including the depth information of each point on the top surface of the container, and the depth image of the top surface of the container is obtained based on the combination of the depth information of each point). The principle of acquiring depth images based on laser scanning is: emitting lasers to the top surface of the container to be tested at certain time intervals, and recording the time interval between the signal of each scanning point from the laser radar to the top surface of the container to be tested, and then reflecting back to the laser radar through the top surface of the container to be tested, and the distance between each point on the top surface of the container to be tested and the laser radar is calculated based on this. The principle of acquiring depth images based on TOF cameras is: by emitting continuous near-infrared pulses to the top surface of the container to be tested, and then using sensors to receive light pulses reflected back from the top surface of the container to be tested, by comparing the phase difference between the emitted light pulses and the light pulses reflected from the top surface of the container, the transmission delay between the light pulses can be calculated, and then the distance of the top surface of the container to be tested relative to the transmitter can be obtained, and finally the depth image of the top surface of the container can be obtained.

[0096] In some embodiments, the depth image of the side surface of the box-type structure may also be obtained in the following manner: three-dimensional point cloud data for the box-type structure is obtained, and the depth image of the side surface is extracted from the three-dimensional point cloud data.

[0097] For example, taking the box-type structure as a container, the depth image of the top surface of the container can be obtained in the following way: use a laser radar (2D / 3D), a stereo camera or a TOF camera to scan the top surface of the container to be tested to obtain the corresponding three-dimensional point cloud data, wherein the point cloud data file is in the form of a 3D coordinate file (often referred to as an XYZ file), these files are ASCII, and therefore can be read by all post-processing software, for example, in a 3D grayscale file, the grayscale pixel point cloud data is: X1, Y1, Z1, grayscale value 1; X2, Y2, Z2, grayscale value 2, etc. After obtaining the point cloud data obtained by three-dimensional scanning of the top surface of the container to be tested, the depth image of the top surface of the container to be tested can be extracted from the point cloud data.

[0098] In other embodiments, after obtaining a depth image of the side of the box-type structure, a user (e.g., an inspector) can input the obtained depth image into a client running on the terminal (e.g., a deformation detection client of the box-type structure), and the terminal automatically obtains a deformation detection request for the box-type structure (carrying a depth image), and sends the deformation detection request for the box-type structure to the server, so that after receiving the deformation detection request, the server extracts the depth image, detects the deformation degree of the box-type structure based on the depth image, and returns the detection result to the terminal, so that the terminal calls the human-computer interaction interface of the client to present the detection result returned by the server.

[0099] In step S102, based on the depth information of the depth image, regions corresponding to the plurality of components included in the side surface are identified from the depth image.

[0100] In some embodiments, after receiving a deformation detection request for a box-type structure sent by a terminal, the server extracts a depth image from the deformation detection request, and before identifying the areas corresponding to the multiple components included in the side from the depth image based on the depth information of the depth image, the server may also perform the following processing: slide the depth image with a window of a preset size to determine the jump amplitude of the depth value between different pixel points included in each window; determine the area where the window corresponding to the jump amplitude is greater than the amplitude threshold is located as a noise point in the depth image; and delete the determined noise point from the depth image.

[0101] For example, taking the box-type structure as a container, since there may be many noise points in the original depth image, which affects subsequent calculations, after the server extracts the depth image of the top surface of the container from the deformation detection request, it first needs to denoise the depth image. For example, the server can slide a window of a preset size (for example, 50*50 pixels) through the entire depth image, and for the area corresponding to each window in the sliding process, determine the jump amplitude of the depth value between different pixels in the area. Since the depth value of the noise point changes very quickly, the depth value of the noise point will jump repeatedly in a smaller pixel interval, that is, repeat the process of getting smaller-getting bigger-getting smaller, and the jump amplitude is large. Therefore, the area where the window corresponding to the jump amplitude is greater than the amplitude threshold is determined as a noise point in the depth image. After traversing the entire depth image (that is, after determining whether the area where each window is located is a noise point in the depth image), the determined noise point can be deleted from the depth image.

[0102] It should be noted that, in actual applications, the value of the preset size may be related to the size of the depth image. For example, when the size of the depth image is larger, the corresponding value of the preset size may also be larger (for example, when the size of the depth image is 10000*10000 pixels, the corresponding preset size may be 50*50 pixels; when the size of the depth image is 20000*20000 pixels, the corresponding preset size may be 100*100 pixels); of course, the value of the preset size may also be related to the denoising accuracy. For example, when the required denoising accuracy is higher, the corresponding value of the preset size is smaller; when the required denoising accuracy is lower, the corresponding value of the preset size is larger. In other words, the value of the preset size can be flexibly adjusted, and the embodiment of the present application does not specifically limit the value of the preset size.

[0103] In some embodiments, the server can also implement the above-mentioned depth image based on the depth image in the following manner, and identify the areas corresponding to the multiple components included on the side from the depth image: based on the depth image of the depth image and the gradient difference between different pixel points included in the depth image, identify the areas corresponding to the multiple components included on the side from the depth image.

[0104] For example, taking a container as an example, the area corresponding to the corner fittings of the container can be identified from the depth image in the following way: determine the pixels in the depth image whose depth values ​​are less than the depth threshold (i.e., the highest area in the depth image) and whose gradient difference with adjacent pixels is greater than the gradient difference threshold; determine the area in the depth image that is composed of pixels whose depth values ​​are less than the depth threshold and whose gradient difference with adjacent pixels is greater than the gradient difference threshold, and whose corresponding spatial feature is a corner, as the area corresponding to the corner fittings of the container.

[0105] For example, corner fittings are located at the four corners of a container. At the same time, the features of the corner fittings are obvious. In the depth image, the corner fittings are the highest area in the entire image (that is, the area where the depth value is less than a certain depth threshold). Therefore, the positions of the corner fittings can be located at the four corners of the container according to this height feature and its spatial feature (located at the corner).

[0106] For example, taking a container as an example of a box-type structure, the area corresponding to the reinforcement plate of the container can be identified from the depth image in the following manner: based on the area corresponding to the corner fitting, determine the first boundary of the reinforcement plate of the container that coincides with the corner fitting; take at least one pixel point included in the first boundary as the starting point, traverse and determine the gradient of multiple subsequent pixel points; determine the second boundary of the reinforcement plate based on the pixel points corresponding to the position where the gradient difference is greater than the gradient difference threshold; determine the area composed of the first boundary and the second boundary in the depth image as the area corresponding to the reinforcement plate of the container.

[0107] For example, the reinforcement plate surrounds the corner piece, and there is a weld at the junction of the reinforcement plate and the top beam, extension plate and top plate. Therefore, after confirming the corner piece, the partial outer boundary of the reinforcement plate (that is, the boundary overlapping with the corner piece) can be determined. Taking at least one pixel point included in the boundary as the starting point, the gradients of multiple subsequent pixel points are traversed and calculated along the same row or column. At the weld, the gradient will have a sudden change (that is, the gradient difference of the pixel points on both sides of the weld will be greater than the gradient difference threshold). Therefore, based on this feature, the remaining boundary of the reinforcement plate can be determined, and the area corresponding to the reinforcement plate can be identified from the depth image.

[0108] For example, taking a container with a box-type structure as an example, the area corresponding to the side beam of the container can be identified from the depth image in the following way: based on the area corresponding to the reinforcement plate of the container, the left and right boundaries of the side beam of the container are determined; taking at least one pixel point included in the boundary of the depth image as the starting point, traversing and determining the gradient of multiple subsequent pixel points; based on the pixel points corresponding to the positions where the gradient difference is greater than the gradient difference threshold, the upper and lower boundaries of the side beam are determined; the area composed of the left and right boundaries of the side beam and the upper and lower boundaries of the side beam in the depth image is determined as the area corresponding to the side beam of the container.

[0109] For example, taking the top surface of the container as an example, the top side beam is located on the upper and lower sides of the top surface of the container, and the left and right boundaries of the top side beam coincide with the left and right boundaries of the reinforcing plate. The upper boundary of the upper top side beam is the upper boundary of the container, and the lower boundary is connected to the top plate, and there is a weld; similarly, the lower boundary of the lower top side beam is the lower boundary of the container, and the upper boundary is connected to the top plate, and there is a weld. Therefore, when determining the boundary of the top side beam, at least one pixel point included in the upper and lower boundaries of the container can be used as the starting point, and the gradients of multiple pixel points subsequently located in the same column can be traversed to determine. At the upper and lower boundaries, the gradient value will show a large jump. Therefore, the line segment composed of pixel points corresponding to the position where the gradient difference is greater than the gradient difference threshold can be used as the upper and lower boundaries of the top side beam, thereby identifying the corresponding area of ​​the top side beam in the depth image.

[0110] For example, taking a container as a box-type structure, the area corresponding to the beam of the container can be identified from the depth image in the following way: based on the area corresponding to the corner fittings of the container, the upper and lower boundaries of the beam of the container are determined; the interval consisting of the minimum horizontal coordinate and the maximum horizontal coordinate corresponding to the corner fittings is determined, and the gradient of each pixel point in the interval is traversed to determine; based on the pixel points corresponding to the position where the gradient difference is greater than the gradient difference threshold, the left and right boundaries of the beam are determined; the area in the depth image consisting of the upper and lower boundaries of the beam and the left and right boundaries of the beam is determined as the area corresponding to the beam of the container.

[0111] For example, taking the top surface of the container as an example, the top crossbeam is located at the two side edges of the top surface of the container, one on each side, and must be between the corner fittings on both sides. That is, after determining the positions of the four corner fittings, the upper and lower boundaries of the two top crossbeams can be located, and the left and right boundaries of the top crossbeam can be determined based on the depth image. Assuming that the minimum horizontal coordinate of the corner fitting on the corresponding side of the top crossbeam is x1, and the maximum horizontal coordinate is x2, then traverse the pixel points in the interval [x1, x2] composed of x1 and x2, and calculate the gradient of the pixel points. At the left and right boundaries of the top crossbeam, the gradient will show a large jump, and the left and right boundaries of the top crossbeam can be determined, thereby identifying the area corresponding to the top crossbeam in the depth image.

[0112] For example, taking a container as a box-type structure, the area corresponding to the extension plate of the container can be identified from the depth image in the following manner: based on the area corresponding to the crossbeam of the container, determine the boundary of the extension plate of the container that coincides with the crossbeam; based on the area corresponding to the reinforcement plate of the container, determine the upper and lower boundaries of the extension plate; taking at least one pixel point included in the boundary that coincides with the crossbeam as the starting point, traverse and determine the gradient of subsequent multiple pixel points; based on the pixel points corresponding to the position where the gradient difference is greater than the gradient difference threshold, determine the remaining boundary of the extension plate; and determine the area in the depth image composed of the boundary that coincides with the crossbeam, the upper and lower boundaries of the extension plate, and the remaining boundary as the area corresponding to the extension plate of the container.

[0113] For example, after locating the top crossbeam, one side of the top crossbeam is the boundary of the container, and the other side is the extension plate. That is, after locating the top crossbeam, part of the boundary of the extension plate (that is, the boundary overlapping with the top crossbeam) can be determined, and the upper and lower boundaries of the extension plate are the boundaries of the two reinforcing plates on the corresponding sides (that is, after locating the reinforcing plate, the upper and lower boundaries of the extension plate can be determined). In addition, the remaining boundary of the extension plate is connected to the top plate, and there is a weld at the connection, that is, the gradient of the pixel point at the weld will undergo a large jump, then the remaining boundary of the extension plate can be determined based on the gradient difference, so that the position of the extension plate can be identified from the depth image.

[0114] For example, continuing with the box-type structure as a container, after locating the remaining components of the top surface of the container (such as corner pieces, reinforcement plates, extension plates, top cross beams and top side beams), the position of the top plate can be automatically obtained (that is, the remaining area is automatically determined as the area corresponding to the top plate of the container). In this way, the areas corresponding to all the important components included in the container are identified from the depth image.

[0115] In step S103, a reference plane corresponding to at least one component is determined, and a distance between pixels included in each region and the reference plane is determined.

[0116] In some embodiments, a reference plane corresponding to at least one component may be determined in the following manner: selecting a target component of the largest size from multiple components; determining a depth mean of pixels included in an area corresponding to the target component, and determining a reference plane corresponding to the target component based on the depth mean.

[0117] For example, taking a container as a box-type structure, after identifying the areas corresponding to multiple components (such as corner pieces, reinforcing plates, extension plates, top cross beams, top side beams and top plates) included in the top surface of the container from the depth image based on the depth information of the depth image and the gradient difference between different pixel points, the target component (such as the top plate) corresponding to the largest size can be selected from the multiple components, and then the depth mean of all pixel points included in the area corresponding to the target component is calculated, and then the plane corresponding to the calculated depth mean can be determined as the reference plane corresponding to the target component.

[0118] In some other embodiments, the reference plane corresponding to at least one component can also be determined in the following manner: determine the convex groove area and the concave groove area included in the side; determine the first depth mean of the pixels included in the convex groove area, and delete the pixels in the convex groove area whose distance from the first depth mean is greater than the depth mean threshold; determine the second depth mean of the remaining pixels in the convex groove area, and determine the first reference plane corresponding to the convex groove area based on the second depth mean; determine the third depth mean of the pixels included in the concave groove area, and delete the pixels in the concave groove area whose distance from the third depth mean is greater than the depth mean threshold; determine the fourth depth mean of the remaining pixels in the concave groove area, and determine the second reference plane corresponding to the concave groove area based on the fourth depth mean.

[0119] For example, taking a container as an example of a box-type structure, the top plate of the top surface of the container is composed of a corrugated plate, and the corrugated plate includes a raised area (i.e., a convex groove area) and a concave area (i.e., a concave groove area). Therefore, it is necessary to determine the reference planes corresponding to the convex groove area and the concave groove area respectively. For example, taking the convex groove area as an example, firstly, all the pixel points included in the convex groove area are counted, and the depth mean m1 of all the pixel points is calculated. Then, the pixel points whose distance from the depth mean m1 exceeds the depth mean threshold (e.g., 10 mm) among all the pixel points included in the convex groove area are determined and deleted. Subsequently, the depth mean m2 of the remaining pixel points included in the convex groove area is recalculated. The depth mean m2 can be used to represent the height of the reference plane corresponding to the convex groove area. Similarly, similar processing is performed on the concave groove area to obtain the reference plane corresponding to the concave groove area.

[0120] In some embodiments, when there is a specific component whose intensity is greater than the intensity threshold among the multiple components identified in step S102, the following processing can be performed on the specific component: determine the fifth depth mean of the pixel points included in the area corresponding to the specific component; based on the fifth depth mean, determine the specific reference plane corresponding to the specific component; determine the distance between the pixel points included in the specific component and the specific reference plane; based on the distance between the pixel points included in the specific component and the specific reference plane, determine the deformation area in the area corresponding to the specific component.

[0121] For example, taking the box-type structure of a container as an example, the corner fittings are the strongest components in the container (that is, specific components with strength greater than the strength threshold). The deformation of the corner fittings is relatively small, and there is no deformation at most points. Therefore, when calculating the deformation of the corner fittings, the corresponding reference plane is determined by the plane formed by the four corner fittings. For example, for the four corner fittings, unlike other areas of the container, there is a hole in the middle of the corner fittings, and the hole does not participate in the deformation calculation, so it is necessary to locate the position of the hole first; traverse from the edge of the corner fitting to the middle, calculate the maximum gradient in a small size (for example, 5*5 pixels) interval, and the gradient of the pixels around the hole will have a large jump, so the position of the hole can be determined based on the gradient difference of the pixels, and then calculate the depth mean m3 of the remaining pixels included in the area corresponding to the four corner fittings. After obtaining the depth mean m3, the pixels in the area corresponding to the four corner fittings whose distance from the depth mean m3 is greater than the depth mean threshold (for example, 5 mm) can be removed. Subsequently, for the remaining pixels, the depth mean of the remaining pixels is calculated to obtain the depth mean m4, which can be used to represent the height of the reference plane corresponding to the corner fitting. After obtaining the reference plane corresponding to the corner piece, the distance between the pixel points excluding the holes in the area corresponding to the corner piece and the reference plane of the corner piece can be calculated, and the pixel points whose distance exceeds the distance threshold (for example, 8 mm) are determined as deformation points. The deformation points are combined to obtain the deformation area, and the size, coordinates, maximum deformation value (that is, the farthest distance from the reference plane of the corner piece) and confidence level of the deformation area can be further determined (the confidence level is determined by the ratio of the number of noise points included in the deformation area to the total number of pixels in the deformation area).

[0122] In some embodiments, the above-mentioned determination of the distance between the pixel points included in each area and the reference plane can be achieved in the following way: for each area, determine the distance between the depth value of each pixel point included in the area and the reference plane. For example, assuming that the height of the reference plane is 20 mm, the depth value of a pixel point is 18 mm, then the distance between the pixel point and the reference plane is 2 mm.

[0123] In step S104, the deformation region in each region is determined based on the distance between the pixel points included in each region and the reference plane.

[0124] In some embodiments, Figure 3 Step S104 shown may be performed by Figure 4 Steps S1041 to S1042 shown in the figure are implemented by combining Figure 4 The steps shown are explained.

[0125] In step S1041, pixel points in the region whose distances are greater than a distance threshold are determined.

[0126] In some embodiments, after the distance between the pixels included in each region and the reference plane is determined based on step S103, the following processing may be performed for each region: determining the pixels in the region corresponding to the pixels whose distance is greater than the distance threshold.

[0127] For example, taking a container as a box-type structure, after identifying the areas corresponding to the multiple components (such as corner pieces, reinforcing plates, extension plates, top side beams and top cross beams, etc.) included in the top surface of the container from the depth image, and the distance between the pixels included in each area and the reference plane, the pixels corresponding to the distance greater than the distance threshold in each area can be determined. For example, taking the reinforcing plate as an example, after determining the distance between each pixel included in the area corresponding to the reinforcing plate and the reference plane (such as the reference plane corresponding to the top plate, that is, the reference plane determined based on the top plate), the pixels corresponding to the distance greater than the distance threshold can be selected, and then the deformation area can be obtained based on the combination of the selected pixels.

[0128] It should be noted that, in actual applications, for different components included in the side of the box-type structure identified from the depth image, when calculating the degree of deformation of different components, the corresponding distance threshold values ​​can be the same or different. For example, when calculating the degree of deformation of different components, a fixed distance threshold can be selected to determine the pixel points whose distance is greater than the distance threshold; of course, corresponding distance threshold values ​​can also be set for different components. For example, when the strength of a component is relatively large (that is, it is not easy to deform), the corresponding distance threshold value can be smaller; when the strength of a component is relatively small (that is, it is easy to deform), the corresponding distance threshold value can be larger. In other words, the value of the distance threshold can be flexibly adjusted, and the embodiments of the present application do not limit this.

[0129] In step S1042, a region consisting of pixel points whose distances are greater than the distance threshold is determined as a deformation region in the region.

[0130] In some embodiments, after determining the pixel points in the region whose distances are greater than the distance threshold based on step S1041, the region obtained by combining the pixel points whose distances are greater than the distance threshold can be determined as the deformation region in the region.

[0131] Taking a container as an example, the deformation area in the top plate of the container can be determined in the following way: calculate the distance between the depth value of each pixel point included in the area corresponding to the top plate and the reference plane, then select the pixel points corresponding to the distance greater than the distance threshold (for example, 20 mm), and then combine the selected pixel points, and use the combined area as the deformation area in the top plate. In this way, the deformation degree of the container can be automatically detected, which improves the detection efficiency and the accuracy of the detection results.

[0132] In other embodiments, see Figure 5 After executing Figure 3 After step S104 shown, you can also perform Figure 5 Steps S105 to S106 shown in FIG. Figure 5 The steps shown are explained.

[0133] In step S105 , a two-dimensional image of the side surface of the box-type structure is acquired.

[0134] In some embodiments, a two-dimensional image of the side of the box-type structure can be obtained in the following manner: the side of the box-type structure is photographed by a 2D camera to obtain a two-dimensional image of the side of the box-type structure, wherein the two-dimensional image and the depth image are completely aligned, and the coordinates of the pixel points in each area are consistent in the two images.

[0135] Taking a container as an example, a 2D camera (such as a high-definition camera) is used to perform a linear scan on the top surface of the container to obtain a line scan 2D grayscale image of the top surface of the container, wherein the line scan 2D grayscale image is completely aligned with the depth image of the top surface of the container obtained in step S101, and the coordinates of the pixel points in each area are consistent in the two images. In this way, after identifying the areas corresponding to the multiple components included in the top surface of the container and the deformation area of ​​the container based on the depth image, the corresponding positions of the line scan 2D grayscale image can be marked, thereby facilitating the inspection personnel to clearly determine the area where the container is deformed.

[0136] In some other embodiments, the two-dimensional image of the side of the box-type structure may also be obtained in the following manner: obtaining three-dimensional point cloud data for the box-type structure; and extracting the two-dimensional image of the side from the three-dimensional point cloud data.

[0137] For example, taking a container as an example, first obtain three-dimensional point cloud data for the top surface of the container (for example, scan the top surface of the container through a lidar to obtain three-dimensional point cloud data for the top surface of the container), then perform format conversion and extraction operations on the three-dimensional point cloud data to extract a two-dimensional image of the top surface of the container from the three-dimensional point cloud data.

[0138] In step S106, based on the correspondence between the depth image and the two-dimensional image, regions corresponding to the multiple components included in the side surface and the deformation region in each region are marked in the two-dimensional image.

[0139] In some embodiments, the depth image of the side of the box-type structure obtained in step S101 and the two-dimensional image of the side of the box-type structure obtained in step S105 are completely aligned, that is, the coordinates of the pixel points in each area are consistent in the two images. Therefore, after identifying the areas corresponding to the multiple components included in the side and the deformed areas in each area from the depth image based on the depth information of the depth image and the gradient difference between different pixels, the areas corresponding to the multiple components included in the side and the deformed areas in each area can be marked in the two-dimensional image based on the correspondence between the depth image and the two-dimensional image. In this way, by marking the areas where the box-type structure is deformed in the two-dimensional image, the deformed areas of the box-type structure can be intuitively displayed, which facilitates the subsequent processing of the deformed box-type structure by the inspection personnel.

[0140] The deformation detection method of a box-type structure provided in an embodiment of the present application obtains a depth image of the side of the box-type structure, so that the areas corresponding to the multiple components included in the side can be identified from the depth image based on the depth information of the depth image. Then, at least one component can be selected from the multiple components to determine the reference plane corresponding to the at least one component. Subsequently, the deformation area in each area can be determined based on the distance between the pixel points included in each area and the reference plane. In this way, automatic detection of the deformation degree of the box-type structure is achieved, and detection efficiency and detection accuracy are improved.

[0141] Below, taking a container as an example of a box-type structure, an exemplary application of an embodiment of the present application in an actual application scenario is explained.

[0142] In actual application scenarios, the top surface of the container may be concave or convex due to squeezing, impact, etc. For the deformation degree detection of the top surface of the container, the relevant technology is usually through manual measurement, which requires a lot of manpower and leads to high cost. In view of this, the embodiment of the present application provides a deformation detection method of a box-type structure, which is combined with computer vision technology to improve the efficiency and accuracy of the deformation degree detection of the top surface of the container.

[0143] For example, see FIG. 6A to FIG. 6C , Fig. 6A is a line scan 2D schematic diagram of the top surface of a container provided in an embodiment of the present application, Figure 6B is a depth image of the top surface of the container provided in the embodiment of the present application (such as Figure 6BAs shown, different areas have different corresponding depth values. For example, the depth value of the pixel points included in the gray area 601 is greater than the depth value of the pixel points included in the white area 602, that is, the distance between the gray area 601 and the image acquisition device is greater than that of the white area 602). Figure 6B The depth image shown is acquired by a line scan 3D camera, and Figure 6B The depth image shown is Fig. 6A The line scan 2D schematic diagram shown is completely aligned, that is, the pixel coordinates of each area are consistent in the two images. The deformation detection method of the box-type structure provided in the embodiment of the present application can accurately output the specific location and deformation degree information of the deformation of the top surface of the container, for example Figure 6C The solid line box 603 shown in FIG. 6 represents the area where the deformation of the top surface of the container is relatively large.

[0144] The following is a detailed description of the deformation detection method of the box-type structure provided in the embodiment of the present application. Regarding the problem of detecting the deformation degree of the top surface of the container, the applicant conducted a detection practice on the containers transported into the container yard. In the specific practice process, the following two difficulties were found:

[0145] 1) When the container is transported to the yard, the image formed by the line scan camera will be distorted due to uneven road surface and turning, and the depth values ​​at the beginning and end of the container's travel direction will change greatly;

[0146] 2) The container deformation degree detection methods provided by related technologies basically require instruments such as laser ranging or metal flaw detectors, which leads to high costs, and require the container to remain stationary. Containers and detection instruments must be placed in specific sites, which has high requirements. In actual production, there may not be so many sites available. In other words, the solutions provided by related technologies are relatively demanding and cannot meet the detection needs of real scenarios.

[0147] In actual production, the applicant also discovered the following two facts regarding the containers in the container yard:

[0148] 1) The free area in the yard is small and the business is busy. If the deformation is detected after the truck stops, it will affect the normal operation of the yard. Therefore, the best way is to complete the imaging and detection while the container is transported into the yard without stopping;

[0149] 2) Although the reliability of the detection results output by the algorithm is relatively high, it does not reach 100%. For actual production, a small amount of manual inspection is still required.

[0150] In view of this, in order to support project development and ensure the reliability of detection results, the embodiment of the present application provides a deformation detection method for a box-type structure, which can automatically detect the deformation degree of the top surface of the container to improve detection efficiency and reduce detection costs.

[0151] For example, see Figure 7 , Figure 7 is a flow chart of a deformation detection method for a box-type structure provided in an embodiment of the present application, such as Figure 7 As shown, the deformation detection method of the box-type structure provided in the embodiment of the present application mainly includes five steps: input, denoising, area positioning, positioning reference plane and deformation calculation. The above five steps are specifically described below.

[0152] (I) Input: First, obtain the image that needs to be deformed. The image is the top surface image of the container, for example, including Fig. 8A The line scan 2D schematic diagram shown in Figure 8B The depth image shown (such as Figure 8B As shown, different areas correspond to different depth values, for example, the depth value corresponding to the gray area 801 is greater than the depth value corresponding to the white area 802), and the top surface of the container can be further divided into six parts: corner pieces, reinforcement plates, extension plates, top cross beams, top side beams and top plates, wherein the top plate is composed of corrugated plates, and the corrugated plates include groove areas and convex groove areas.

[0153] (ii) Denoising: traverse the depth image and remove the noise points to ensure the accuracy of subsequent calculations.

[0154] In some embodiments, there are many noise points in the original depth image, which affects subsequent calculations. Therefore, the depth image needs to be denoised first. In actual situations, whether it is a deformation point or a junction of components, the changes in depth values ​​between adjacent different pixel points are regular, generally changing slowly, and some have a large change, while the depth value of the noise point changes very quickly. In a smaller pixel interval, the depth value of the noise point will jump repeatedly, that is, repeat the process of getting bigger-getting smaller-getting bigger, and the amplitude is large. Therefore, according to this rule, the entire depth image can be slid through to remove the noise points that jump repeatedly in a small interval (for example, 50*50 pixels).

[0155] (iii) Area positioning: According to the depth values ​​and gradient differences of each area in the depth image, the corner pieces, reinforcement plates, extension plates, top cross beams, top side beams and top plates are positioned and marked accordingly in the line scan 2D grayscale image.

[0156] In some embodiments, the top surface of the container includes six areas, namely corner fittings, reinforcement plates, extension plates, top cross beams, top side beams and top plates, wherein the corner fittings, reinforcement plates, extension plates and top cross beams are distributed at the left and right ends of the top surface of the container, with two corner fittings, two reinforcement plates, one extension plate and one top cross beam at each end, and the top side beams are distributed above and below the top surface of the container, with a total of two, and the remaining area is the top plate.

[0157] The positioning of the corner pieces and the reinforcing plates will first be described below.

[0158] The corner fittings and reinforcement plates are located at the four corners of the top surface of the container. The features of the corner fittings are very obvious. In the depth image, the corner fittings are the highest area in the entire image. Therefore, the positions of the corner fittings can be located at the four corners of the top surface of the container according to this height feature and its spatial feature (located in the corner).

[0159] The reinforcing plate surrounds the corner fitting, and there is a weld at the junction of the reinforcing plate and the top beam, extension plate and top plate. After confirming the corner fitting, the partial outer boundary of the reinforcing plate can be determined. Taking any one or more pixel points included in the partial outer boundary as the starting point, the gradient of each pixel point is traversed and calculated. At the weld, the gradient will have a sudden change. Based on this feature, the remaining boundary of the reinforcing plate can be determined, and the positioning of the corner fitting and the reinforcing plate is completed.

[0160] The positioning of the extension plate and the top crossbeam is described below.

[0161] The top crossbeam is located at the two side edges of the top surface of the container, one on each side, and must be between the corner fittings on both sides. After determining the positions of the four corner fittings, the upper and lower boundaries of the two top crossbeams can be located, and the left and right boundaries of the top crossbeam can be determined based on the depth image. Assuming that the minimum horizontal coordinate of the corner fitting on the corresponding side of the top crossbeam is x1, and the maximum horizontal coordinate is x2, traverse the pixel points in the interval [x1, x2], calculate the gradient of each pixel point, and at the left and right boundaries of the top crossbeam, the gradient will show a large jump, and the left and right boundaries of the top crossbeam can be determined, thereby obtaining the accurate position of the top crossbeam.

[0162] One side of the top crossbeam is the boundary of the container, and the other side is the extension plate. The upper and lower boundaries of the extension plate are the boundaries of the two reinforcing plates on the corresponding sides. The remaining boundary of the extension plate is connected to the top plate, and there is a weld. Therefore, the remaining boundary of the extension plate can also be determined based on the gradient difference, that is, the area corresponding to the extension plate is positioned.

[0163] The positioning of the top side beams and the top plate is described below.

[0164] The top side beam is located on the upper and lower sides of the top surface of the container, and its left and right boundaries are the left and right boundaries of the reinforcement plate. The upper boundary of the upper top side beam is the upper boundary of the container, and the lower boundary is connected to the top plate, and there is a weld. The lower boundary of the lower top side beam is the lower boundary of the container, and the upper boundary is connected to the top plate, and there is a weld. Then the positioning method is the same as that of the top cross beam. By traversing the pixel points and calculating the gradient of each pixel point, the gradient will show a large jump at the upper and lower boundaries, and the position of the top side beam can be located.

[0165] After locating the remaining components of the container top surface, the top plate position can be automatically obtained. The top plate is located between the two extension plates and the two top side beams. Next, in order to facilitate the determination of the two concave and convex reference planes, the concave and convex grooves of the top plate need to be subdivided and each concave and convex groove needs to be located. In actual processing, the top plate is traversed from left to right. The first and last areas must be concave grooves, and the boundary between the concave groove and the convex groove has a large gradient jump, so the position of each concave and convex groove can be determined.

[0166] For example, see Fig. 9 , Fig. 9 Schematic diagram of the region segmentation result of the container top surface provided in the embodiment of the present application. Fig. 9 As shown in FIG. 1 , after locating the corner pieces, reinforcing plates, extension plates, top side beams, top cross beams, and top plates based on the depth information of the depth image and the gradient differences between different pixel points, it is possible to Fig. 9 Corresponding positions of the line scan 2D schematic diagram shown are marked, such as the corner fittings 901 located at the four corners of the container, the reinforcing plates 902 surrounding the corner fittings 901, the top cross beams 904 located at the left and right edges of the top surface of the container, the extension plates 903 connected to the top cross beams 904, the top side beams 905 located at the upper and lower edges of the top surface of the container, and the top plate 906 outside the corner fittings 901, the reinforcing plates 902, the extension plates 903, the top cross beams 904 and the top side beams 905 on the top surface of the container.

[0167] (IV) Locating the reference plane: Calculating the deformation degree of the top surface of the container is actually calculating the distance between the top surface of the container and the reference plane. Except for the corner fittings, the top plate groove plane is used as the reference plane for the depression (that is, it is compared with the reference plane when calculating the depression deformation), and the top plate convex groove plane is used as the reference plane for the protrusion (that is, it is compared with the reference plane when calculating the protrusion deformation). In this way, based on the segmented area, the two reference planes of the container protrusion and depression can be found in the depth image; the corner fittings are the strongest parts of the container, and the deformation of the corner fittings is relatively small. Most points do not have deformation. Therefore, when calculating the deformation of the corner fittings, the plane formed by the four corner fittings is used as the reference plane.

[0168] In some embodiments, taking the calculation of the deformation of the top surface as an example, two reference planes need to be located first, and the plane equation is ax+by+cz=D. Considering the characteristics of the top surface line scan camera, a and b are both 0, and c is 1, then the equation becomes z=D, and the height of the reference plane can be obtained to locate the reference plane. The specific steps are:

[0169] For the top plate convex groove area {X1, X2, ..., X n}, each convex groove area can be further divided into three sub-areas, namely, the first sub-area is the one whose coordinate difference from the lower boundary y1 of the upper top side beam is less than the first preset difference (for example, 20 pixels), the second sub-area is the one whose coordinate difference from the upper boundary y2 of the lower top side beam is less than 20 pixels, and the remaining area is the third sub-area. The depth mean of the first sub-area in all convex groove areas is calculated, and then the area composed of pixels whose difference from the mean exceeds a preset threshold (for example, 10 mm) is excluded, and the mean is calculated again for the remaining area to obtain the depth mean m1 corresponding to the first sub-area of ​​the convex groove area. Similarly, the depth mean m2 corresponding to the second sub-area is obtained, and then the mean m of m1 and m2 is taken. The mean m represents the height of the reference plane corresponding to the convex groove area; the same operation is performed on the groove area to locate the height of the reference plane corresponding to the groove area.

[0170] As for the four corner fittings, unlike other parts on the top surface of the container, there is a hole in the middle of the corner fittings, which does not participate in the deformation calculation. Therefore, it is necessary to locate the position in space first, traverse from the edge of the corner fitting to the middle, and calculate the maximum gradient value within the preset size (for example, 5*5 pixels). The gradient around the hole will jump once, so the position of the hole can be located. Then calculate the depth mean of the remaining points of the four corner fittings. After obtaining the mean, remove the pixels whose difference with the mean is greater than the preset threshold (for example, 5 mm), and then recalculate the mean for the remaining pixels. This value is the height of the corner fitting reference plane. Since z=D, the corner fitting reference plane can be obtained.

[0171] (V) Deformation calculation: Traverse all areas of the top surface of the container and calculate the degree of deformation for each area. For example, for a pixel point in a certain area, the point with a large distance from the reference plane (for example, greater than the distance threshold) is the deformation point, and all deformation points in a single area are combined into a deformation area to obtain the final output result, and the detection confidence of the area is output at the same time.

[0172] In some embodiments, the confidence level needs to be outputted while calculating the deformation. For different regions of the top surface, a confidence level is calculated for each region (for the top plate region, a confidence level is calculated for each small concave-convex groove); the calculation method is the same, and the confidence level is determined by the ratio of the number of noise points in the region to the total number of pixels included in the region. Assuming that the ratio of the number of noise points to the total number of pixels is i, the confidence level is 1-2*i. The more noise points there are, the lower the corresponding confidence level is, and the lowest is 0. The deformation calculation methods for different regions are different, as follows:

[0173] 1) Corner fittings: After locating the position of the corner fittings and the corner fitting reference plane, calculate the distance between the pixel points in the corner fittings except the holes and the corner fitting reference plane, regard the pixel points whose distance exceeds the distance threshold (for example, 8 mm) as deformation points, combine the deformation points into a deformation area, and give the size, coordinates, maximum deformation value and confidence of the deformation area.

[0174] 2) Reinforcement plate: Calculate the distance between each pixel point included in the reinforcement plate and the reference plane corresponding to the convex groove area, regard the pixel points that exceed the reference plane and the distance greater than the distance threshold (for example, 20 mm) as deformation points, combine the deformation points into a deformation area (i.e., convex deformation), and give the size, coordinates, maximum deformation value and confidence of the deformation area; then, calculate the distance between each pixel point included in the reinforcement plate and the reference plane corresponding to the groove area, regard the pixel points that are lower than the reference plane and the distance greater than 20 mm as deformation points, combine the deformation points into a deformation area (i.e., concave deformation), and give the size, coordinates, maximum deformation value and confidence of the deformation area.

[0175] 3) Extension plate: Calculate the distance between each pixel point included in the extension plate and the reference plane corresponding to the convex groove area, and regard the pixel points that exceed the reference plane and the distance is greater than the distance threshold (for example, 35 mm) as deformation points, and combine the deformation points into a deformation area (i.e., convex deformation), and give the size, coordinates, maximum deformation and confidence of the deformation area; Then, calculate the distance between each pixel point included in the extension plate and the reference plane corresponding to the groove area, and regard the pixel points that exceed the reference plane and the distance is greater than 35 mm as deformation points, and combine the deformation points into a deformation area (i.e., concave deformation), and give the size, coordinates, maximum deformation and confidence of the deformation area.

[0176] 4) Top crossbeam: Calculate the distance between each pixel point included in the top crossbeam and the reference plane corresponding to the convex groove area, regard the pixel points that exceed the reference plane and the distance is greater than the distance threshold (for example, 20 mm) as deformation points, combine the deformation points into a deformation area (i.e., convex deformation), and give the size, coordinates, maximum deformation value and confidence of the deformation area; then, calculate the distance between each pixel point included in the top crossbeam and the reference plane corresponding to the groove area, regard the pixel points that exceed the reference plane and the distance is greater than 20 mm as deformation points, combine the deformation points into a deformation area (i.e., concave deformation), and give the size, coordinates, maximum deformation value and confidence of the deformation area.

[0177] 5) Top side beam: Calculate the distance between each pixel point included in the top side beam and the reference plane corresponding to the convex groove area, regard the pixel points that exceed the reference plane and the distance greater than the distance threshold (for example, 30 mm) as deformation points, combine the deformation points into a deformation area (i.e., convex deformation), and give the size, coordinates, maximum deformation value and confidence of the deformation area; then, calculate the distance between each pixel point included in the top side beam and the reference plane corresponding to the groove area, regard the pixel points that exceed the reference plane and the distance greater than 30 mm as deformation points, combine the deformation points into a deformation area (i.e., concave deformation), and give the size, coordinates, maximum deformation value and confidence of the deformation area.

[0178] 6) Top plate: Calculate the distance between each pixel point included in the top plate and the reference plane corresponding to the convex groove area, regard the pixel points that exceed the reference plane and the distance greater than the distance threshold (for example, 20 mm) as deformation points, combine the deformation points into a deformation area (i.e., convex deformation), and give the size, coordinates, maximum deformation value and confidence of the deformation area; then, calculate the distance between each pixel point included in the top plate and the reference plane corresponding to the groove area, regard the pixel points that exceed the reference plane and the distance greater than 20 mm as deformation points, combine the deformation points into a deformation area (i.e., concave deformation), and give the size, coordinates, maximum deformation value and confidence of the deformation area.

[0179] It should be noted that in actual production, for further confirmation, the low confidence area given by the system (for example, the area corresponding to the confidence level less than 0.8) can be manually reviewed to ensure that there are no problems.

[0180] For example, see Fig.10 and Fig.11 , Fig.10 It is a flow chart of a container top surface deformation detection method provided by related technology. Fig.11 is a flow chart of the container top surface deformation detection method provided in the embodiment of the present application. By comparing Fig.10 and Fig.11It can be seen that the deformation detection method provided in the embodiment of the present application incorporates computer vision technology, assisted by less manual work, and the results in actual production show that Fig.10 Compared with the deformation detection method shown in the figure, the deformation detection method provided in the embodiment of the present application can obtain the detection result faster and better.

[0181] The deformation detection method of a box-type structure provided in an embodiment of the present application first denoises the depth image, then locates the positions of the multiple components (such as corner pieces, reinforcing plates, extension plates, top side beams, top cross beams and top plates) included in the top surface of the container based on the depth information of the denoised depth image, then locates the reference planes corresponding to the groove area and the convex groove area, and finally calculates the deformation degree of each area based on the reference plane to confirm the deformation area. In this way, the deformation degree of the top surface of the container is automatically detected by using an algorithm, thereby reducing labor costs.

[0182] The beneficial effects of the deformation detection method of the box-type structure provided in the embodiment of the present application are further described below in combination with the deformation detection results in actual production.

[0183] For example, see FIG. 12A to FIG. 12C , Fig. 12A is a line scan 2D schematic diagram of the top surface of the container to be inspected provided in an embodiment of the present application, Fig. 12B is a depth image of the top surface of the container to be detected provided by the embodiment of the present application (such as Fig. 12B As shown, different areas have different depth values, for example, the depth value corresponding to the gray area 1201 is greater than the depth value corresponding to the white area 1202). Fig. 12B The depth image shown in FIG. 1 is used to detect the deformation degree of the top surface of the container by calling the deformation detection method of the box-type structure provided in the embodiment of the present application, and the following is obtained: Fig. 12C The deformation detection results shown in Figure 2 are as follows: Fig. 12C The solid line box 1203 shown represents the area of ​​the container top surface where the deformation is relatively large.

[0184] For example, see FIG. 13A to FIG. 13C , Fig.13A is a line scan 2D schematic diagram of the top surface of the container to be inspected provided in an embodiment of the present application, Fig. 13B is a depth image of the top surface of the container to be detected provided by the embodiment of the present application (such as Fig. 13B As shown, different areas have different depth values, for example, the depth value corresponding to the gray area 1301 is greater than the depth value corresponding to the white area 1302). Fig. 13B The depth image shown in FIG. 1 is used to detect the deformation degree of the top surface of the container by calling the deformation detection method of the box-type structure provided in the embodiment of the present application, and the following is obtained: Fig. 13C The deformation detection results shown in Figure 2 are as follows: Fig. 13C The solid line box 1303 shown represents the area of ​​the container top surface with a larger deformation. FIG. 13A to FIG. 13C It can be seen that even if the collected container image itself is severely distorted, the deformation detection method of the box-type structure provided in the embodiment of the present application can still obtain a relatively ideal deformation detection result.

[0185] It should be noted that the deformation detection method of the box-type structure provided in the embodiment of the present application can not only be applied to the deformation detection of the top surface of the container in the yard, but also can be used to detect the overall condition of the top surface of the container just produced by the factory to ensure good product quality.

[0186] In addition, the deformation detection method of the box-type structure provided in the embodiment of the present application may not be combined with the 2D information of the top surface of the container to be detected, but directly collects the three-dimensional point cloud data of the top surface of the container (for example, a 3D point cloud map), and uses 3D data processing technology to detect the deformation degree of the top surface of the container. For example, first collect the three-dimensional point cloud data for the top surface of the container, then extract the depth image of the top surface of the container from the three-dimensional point cloud data, and determine the area where the top surface of the container is deformed based on the depth information of the depth image, and then extract the two-dimensional image of the top surface of the container from the three-dimensional point cloud data, and mark it in the two-dimensional image based on the determined deformation area.

[0187] The following is a description of an exemplary structure of a box-type deformation detection device 243 provided in an embodiment of the present application implemented as a software module. In some embodiments, for example, Figure 2 As shown, the software modules stored in the box-type structure deformation detection device 243 of the memory 240 may include: an acquisition module 2431 , an identification module 2432 and a determination module 2433 .

[0188] The acquisition module 2431 is used to acquire the depth image of the side of the box-type structure; the identification module 2432 is used to identify the areas corresponding to the multiple components included in the side from the depth image based on the depth information of the depth image; the determination module 2433 is used to determine the reference plane corresponding to at least one component, and determine the distance between the pixels included in each area and the reference plane; the determination module 2433 is also used to determine the deformation area in each area based on the distance between the pixels included in each area and the reference plane.

[0189] In some embodiments, the determination module 2433 is also used to slide through the depth image with a window of preset size to determine the jump amplitude of the depth value between different pixel points included in each window; the area where the window corresponding to the jump amplitude is greater than the amplitude threshold is located is determined as a noise point in the depth image; the box-type structure deformation detection device 243 also includes a deletion module 2434, which is used to delete noise points from the depth image.

[0190] In some embodiments, the identification module 2432 is further used to identify regions corresponding to the multiple components included in the side from the depth image based on the depth information of the depth image and the gradient difference between different pixels included in the depth image.

[0191] In some embodiments, the determination module 2433 is also used to determine the pixels in the depth image whose depth values ​​are less than the depth threshold and whose gradient difference with the adjacent pixels is greater than the gradient difference threshold; and determine the area in the depth image composed of pixels whose depth values ​​are less than the depth threshold and whose gradient difference with the adjacent pixels is greater than the gradient difference threshold, and whose corresponding spatial features are corners, as the area corresponding to the corner piece included in the side.

[0192] In some embodiments, the determination module 2433 is also used to determine the first boundary of the reinforcing plate included on the side that coincides with the corner piece based on the area corresponding to the corner piece included on the side; taking at least one pixel point included in the first boundary as the starting point, traversing and determining the gradient of multiple subsequent pixel points; determining the second boundary of the reinforcing plate based on the pixel points corresponding to the position where the gradient difference is greater than the gradient difference threshold; and determining the area composed of the first boundary and the second boundary in the depth image as the area corresponding to the reinforcing plate included on the side.

[0193] In some embodiments, the determination module 2433 is also used to determine the left and right boundaries of the side beam included in the side based on the area corresponding to the reinforcing plate included in the side; take at least one pixel point included in the boundary of the depth image as the starting point, traverse and determine the gradient of subsequent multiple pixel points; determine the upper and lower boundaries of the side beam based on the pixel points corresponding to the positions where the gradient difference is greater than the gradient difference threshold; and determine the area in the depth image composed of the left and right boundaries of the side beam and the upper and lower boundaries of the side beam as the area corresponding to the side beam included in the side.

[0194] In some embodiments, the determination module 2433 is also used to determine the upper and lower boundaries of the beam included in the side based on the area corresponding to the corner piece included in the side; determine the interval composed of the minimum horizontal coordinate and the maximum horizontal coordinate corresponding to the corner piece, and traverse to determine the gradient of each pixel point in the interval; determine the left and right boundaries of the beam based on the pixel points corresponding to the position where the gradient difference is greater than the gradient difference threshold; determine the area in the depth image composed of the upper and lower boundaries of the beam and the left and right boundaries of the beam as the area corresponding to the beam included in the side.

[0195] In some embodiments, the determination module 2433 is also used to determine the boundary of the extension plate included in the side that coincides with the crossbeam based on the area corresponding to the crossbeam included in the side; determine the upper and lower boundaries of the extension plate based on the area corresponding to the reinforcing plate included in the side; take at least one pixel point included in the boundary that coincides with the crossbeam as the starting point, traverse and determine the gradient of subsequent multiple pixel points; determine the remaining boundary of the extension plate based on the pixel points corresponding to the position where the gradient difference is greater than the difference threshold; determine the area in the depth image consisting of the boundary that coincides with the crossbeam, the upper and lower boundaries of the extension plate and the remaining boundary as the area corresponding to the extension plate included in the side; and determine the area in the side except for the corner piece, reinforcing plate, crossbeam, side beam and extension plate as the area corresponding to the side panel included in the side.

[0196] In some embodiments, the box-type structure deformation detection device 243 also includes a selection module 2435 for selecting a target component with the largest size from multiple components; the determination module 2433 is also used to determine the depth mean of the pixel points included in the area corresponding to the target component, and determine the reference plane corresponding to the target component based on the depth mean.

[0197] In some embodiments, the determination module 2433 is also used to determine the convex groove area and the concave groove area included in the side; determine the first depth mean of the pixels included in the convex groove area, and delete the pixels in the convex groove area whose distance from the first depth mean is greater than the depth mean threshold; determine the second depth mean of the remaining pixels in the convex groove area, and determine the first reference plane corresponding to the convex groove area based on the second depth mean; determine the third depth mean of the pixels included in the concave groove area, and delete the pixels in the concave groove area whose distance from the third depth mean is greater than the depth mean threshold; determine the fourth depth mean of the remaining pixels in the concave groove area, and determine the second reference plane corresponding to the concave groove area based on the fourth depth mean.

[0198] In some embodiments, the determination module 2433 is also used to perform the following processing for each region: determining the pixel points in the region corresponding to a distance greater than a distance threshold; determining the region composed of pixel points corresponding to a distance greater than the distance threshold as a deformation region in the region; and determining the confidence of the deformation region based on the ratio between the number of noise points included in the deformation region and the total number of pixel points included in the deformation region.

[0199] In some embodiments, when there is a specific component among multiple components whose intensity is greater than the intensity threshold, the determination module 2433 is also used to determine the fifth depth mean of the pixel points included in the area corresponding to the specific component; based on the fifth depth mean, determine the specific reference plane corresponding to the specific component; and to determine the distance between the pixel points included in the specific component and the specific reference plane; based on the distance between the pixel points included in the specific component and the specific reference plane, determine the deformation area in the area corresponding to the specific component.

[0200] In some embodiments, the acquisition module 2431 is also used to acquire a two-dimensional image of the side of the box-type structure; the deformation detection device 243 of the box-type structure also includes a labeling module 2436, which is used to label the areas corresponding to the multiple components included in the side and the deformation area in each area in the two-dimensional image based on the correspondence between the depth image and the two-dimensional image.

[0201] In some embodiments, the acquisition module 2431 is also used to acquire three-dimensional point cloud data for the box-type structure; the deformation detection device 243 of the box-type structure also includes an extraction module 2437, which is used to extract a two-dimensional image of the side from the three-dimensional point cloud data; and is used to extract a depth image of the side from the three-dimensional point cloud data.

[0202] It should be noted that the description of the device in the embodiment of the present application is similar to the description of the method embodiment above, and has similar beneficial effects as the method embodiment, so it will not be repeated. Figure 3-5 , Figure 7 ,or Fig.11 The present invention can be understood by referring to the description of any of the accompanying drawings.

[0203] The embodiment of the present application provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the deformation detection method of the box-type structure described in the embodiment of the present application.

[0204] The present application embodiment provides a computer-readable storage medium storing executable instructions, wherein the executable instructions are stored. When the executable instructions are executed by a processor, the processor will be caused to execute the method provided by the present application embodiment, for example, Figure 3-5 , Figure 7 ,or Fig.11 A deformation detection method for a box-type structure is shown.

[0205] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface storage, optical disk, or CD-ROM; or it may be various devices including one or any combination of the above memories.

[0206] In some embodiments, executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0207] As an example, executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0208] By way of example, executable instructions may be deployed to be executed on one computing device, or on multiple computing devices located at one site, or on multiple computing devices distributed across multiple sites and interconnected by a communication network.

[0209] To summarize, the embodiment of the present application acquires a depth image of the side of the box-type structure, so that the areas corresponding to the multiple components included in the side can be identified from the depth image based on the depth information of the depth image. Then, at least one component can be selected from the multiple components to determine the reference plane corresponding to the at least one component. Subsequently, the deformation area in each area can be determined based on the distance between the pixels included in each area and the reference plane. In this way, automatic detection of the deformation degree of the box-type structure is achieved, thereby improving detection efficiency and detection accuracy.

[0210] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A deformation detection method for a box-type structure, It is characterized in that The method comprises: Acquire a depth image of the top surface of the box-like structure; Based on the depth information of the depth image and the gradient difference between different pixel points included in the depth image, identifying from the depth image the regions corresponding to the multiple components included in the top surface, the multiple components at least including: a top plate composed of a corrugated plate, a corner piece having a strength greater than a strength threshold, and a reinforcing plate; Wherein, the corrugated plate includes a convex groove area and a concave groove area, the corner piece includes a cavity, the reinforcing plate surrounds the corner piece, and a weld exists at the junction of the reinforcing plate and the top cross beam, the extension plate and the top plate of the box-type structure; The identifying, from the depth image, areas corresponding to the multiple components included in the top surface based on the depth information of the depth image and the gradient difference between different pixels included in the depth image, includes: for the reinforcing plate, based on the area corresponding to the corner piece included in the top surface, determining a first boundary where the reinforcing plate overlaps with the corner piece, taking at least one pixel point included in the first boundary as a starting point, traversing and determining the gradients of subsequent multiple pixel points, determining a second boundary of the reinforcing plate based on pixel points corresponding to positions where the gradient difference is greater than a gradient difference threshold, and determining the area composed of the first boundary and the second boundary in the depth image as the area corresponding to the reinforcing plate; Determine the distance between each pixel point included in the area corresponding to the reinforcing plate and the reference plane corresponding to the reinforcing plate, and determine the deformation area in the area corresponding to the reinforcing plate based on the pixel points corresponding to the distance greater than the distance threshold in the area corresponding to the reinforcing plate; For the top plate, determining the convex groove area and the concave groove area included in the top plate, respectively determining a first reference plane corresponding to the convex groove area and a second reference plane corresponding to the concave groove area, and determining a deformation area in the convex groove area based on a distance between a pixel point included in the convex groove area and the first reference plane, and determining a deformation area in the concave groove area based on a distance between a pixel point included in the concave groove area and the second reference plane; The identifying, from the depth image, regions corresponding to the plurality of components included in the top surface based on the depth information of the depth image and the gradient difference between different pixels included in the depth image, comprises: for the corner piece, determining the pixel points in the depth image whose depth value is less than the depth threshold and whose gradient difference with the adjacent pixel points is greater than the gradient difference threshold, and determining the region in the depth image composed of the pixel points whose depth value is less than the depth threshold and whose gradient difference with the adjacent pixel points is greater than the gradient difference threshold, and whose corresponding spatial feature is a corner, as the region corresponding to the corner piece included in the top surface; Traverse from the edge of the corner piece to the middle of the corner piece, determine the position of the hole based on the gradient difference of pixel points within a preset interval of the corner piece, determine the depth mean of the pixel points in the area corresponding to the corner piece excluding the hole, delete the pixel points in the area corresponding to the corner piece whose distance from the depth mean is greater than a depth mean threshold, determine the depth mean of the remaining pixel points in the corner piece, and determine the reference plane corresponding to the corner piece based on the depth mean of the remaining pixel points in the corner piece, and determine the deformation area in the area corresponding to the corner piece based on the distance between the pixel points in the area corresponding to the corner piece excluding the hole and the reference plane corresponding to the corner piece.

2. The method according to claim 1, It is characterized in that Before identifying regions corresponding to the multiple components included in the top surface from the depth image based on the depth information of the depth image and the gradient difference between different pixels included in the depth image, the method further includes: Sliding the depth image with a window of a preset size to determine the jump amplitude of the depth value between different pixel points included in each window; Determine the area where the window corresponding to the jump amplitude being greater than the amplitude threshold is located as a noise point in the depth image; The noise is deleted from the depth image.

3. The method according to claim 1, It is characterized in that The plurality of components further includes side beams; The identifying, from the depth image, regions corresponding to the multiple components included in the top surface based on the depth information of the depth image and the gradient difference between different pixels included in the depth image, comprises: Determine left and right boundaries of the side beam included on the top surface based on the area corresponding to the reinforcing plate included on the top surface; Taking at least one pixel point included in the boundary of the depth image as a starting point, traversing and determining the gradients of subsequent multiple pixel points; Determining upper and lower boundaries of the side beam based on pixel points corresponding to positions where the gradient difference is greater than a gradient difference threshold; An area consisting of the left and right boundaries of the side beam and the upper and lower boundaries of the side beam in the depth image is determined as an area corresponding to the side beam included in the top surface.

4. The method according to claim 1, It is characterized in that The plurality of components further includes a beam; The identifying, from the depth image, regions corresponding to the multiple components included in the top surface based on the depth information of the depth image and the gradient difference between different pixels included in the depth image, comprises: Determine the upper and lower boundaries of the beam included in the top surface based on the area corresponding to the corner piece included in the top surface; Determine an interval consisting of the minimum horizontal coordinate and the maximum horizontal coordinate corresponding to the corner piece, and traverse to determine the gradient of each pixel point in the interval; Determining left and right boundaries of the beam based on pixel points corresponding to positions where the gradient difference is greater than a gradient difference threshold; An area consisting of the upper and lower boundaries of the beam and the left and right boundaries of the beam in the depth image is determined as an area corresponding to the beam included in the top surface.

5. The method according to claim 1, It is characterized in that The plurality of components further includes an extension plate; The identifying, from the depth image, regions corresponding to the multiple components included in the top surface based on the depth information of the depth image and the gradient difference between different pixels included in the depth image, comprises: Based on the area corresponding to the crossbeam included in the top surface, determining the boundary of the extension plate included in the top surface that coincides with the crossbeam; Determine the upper and lower boundaries of the extension plate based on the area corresponding to the reinforcement plate included in the top surface; Taking at least one pixel point included in the boundary coinciding with the crossbeam as a starting point, traversing and determining the gradients of a plurality of subsequent pixel points; Determining the remaining boundary of the extension plate based on the pixel points corresponding to the positions where the gradient difference is greater than the difference threshold; Determine the area in the depth image consisting of the boundary coinciding with the crossbeam, the upper and lower boundaries of the extension plate, and the remaining boundary as the area corresponding to the extension plate included in the top surface; The method further comprises: The area of ​​the top surface excluding the corner pieces, the reinforcing plates, the cross beams, the side beams and the extension plates is determined as the area corresponding to the top plate included in the top surface.

6. The method according to claim 1, It is characterized in that The step of respectively determining a first reference plane corresponding to the convex groove area and a second reference plane corresponding to the concave groove area comprises: Determine a first depth mean value of the pixels included in the convex groove area, and delete the pixels in the convex groove area whose distance from the first depth mean value is greater than a depth mean threshold; Determine a second depth mean of the remaining pixels in the convex groove area, and determine a first reference plane corresponding to the convex groove area based on the second depth mean; Determine a third depth mean of the pixels included in the groove area, and delete the pixels in the groove area whose distance from the third depth mean is greater than the depth mean threshold; A fourth depth average of the remaining pixels in the groove area is determined, and a second reference plane corresponding to the groove area is determined based on the fourth depth average.

7. The method according to claim 1, It is characterized in that The method further comprises: The confidence of the deformed region is determined based on a ratio between the number of noise points included in the deformed region and the total number of pixel points included in the deformed region.

8. The method according to any one of claims 1 to 7, It is characterized in that The method further comprises: Acquiring a two-dimensional image of the top surface of the box-type structure; Based on the corresponding relationship between the depth image and the two-dimensional image, the regions corresponding to the multiple components included in the top surface and the deformation region in each of the regions are marked in the two-dimensional image.

9. A box-type deformation detection device, It is characterized in that The device comprises: An acquisition module, used for acquiring a depth image of the top surface of the box-type structure; an identification module, configured to identify, from the depth image, regions corresponding to a plurality of components included in the top surface based on depth information of the depth image and gradient differences between different pixel points included in the depth image, wherein the plurality of components at least include: a top plate composed of a corrugated plate, a corner piece having a strength greater than a strength threshold, and a reinforcing plate; wherein the corrugated plate includes a convex groove region and a concave groove region, the corner piece includes a cavity, the reinforcing plate surrounds the corner piece, and a weld exists at a junction between the reinforcing plate and the top cross beam, the extension plate, and the top plate of the box-type structure; The recognition module recognizes regions corresponding to the multiple components included in the top surface from the depth image based on the depth information of the depth image and the gradient difference between different pixels included in the depth image, including: For the reinforcing plate, the recognition module determines, based on the area corresponding to the corner piece included in the top surface, a first boundary where the reinforcing plate coincides with the corner piece, takes at least one pixel point included in the first boundary as a starting point, traverses and determines the gradients of a plurality of subsequent pixels, determines, based on the pixel points corresponding to the positions where the gradient difference is greater than a gradient difference threshold, a second boundary of the reinforcing plate, and determines the area consisting of the first boundary and the second boundary in the depth image as the area corresponding to the reinforcing plate; a determination module, configured to determine the distance between each pixel point included in the area corresponding to the reinforcing plate and the reference plane corresponding to the reinforcing plate, and determine the deformation area in the area corresponding to the reinforcing plate based on the pixel points corresponding to the distance greater than the distance threshold in the area corresponding to the reinforcing plate; The determination module is further used to determine, for the top plate, the convex groove area and the concave groove area included in the top plate, respectively determine a first reference plane corresponding to the convex groove area and a second reference plane corresponding to the concave groove area, and determine a deformation area in the convex groove area based on a distance between a pixel point included in the convex groove area and the first reference plane, and determine a deformation area in the concave groove area based on a distance between a pixel point included in the concave groove area and the second reference plane; The recognition module recognizes regions corresponding to the multiple components included in the top surface from the depth image based on the depth information of the depth image and the gradient difference between different pixels included in the depth image, including: For the corner piece, the recognition module determines the pixel points in the depth image whose depth value is less than the depth threshold and whose gradient difference with the adjacent pixel points is greater than the gradient difference threshold, and determines the area in the depth image composed of the pixel points whose depth value is less than the depth threshold and whose gradient difference with the adjacent pixel points is greater than the gradient difference threshold, and whose corresponding spatial feature is a corner, as the area corresponding to the corner piece included in the top surface; The determination module is further used to traverse from the edge of the corner piece to the middle of the corner piece, determine the position of the hole based on the gradient difference of the pixel points within a preset interval of the corner piece, determine the depth mean of the pixel points in the area corresponding to the corner piece excluding the hole, delete the pixel points in the area corresponding to the corner piece whose distance from the depth mean is greater than a depth mean threshold, determine the depth mean of the remaining pixel points in the corner piece, and determine the reference plane corresponding to the corner piece based on the depth mean of the remaining pixel points in the corner piece, and determine the deformation area in the area corresponding to the corner piece based on the distance between the pixel points in the area corresponding to the corner piece excluding the hole and the reference plane corresponding to the corner piece.

10. An electronic device, It is characterized in that The electronic device comprises: A memory for storing executable instructions; The processor is used to implement the deformation detection method of the box-type structure described in any one of claims 1 to 8 when executing the executable instructions stored in the memory.

11. A computer-readable storage medium, It is characterized in that Executable instructions are stored, which are used to implement the deformation detection method of the box-type structure described in any one of claims 1 to 8 when executed by a processor.

12. A computer program product comprising computer executable instructions, It is characterized in that When the computer executable instructions are executed by a processor, the deformation detection method of the box-type structure described in any one of claims 1 to 8 is implemented.

Images