A deformation detection method, device, equipment and computer readable storage medium

Abstract

The application provides a deformation detection method and device, equipment and a computer readable storage medium; relates to the image processing technology in the field of artificial intelligence; the method comprises the following steps: collecting an input image containing the surface of a to-be-detected box; the to-be-detected box is a box waiting for deformation detection; the surface feature extraction and deformation feature extraction of the input image are carried out in parallel, and the surface feature and deformation feature corresponding to the input image are obtained; the surface feature represents the appearance information of the object in the input image, and the deformation feature represents the bending and wrinkle degree of the object texture in the input image; based on the surface feature and the deformation feature, a deformation region is determined from the input image; the deformation region is a region of the surface of the to-be-detected box that has deformed; the deformation region is labeled to obtain a deformation detection result of the input image. Through the application, the efficiency of deformation detection can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the image processing technology in the field of artificial intelligence, and particularly relates to a deformation detection method and device, equipment and a computer readable storage medium. BACKGROUND

[0002] The container is often placed in the ocean freighter for cargo transportation due to its uniform specification. However, once the deformation of the container plate occurs, the transportation efficiency of the container is greatly reduced. Therefore, timely detection of the deformation of the container plate is important for improving the transportation efficiency of the container.

[0003] In the related art, the commonly used method for detecting the deformation of the container is still to perform on-site inspection and measurement by manual work, so that the deformation detection efficiency of the container is low. SUMMARY

[0004] The embodiment of the present application provides a deformation detection method, device, equipment and computer readable storage medium, which can improve the efficiency of deformation detection.

[0005] The technical scheme of the embodiment of the present application is as follows:

[0006] The embodiment of the present application provides a deformation detection method, comprising:

[0007] An input image containing a surface of a to-be-detected container is collected; the to-be-detected container is a container waiting for deformation detection;

[0008] Surface feature extraction and deformation feature extraction are performed on the input image in parallel to obtain surface features and deformation features corresponding to the input image; the surface features represent appearance information of an object in the input image, and the deformation features represent bending and wrinkle degrees of a texture of the object in the input image;

[0009] Based on the surface features and the deformation features, a deformation region is determined from the input image; the deformation region is a region of the surface of the to-be-detected container that has been deformed;

[0010] The deformation region is labeled to obtain a deformation detection result of the input image.

[0011] The embodiment of the present application provides a deformation detection device, comprising:

[0012] An image collection module is configured to collect an input image containing a surface of a to-be-detected container; the to-be-detected container is a container waiting for deformation detection;

[0013] The feature extraction module is configured to perform extraction of surface features and extraction of deformation features on the input image in parallel to obtain surface features and deformation features corresponding to the input image; the surface features represent appearance information of an object in the input image, and the deformation features represent bending and wrinkle degrees of a texture of the object in the input image.

[0014] The deformation determination module is configured to determine a deformation region from the input image based on the surface features and the deformation features; the deformation region is a region on a surface of the to-be-detected box in which deformation occurs.

[0015] The result labeling module is configured to label the deformation region to obtain a deformation detection result of the input image.

[0016] In some embodiments of the present application, the deformation determination module is further configured to control attention of the deformation features according to the surface features to obtain to-be-detected deformation features; the to-be-detected deformation features represent deformation features on which attention is focused on bending and wrinkle degrees of the surface of the to-be-detected box; and the deformation region is detected from the input image according to the to-be-detected deformation features.

[0017] In some embodiments of the present application, the deformation determination module is further configured to calculate an image weight occupied by the surface of the to-be-detected box in the input image by using the surface features; extract global features of the surface features to obtain a surface global feature vector; calculate attention by using the image weight on the surface global feature vector to obtain an attention weight vector; and perform dot multiplication of the attention weight vector and the deformation features to obtain the to-be-detected deformation features.

[0018] In some embodiments of the present application, the deformation determination module is further configured to perform a plurality of types of pooling operations on the surface features respectively to obtain global feature maps corresponding to the plurality of types of pooling operations respectively; the global feature maps include features of the surface of the to-be-detected box and features of a background region; and the global feature maps corresponding to the plurality of types of pooling operations respectively are fused by a fully connected layer to obtain the surface global feature vector.

[0019] In some embodiments of the present application, the feature extraction module is further configured to classify a surface of the to-be-detected box included in the input image by using an image classifier to obtain a surface type; and perform extraction of surface features and extraction of deformation features on the input image in parallel by using a target feature extraction model to obtain the surface features and the deformation features when the surface type belongs to a preset type.

[0020] In some embodiments of the present application, the feature extraction module is further configured to perform image feature extraction on the input image by using a backbone network in the target feature extraction model to obtain a feature map of the input image; extract the surface feature from the feature map by using a surface feature extraction branch in the target feature extraction model; and extract the deformation feature from the feature map by using a deformation feature extraction branch in the target feature extraction model.

[0021] In some embodiments of the present application, the feature extraction module is further configured to segment a box surface region of the box to be detected from the input image by using the surface feature; and label the box surface region from the input image to obtain a box surface detection result of the input image.

[0022] In some embodiments of the present application, the deformation detection device further comprises a deformation classification module.

[0023] The deformation classification module is configured to classify the deformation detection result of the input image to obtain a deformation degree of the deformation detection result.

[0024] In some embodiments of the present application, the deformation detection device further comprises a deformation denoising module.

[0025] The deformation denoising module is configured to denoise the deformation region to obtain a denoised deformation region.

[0026] The result labeling module is further configured to label the denoised deformation region to obtain the deformation detection result of the input image.

[0027] In some embodiments of the present application, the deformation region comprises a plurality of deformation sub-regions; and the deformation denoising module is further configured to calculate a region area of each deformation sub-region in the plurality of deformation sub-regions; select a deformation sub-region with a region area greater than or equal to an area threshold from the plurality of deformation sub-regions to obtain at least one candidate deformation sub-region; and take the at least one candidate deformation sub-region as the denoised deformation region.

[0028] In some embodiments of the present application, the feature extraction module is further configured to select a matching feature extraction model corresponding to the input image from a plurality of preset feature extraction models by using the surface type; and perform parallel extraction of the surface feature and the deformation feature on the input image by using the matching feature extraction model to obtain the surface feature and the deformation feature.

[0029] An embodiment of the present application provides a deformation detection device, comprising:

[0030] a memory configured to store executable deformation detection instructions;

[0031] a processor configured to implement a deformation detection method provided by an embodiment of the present application when executing the executable deformation detection instructions stored in the memory.

[0032] An embodiment of the present application provides a computer readable storage medium storing executable deformation detection instructions, which are used to cause a processor to implement a deformation detection method provided by an embodiment of the present application when executing the deformation detection instructions.

[0033] An embodiment of the present application has the following beneficial effects: The deformation detection device first collects an input image containing a surface of a box to be deformed, then extracts surface features representing the texture of the surface of the object and deformation features representing the bending and wrinkle degree of the object from the input image, and then determines the deformed region from the input image in combination with the deformation features and the surface features, and labels the deformed region to obtain a deformation detection result. In this way, the deformation detection device can automatically determine the deformation region of the box to be detected from the collected input image by means of image processing technology, reduce the time required for deformation detection, and greatly improve the efficiency of deformation detection. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1 is a schematic diagram of a container panel;

[0035] Figure 2 is a schematic diagram of deformation of a container panel;

[0036] Figure 3 is a schematic diagram of a deformation detection system 100 provided by an embodiment of the present application;

[0037] Figure 4 is another schematic diagram of a deformation detection system 100 provided by an embodiment of the present application;

[0038] Figure 5 is a schematic diagram of a deformation detection device provided by an embodiment of the present application;

[0039] Figure 6 is a schematic diagram of a deformation detection method provided by an embodiment of the present application Figure 1 ;

[0040] Figure 7 is a schematic diagram of an input image provided by an embodiment of the present application;

[0041] Figure 8A is a schematic diagram of a deformation detection result provided by an embodiment of the present application Figure 1 ;

[0042] Figure 8Bis a schematic diagram of a deformation detection result provided by an embodiment of the present application Figure 2 ;

[0043] Figure 9 is a schematic diagram of a deformation detection method provided by an embodiment of the present application Figure 2 ;

[0044] Figure 10 is a schematic diagram of a deformation region provided by an embodiment of the present application

[0045] Figure 11 is a schematic diagram of generating a deformation feature to be detected provided by an embodiment of the present application

[0046] Figure 12 is a schematic diagram of a process of determining a surface type provided by an embodiment of the present application

[0047] Figure 13 is a schematic diagram of a process of extracting a surface feature and a deformation feature provided by an embodiment of the present application

[0048] Figure 14 is a schematic diagram of a surface region of a container provided by an embodiment of the present application

[0049] Figure 15 is a schematic diagram of a container surface detection result provided by an embodiment of the present application

[0050] Figure 16 is a schematic diagram of a deformation degree provided by an embodiment of the present application

[0051] Figure 17 is a schematic diagram of a process of performing deformation detection on a container plate surface provided by an embodiment of the present application

[0052] Figure 18 is a schematic diagram of a multi-branch network structure provided by an embodiment of the present application DETAILED DESCRIPTION

[0053] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be described in further detail below with reference to the drawings, and the described embodiments should not be regarded as limiting the present application, and all other embodiments obtained by a person of ordinary skill in the art without making creative efforts fall within the scope of protection of the present application.

[0054] In the following description, “some embodiments” are described, which describe a subset of all possible embodiments, but it can be understood that “some embodiments” can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0055] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the description herein is for describing particular embodiments only and is not intended to be limiting of the application.

[0056] The relevant data collection processing in the embodiments of the application should be strictly in accordance with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and within the scope of authorization of laws and regulations and the personal information subject, carry out subsequent data use and processing.

[0057] Before the embodiments of the application are further described in detail, the terms and phrases involved in the embodiments of the application are explained, and the terms and phrases involved in the embodiments of the application are applicable to the following explanations.

[0058] 1) Cloud Technology refers to a kind of hosting technology that combines hardware, software, network and other series of resources in a wide area network or local area network to realize data calculation, storage, processing and sharing.

[0059] Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, application technology and other technologies based on cloud computing business model application, which can form a resource pool, be used on demand, and be flexible and convenient. Cloud computing technology will become an important support. The background service of the technical network system needs a large amount of computing and storage resources, which is a video website, a picture website and more portal websites. With the high development and application of the Internet industry, every item may have its own identification mark in the future, which needs to be transmitted to the background system for logical processing. Different levels of data will be processed separately, and various industry data will need strong system support, which can only be realized through cloud computing.

[0060] 2) Artificial Intelligence (AI) is to use digital computers or digital computer controlled machines to simulate, extend and expand human intelligence, perceive environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology of computer science, which tries to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is to design and implement principles and methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making.

[0061] Artificial intelligence technology is a comprehensive discipline, involving a wide range of fields, both hardware and software level technology. Artificial intelligence foundation generally includes, such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology and machine learning / deep learning and other several major directions.

[0062] 3) Computer vision (CV) is a science that studies how to make computers "see". Further, it refers to using cameras and computers to replace human eyes to identify, track and measure targets, and further process graphics, so that the computer processing becomes images more suitable for human eye observation or transmission to instrument detection. As a scientific discipline, computer vision researches related theories and technologies, trying to establish artificial intelligence systems that can obtain information from images or multidimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping, and other technologies. It also includes common face recognition, fingerprint recognition and other biometric identification technologies.

[0063] 4) Machine learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It is a discipline that studies how computers simulate or implement human learning behavior to acquire new knowledge or skills, and reorganize existing knowledge structure to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental approach to making computers intelligent. Its applications are widespread in various fields of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning and other technologies.

[0064] 5) Container is a standardized steel box used for transporting various goods. Its format is standardized, which facilitates stacking and can be placed in large quantities on specially designed ocean-going ships for cargo transportation.

[0065] 6) Container panel is a panel that can be assembled into a container, including six panels, namely the bottom panel, the door, the front panel, the left panel, the right panel and the top panel. Among them, the front panel, the left panel, the right panel and the top panel are collectively referred to as the panel of the container, and the panels are connected and reinforced by beams or corner columns. Exemplarily, Figure 1is a schematic diagram of a container panel, and the six faces of the container panel 1-1 are a door 1-11, a floor 1-12, a left face 1-13, a right face 1-14, a front face 1-15 and a top face 1-16. Among them, the left face 1-13 and the right face 1-14 are relatively similar in shape.

[0066] 7) Container panel deformation refers to abnormal bending deformation existing on the container panel. Once the container panel has a major deformation, such as the panel being convex outward, the stacking of goods is affected, or the panel being concave inward, the volume for storing goods is reduced, etc., which will affect the transportation efficiency and cost of the container.

[0067] Exemplarily, Figure 2 is a schematic diagram of the deformation of the container panel. In the deformed container panel 2-1, the panel 2-11 is deformed due to rust, the panels 2-12, 2-13 and 2-14 are deformed due to rust and impact at the same time, and the panels 2-15 and 2-16 are deformed due to impact. It can be seen that there are various reasons for the deformation of the container panel.

[0068] 8) High-resolution network is a convolutional neural network structure widely used in deep learning methods, which has achieved good results in computer vision fields such as image classification, image detection and image segmentation. Unlike the network design method of concatenating high-resolution feature maps and low-resolution feature maps, the high-resolution network retains the high-resolution detail feature map when extracting high-level semantic information, so that the details in the picture can be better preserved at the output end of the network, thereby greatly improving the extraction effect of the network on multi-scale information.

[0069] 9) Attention mechanism is a mechanism proposed in natural language processing and widely used in computer vision field. It refers to the integration of information, calculation of channels or spatial pixels, and interaction between them, so as to adjust the attention direction and weight screening to obtain better feature extraction effect.

[0070] Due to its uniform specifications, containers are often placed in ocean-going cargo ships for cargo transportation. However, once the container panel is deformed, the transportation efficiency of the container will be greatly reduced, therefore, it is necessary to detect the deformation of the container panel, and repair it in time when the deformation of the container panel is found to be serious, which is important for improving the transportation efficiency of the container.

[0071] In the related art, there are some methods for detecting containers by using image processing technology. For example, after normalizing a picture and obtaining a feature map layer, the feature map layer is input into a region proposal network to obtain a proposal region, the feature map corresponding to the proposal region is input into a region of interest alignment network, and is unified into a fixed size, and the region of interest unified into the fixed size is input into three networks, i.e., a classification network, a detection network and a segmentation network, to obtain the position, shooting angle and edge information of each container.

[0072] It can be seen that in the related art, only the angle of the container can be analyzed by using image processing technology, and there is no method for detecting the deformation of the container by using image processing technology. Moreover, the common method for detecting the deformation of the container is still to perform on-site inspection and measurement by manual work, which is low in efficiency and high in labor cost. Meanwhile, the deformation on the surface of the container is various, such as large and serious deformation, gentle deformation, or deformation in a large amount of wear and rust, which is difficult to be observed by naked eyes, thereby reducing the accuracy of the deformation detection of the container.

[0073] With the research and progress of artificial intelligence technology, artificial intelligence technology is applied in many fields, such as smart home, smart wearable device, virtual assistant, smart speaker, smart marketing, unmanned vehicle, autonomous vehicle, unmanned aerial vehicle, robot, smart medical treatment, smart customer service, etc. It is believed that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0074] The deformation detection of the surface of the container is also an application direction of artificial intelligence. The embodiments of the present application provide a deformation detection method, device and equipment and a computer readable storage medium, which involve image processing technology in artificial intelligence and can improve the efficiency of deformation detection.

[0075] The following describes an exemplary application of the deformation detection device provided by the embodiments of the present application. The device provided by the embodiments of the present application can be implemented as a notebook computer, a tablet computer, a desktop computer, a set-top box, a mobile device (for example, a mobile phone, a portable music player, a personal digital assistant, a dedicated message device, a portable game device) and various types of user terminals, or can be implemented as a server. The following describes an exemplary application of the deformation detection device.

[0076] In some embodiments, the deformation detection device can be implemented as a device cluster, which can be composed of a server and a mobile terminal. At this time, some steps in the deformation detection method provided by the present application can be implemented on the mobile terminal, and other steps can be implemented by the server.

[0077] For example, referring to Figure 3 , Figure 3is an optional architecture diagram of the deformation detection system 100 provided by the embodiment of the present application, for realizing supporting a deformation detection application, the deformation detection device is implemented as a device cluster, in the device cluster, the terminal 400 (exemplarily shows the terminal 400-1 and the terminal 400-2) is connected to the server 200 through the network 300. That is, the deformation detection method provided by the embodiment of the present application is realized by means of cloud technology. Wherein, the network 300 can be a wide area network or a local area network, or a combination of the two.

[0078] The terminal 400-1 and the terminal 400-2 are used to collect input images of the box to be detected. When performing deformation detection, a certain worker can trigger a photographing button on the graphical interface 400-11 of the terminal 400-1 to collect input images containing the surfaces of some box to be detected, and send the collected input images to the server 200 through the network 300. At the same time, another worker can trigger a photographing button on the graphical interface 400-21 of the terminal 400-2 to collect input images containing the surfaces of some other box to be detected, and send the collected input images to the server 200 through the network 300.

[0079] The server 200 performs parallel extraction of surface features and deformation features on the input images to obtain surface features and deformation features corresponding to the input images. Wherein, the surface features represent appearance information of the object in the input image, and the deformation features represent the bending and wrinkle degree of the texture of the object in the input image. The server 200 determines the deformation region from the input image based on the surface features and the deformation features, and finally labels the deformation region to obtain the deformation detection result of the input image. In this way, through the device cluster composed of the terminal 400 and the server 200, the deformation detection of the surface of the box to be detected is realized.

[0080] In some embodiments, the server 200 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDNs, and basic cloud computing services such as big data and artificial intelligence platforms.

[0081] In another embodiment, the deformation detection device can be implemented as a mobile terminal, at this time, all steps of the deformation detection method provided by the present application are implemented in the mobile terminal.

[0082] Exemplarily, see Figure 4 , Figure 4 is another optional architecture diagram of the deformation detection system 100 provided by the embodiment of the present application. For realizing supporting a deformation detection application, the deformation detection device is implemented as a terminal 400.

[0083] When a worker triggers the camera button on the graphical interface 410 of terminal 400, an input image containing the surface of the box to be inspected is acquired. The box to be inspected is the one awaiting deformation detection. Terminal 400 extracts surface features and deformation features from the input image, obtaining the corresponding surface features and deformation features. Next, based on the surface features and deformation features, terminal 400 determines the deformation region from the surface image, where the deformation region is the area on the surface of the box to be inspected that has undergone deformation. Finally, terminal 400 labels the deformation region to obtain the deformation detection result of the input image. Thus, deformation detection of the surface of the box to be inspected is achieved through terminal 400.

[0084] The terminal 400 can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. The terminal and the server can be connected directly or indirectly through wired or wireless communication, which is not limited in this embodiment of the invention.

[0085] See Figure 5 , Figure 5 This is a schematic diagram of the deformation detection device provided in the embodiments of this application. Figure 5 The deformation detection device 500 shown includes at least one processor 510, a memory 550, at least one network interface 520, and a user interface 530. The various components in the deformation detection device 500 are coupled together via a bus system 540. It is understood that the bus system 540 is used to implement communication between these components. In addition to a data bus, the bus system 540 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 5 The general labeled all buses as Bus System 540.

[0086] The processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0087] User interface 530 includes one or more output devices 531 that enable the presentation of media content, including one or more speakers and / or one or more visual displays. User interface 530 also includes one or more input devices 532, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0088] The memory 550 can be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, and the like. The memory 550 optionally includes one or more storage devices remotely located from the processor(s) 510.

[0089] The memory 550 includes volatile memory or nonvolatile memory, or both. Nonvolatile memory can be read only memory (ROM), and volatile memory can be random access memory (RAM). The memory 550 is intended to include any suitable type of memory.

[0090] In some embodiments, the memory 550 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or superset thereof, which are illustratively described below.

[0091] The operating system 551 includes systems programs for handling various basic system services and for performing hardware dependent tasks, such as a firmware layer, a kernel layer, a driver layer, and the like, for implementing various basic services and handling hardware dependent tasks;

[0092] The network communication module 552 is configured to communicate to other computing devices via one or more network interfaces 520, examples of which include Bluetooth, wireless compatibility certification (Wi-Fi), and universal serial bus (USB), and the like;

[0093] The presentation module 553 is configured to enable presentation of information via one or more output devices 531 associated with the user interface 530 (e.g., a display screen, a speaker, and the like) (e.g., a user interface for operating a peripheral device and displaying content and information);

[0094] The input processing module 554 is configured to detect and interpret one or more user inputs or interactions from one or more input devices 532.

[0095] In some embodiments, the apparatus provided by the embodiments of the present application can be implemented in software, Figure 5A morphing detection apparatus 555 stored in the memory 550 is shown, which can be software in the form of programs and plug-ins, etc., including the following software modules: an image acquisition module 5551, a feature extraction module 5552, a morphing determination module 5553, a result labeling module 5554, a morphing classification module 5555, and a morphing denoising module 5556, which are logical, and thus can be combined or further split according to the implemented functions. The functions of the various modules will be described below.

[0096] In some embodiments, the morphing detection apparatus provided by the embodiments of the present application can be implemented in a hardware manner. As an example, the morphing detection apparatus provided by the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the morphing detection method provided by the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can use one or more application specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs), or other electronic elements. Exemplary,

[0097] The embodiments of the present application provide a morphing detection device, comprising:

[0098] a memory configured to store executable morphing detection instructions;

[0099] a processor configured to execute the executable morphing detection instructions stored in the memory, so as to implement the morphing detection method provided by the embodiments of the present application.

[0100] In the following, the morphing detection method provided by the embodiments of the present application will be described in conjunction with exemplary applications and implementations of the morphing detection device provided by the embodiments of the present application.

[0101] Referring to Figure 6 , Figure 6 is an optional flowchart of the morphing detection method provided by the embodiments of the present application Figure 1 will be described in conjunction with the steps shown. Figure 6

[0102] S101, acquire an input image containing a surface of a box to be detected.

[0103] ​This application embodiment is implemented in a scenario of deformation detection of the surface of a container, such as deformation detection of the outer surface of a shipping container or deformation detection of the inner surface of a wooden crate. In order to perform deformation detection, the deformation detection device first acquires an image of any surface of the container to be inspected, and uses the acquired image as the input image so that it can subsequently determine whether the surface has deformation based on the input image.

[0104] It should be noted that the container to be tested is the container awaiting deformation testing. This container can be a shipping container, wooden box, iron box, or even a suitcase, etc., and this application does not limit it.

[0105] The enclosure to be tested typically has six outer surfaces. In this embodiment, the deformation detection device can acquire images of any one of these six surfaces to obtain an input image. Alternatively, the deformation detection device can also acquire images of any one of the six inner surfaces of the enclosure to obtain an input image.

[0106] In some embodiments, the deformation detection device can acquire a color input image using a color camera. In other embodiments, the deformation device can acquire a grayscale input image using an infrared camera; this application is not limited thereto.

[0107] It should be noted that the input image may contain background content unrelated to the box being detected, in addition to one surface of the box to be detected. For example, Figure 7 This is a schematic diagram of the input image provided in an embodiment of this application. For example... Figure 7 As shown, the container to be detected is a shipping container. The input image 7-1 contains not only one side of the container 7-11, but also some background elements 7-12, such as the ground, sky, pillars, etc.

[0108] It should be noted that in some embodiments of this application, the deformation detection device can be implemented as a terminal. In this case, the terminal's camera captures the input image, enabling the terminal to perform feature extraction and complete the entire deformation detection process. In other embodiments, the deformation detection device can be implemented as a device cluster consisting of a terminal and a server. In this case, the terminal in the device cluster captures the input image and uploads it to the server, whereby the server performs feature extraction and completes the deformation detection process.

[0109] S102. Extract surface features and deformation features from the input image in parallel to obtain the surface features and deformation features corresponding to the input image.

[0110] After obtaining the input image, the deformation detection device uses the multi-branch network model to simultaneously extract the surface feature and the deformation feature from the input image, so as to obtain the surface feature and the deformation feature from the input image, and facilitate subsequent deformation region detection based on the surface feature and the deformation feature.

[0111] It should be noted that the deformation feature represents the bending and wrinkling degree of the object in the input image, and can represent the deformation of the surface of the to-be-detected box in detail. The surface feature represents the appearance information of the object in the input image, and the appearance information includes size, color, texture, pattern and other information. Different objects have different appearances, so the surface feature can be used to distinguish which part of the input image is the surface of the to-be-detected box.

[0112] It can be understood that the structure of the multi-branch network model includes multiple branches, and each branch focuses on extracting different types of features. In the embodiments of the present application, the multi-branch network model can include two branches, one branch extracts the surface feature, and the other branch extracts the deformation feature. The two branches run in parallel and share a main network, which can improve the efficiency of feature extraction and reduce the memory consumption required by the model.

[0113] Further, each branch of the multi-branch network model is designed based on a high-resolution network structure, so that each branch can extract high-level semantic information while retaining high-resolution detail feature maps, i.e., retaining low-level feature maps, so that the output end of each branch can use the detailed information in the input image.

[0114] S103, determining the deformation region from the input image based on the surface feature and the deformation feature.

[0115] The deformation detection device can distinguish the surface feature of the surface of the to-be-detected box from the background, and combine the surface feature with the deformation feature, so that the attention of the bending and wrinkling degree is focused on the deformation feature of the surface of the to-be-detected box. Then, the deformation detection device uses the attention-enhanced deformation feature to segment the deformation region from the input image. In this way, the deformation region can be quickly obtained by image processing technology, and manual measurement and inspection of the to-be-detected box are no longer needed.

[0116] It can be understood that the deformation region is the region of the surface of the to-be-detected box that is deformed. The deformation detection device can classify each pixel of the input image according to the surface feature and the deformation feature, such as the category of the deformation inside the box and other categories (such as the deformation outside the box, the non-deformation inside the box, the non-deformation outside the box, etc.), and then regard the image part composed of the pixels belonging to the category of the deformation inside the box as the deformation region.

[0117] S104, label the deformation region to obtain a deformation detection result of the input image.

[0118] After the deformation detection device determines the deformation region, it will use lines, label boxes, stars, or arrows to mark the deformation region from the input image to obtain the deformation detection result of the input image, so that the deformation region is more direct.

[0119] For example, Figure 8A The deformation detection result provided by the embodiment of the present application Figure 1 , in Figure 8A , the deformation detection device labels the deformation detection result 8-1, the deformation detection result 8-2, the deformation detection result 8-3, and the deformation detection result 8-4 using a polyline; Figure 8B The deformation detection result provided by the embodiment of the present application Figure 2 , in Figure 8A , the deformation detection result 8-5, the deformation detection result 8-6, the deformation detection result 8-7, and the deformation detection result 8-8 are labeled using a curve that fits the deformation region better. Whether it is Figure 8A Or Figure 8B the labeling method, the staff can directly know which regions of the surface of the box to be detected have been deformed.

[0120] In the embodiment of the present application, the deformation detection device will first collect the input image containing the surface of the box waiting for deformation detection, then extract the surface features representing the texture of the object surface and the deformation features representing the bending and wrinkle degree of the object from the input image, and then determine the deformation region from the input image in combination with the deformation features and the surface features, and label the deformation region to obtain the deformation detection result. In this way, the deformation detection device can automatically determine the deformation region of the box to be detected from the collected input image by means of image processing technology, reduce the time required for deformation detection, and greatly improve the efficiency of deformation detection.

[0121] Based on Figure 6 , see Figure 9 , Figure 9 is an optional flowchart of the deformation detection method provided by the embodiment of the present application Figure 2 In some embodiments of the present application, based on the surface features and the deformation features, the deformation region is determined from the input image, that is, the specific implementation process of S103 can include: S1031-S1032, as follows:

[0122] S1031, according to the surface features, control the attention of the deformation features to obtain the deformation features to be detected.

[0123] In the background part of the input image, there can also be bending and wrinkling, so that the deformation features contain feature descriptions of the bending and wrinkling of the background part, which are interference terms in deformation detection. When the input image contains more background, there can be more interference terms, so that the content of the background part is mis-detected as a deformation region. However, the surface features describe the appearance of each object in the input image, and can be used to distinguish the box region of the to-be-detected box from the background. In the embodiment of the present application, the deformation detection device uses the surface features to determine which part of the deformation features belongs to the surface of the to-be-detected box and which part belongs to the background part, focuses the attention of deformation detection on the core deformation features, and excludes the interference of the complex background in the input image to reduce the mis-detection of the deformation region and improve the accuracy of deformation region detection.

[0124] It should be noted that the to-be-detected deformation feature represents the deformation feature that focuses attention on the bending and wrinkling degree of the surface of the to-be-detected box. The deformation detection device can control the attention by fusing the surface features and the deformation features and controlling the weight vector during fusion, so that the attention of deformation detection is focused on the feature part corresponding to the surface of the to-be-detected box, and the attention to the interference terms in the background, i.e., the interference features such as texture, corrugation, bending and wrinkling in the background, is reduced.

[0125] Further, the deformation detection device can classify and identify the surface features to determine the proportion of the surface of the to-be-detected box contained in the input image, and generate a weight vector during fusion using the proportion and the global feature map corresponding to the surface features. It can be understood that the global feature map corresponding to the surface features can reflect the position of the surface of the to-be-detected box in the input image, so that the weight vector generated according to the global feature map can also reflect the position of the surface of the to-be-detected box in the input image. Using the weight vector, the attention can be focused on the feature part corresponding to the surface of the to-be-detected box, and the interference terms corresponding to the background part can be suppressed to obtain the to-be-detected deformation feature.

[0126] S1032, detecting a deformation region from the input image according to the to-be-detected deformation feature.

[0127] The deformation detection device determines from the input image which image regions can obtain the to-be-detected deformation feature through feature extraction, so that these determined image regions are the deformation region corresponding to the input image. In this way, the deformation detection device can accurately determine the deformation region.

[0128] Exemplarily, Figure 10 is a schematic diagram of a deformation region provided by the embodiment of the present application. As Figure 10As shown, the deformation detection device determines the deformation region in the input image 10-1, and the output deformation region is the white region in the output image 10-2 Figure 10-2 The region without deformation is the black region in the output image 10-2 Figure 10-2 .

[0129] In the embodiments of the present application, the deformation detection device can use the surface feature to strengthen the attention of the core deformation feature in the deformation feature, which represents the bending and wrinkle degree of the surface of the to-be-detected box, so as to concentrate the attention of the deformation detection on the region corresponding to the surface of the to-be-detected box, avoid the interference of similar features in the background, and improve the accuracy of the deformation detection.

[0130] In some embodiments of the present application, the attention of the deformation feature is controlled according to the surface feature, and the specific implementation process of obtaining the to-be-detected deformation feature S1031 can include S1031a-S1031d as follows:

[0131] S1031a, using the surface feature, calculating the image weight of the surface of the to-be-detected box in the input image.

[0132] The deformation detection device first identifies the surface feature, segments the part of the surface of the to-be-detected box from the image, and then calculates the proportion of the part of the surface of the to-be-detected box in the input image. The image weight of the surface of the to-be-detected box in the input image is obtained by using the proportion.

[0133] In some embodiments, the deformation detection device can directly use the proportion as the image weight; in other embodiments, the deformation detection device can use half of the proportion or the product of the proportion and the probability that the input image contains the surface of the to-be-detected box as the image weight.

[0134] S1031b, extracting the global feature of the surface feature to obtain the surface global feature vector.

[0135] The deformation detection device can use the pooling operation to extract the global feature map corresponding to the surface feature, and then convert the global feature map into a vector to obtain the surface global feature vector.

[0136] It can be understood that the deformation detection device can use a fully connected layer to convert the global feature map into a surface global feature vector, or extract a diagonal line from the global feature map to obtain a surface global feature vector.

[0137] S1031c, using the image weight to calculate the attention of the surface global feature vector to obtain the attention weight vector.

[0138] Then, the deformation detection device can multiply the image weight and the surface global feature vector to calculate the attention of different channels, and can also add the image weight and the surface global feature vector to obtain the attention weight vector, which is not limited in the present application.

[0139] It can be understood that when the image weight is large, it indicates that the surface of the to-be-detected box accounts for a large proportion in the input image. The calculation of the channel attention of the surface global feature vector by using the image proportion can strengthen the features in the global feature vector.

[0140] S1031d, point-multiply the attention weight vector and the deformation feature to obtain the to-be-detected deformation feature.

[0141] The deformation detection device point-multiplies the attention weight vector and the deformation feature, and the result of the point multiplication is the to-be-detected deformation feature. In this way, the deformation detection device completes the determination process of the to-be-detected deformation feature.

[0142] In the embodiments of the present application, the deformation detection device can first identify the image weight of the surface of the to-be-detected box, extract the global feature of the surface feature to obtain the surface global feature vector, then calculate the channel attention by using the image weight and the surface global feature vector to obtain the attention weight vector, and finally take the point multiplication result of the attention weight vector and the deformation feature as the to-be-detected deformation feature. In this way, the deformation detection device can obtain the to-be-detected deformation feature, so as to detect the deformation region by using the to-be-detected deformation feature subsequently.

[0143] In some embodiments of the present application, the process of extracting the global feature of the surface feature to obtain the surface global feature vector, i.e., the specific implementation process of S1031b, can include: S201-S202, as follows:

[0144] S201, respectively performing a plurality of types of pooling operations on the surface feature to obtain a plurality of types of global feature maps respectively corresponding to the plurality of types of pooling operations.

[0145] The deformation detection device obtains a plurality of different pooling layers, each of which can perform a pooling operation on the surface feature to obtain a global feature map corresponding to the pooling operation, so that by using a plurality of different pooling layers, a plurality of types of pooling operations can be performed on the surface feature to obtain a plurality of types of global feature maps respectively corresponding to the plurality of types of pooling operations. The global feature map includes the features of the surface of the to-be-detected box and the features of the background region.

[0146] It can be understood that the plurality of types of pooling operations can include maximum pooling and average pooling, so that the deformation detection device can obtain a global feature map corresponding to the maximum pooling and a global feature map corresponding to the average pooling, respectively.

[0147] S202, fuse the global feature maps corresponding to the plurality of types of pooling operations respectively by a fully connected layer to obtain a surface global feature vector.

[0148] The deformation detection device can also directly input the global feature maps corresponding to the plurality of types of pooling operations into a fully connected layer, so as to fuse the features of the global feature maps by the fully connected layer. The feature vector output by the fully connected layer is the surface global feature vector.

[0149] For example, Figure 11 is a schematic diagram provided by an embodiment of the present application for generating a deformation feature to be detected. The deformation detection device first performs maximum pooling 11-2 and average pooling 11-3 on the surface feature 11-1 respectively, and inputs the feature maps obtained by the maximum pooling 11-2 and the average pooling 11-3 into a fully connected layer 11-4, and combines with the image weight to obtain an attention weight vector 11-5. Then, the deformation detection device point-multiplies the attention weight vector 11-5 and the deformation feature 11-6 to obtain a new deformation feature, i.e., the deformation feature to be detected 11-7.

[0150] In an embodiment of the present application, the deformation detection device can obtain global feature maps of different dimensions of the surface feature by using pooling operations of different dimensions, and then fuse the global feature maps of different dimensions together to obtain a surface global feature vector, so as to facilitate subsequent calculation of channel attention.

[0151] In some embodiments of the present application, the extraction of the surface feature and the extraction of the deformation feature are performed in parallel on the input image to obtain the surface feature and the deformation feature corresponding to the input image, i.e., the specific implementation process of S102 can include: S1021-S1022, as follows:

[0152] S1021, use an image classifier to classify the surface of the to-be-detected box contained in the input image to obtain a surface type.

[0153] In an embodiment of the present application, deformation detection can be performed on some surfaces of the to-be-detected box, for example, the front surface and the side surface of the to-be-detected box are detected, and the floor and the door of the to-be-detected box are not detected. In this case, the deformation detection device will obtain a trained image classifier, and then identify the surface type to which the surface of the to-be-detected box in the input image belongs by using the image classifier, so as to facilitate subsequent judgment of whether deformation detection needs to be performed on the surface of the to-be-detected box in the input image.

[0154] For example, Figure 12 is a process schematic diagram provided by an embodiment of the present application for determining a surface type. In Figure 12In the embodiment, the input picture is a photo 12-1 of a container board surface, the deformation detection device inputs the container photo 12-1 into the image classifier, i.e., the container surface classifier 12-2, and can determine whether the container photo 12-1 belongs to the left and right surfaces 12-3 of the container, the front surface 12-4 of the container, the top surface 12-5 of the container, the bottom surface 12-6 of the container, the door 12-7 of the container, or the interior 12-8 of the container.

[0155] In S1022, when the surface type belongs to the preset type, the deformation detection device extracts the surface feature and the deformation feature from the input image in parallel by using the target feature extraction model.

[0156] In the embodiment, the deformation detection device determines whether the surface type belongs to the preset type, and determines whether the surface of the to-be-detected container included in the input image needs to be subjected to deformation detection. When the deformation detection device determines that the surface type belongs to the preset type, the deformation detection device acquires the target feature extraction model, and extracts the deformation feature and the surface feature from the input image by using the target extraction model, to realize deformation detection on the surface of the to-be-detected container in the input image.

[0157] In some embodiments, the preset type includes at least one of the front surface of the container, the side surface of the container, and the top surface of the container, to realize deformation detection on the front surface, the side surface, and the top surface of the to-be-detected container. In other embodiments, the preset type is a surface with a texture belonging to a corrugated type, to realize deformation detection on the surface with the corrugated type. Of course, the preset type can also be set to other types according to actual needs, which are not limited herein.

[0158] It can be understood that when the surface type does not belong to the preset type, the deformation detection device can not perform deformation detection on the surface of the to-be-detected container in the input image, or can perform deformation detection by using other manners (for example, reflecting the input image to a worker to perform deformation detection, or instructing the worker to further collect depth data to perform detection, etc.), which are not limited herein.

[0159] In the embodiment, the deformation detection device first performs type determination on the surface of the to-be-detected container in the input image, and then extracts the feature to perform deformation detection according to whether the surface type belongs to the preset type. In this way, some surfaces of the to-be-detected container can be focused on for detection, and a model capable of uniformly extracting features can also be trained.

[0160] In some embodiments of the present application, the surface feature and the deformation feature are extracted from the input image in parallel by using the target feature extraction model, i.e., the specific implementation process of S1022 can include: S1022a-S1022b, as follows:

[0161] In S1022a, the input image is subjected to image feature extraction by using the backbone network in the target feature extraction model, to obtain a feature map of the input image.

[0162] The target feature extraction model is composed of a backbone network, a surface feature extraction branch, and a deformation feature extraction branch. The surface feature extraction branch and the deformation feature extraction branch are both connected to the backbone network. The deformation detection device first inputs the input image into the backbone network in the target feature extraction model, to extract the feature map of the input image by using the backbone network.

[0163] In S1022b, the surface feature is extracted from the feature map by using the surface feature extraction branch in the target feature extraction model, and the deformation feature is extracted from the feature map by using the deformation feature extraction branch in the target feature extraction model.

[0164] After the feature map of the input image is extracted, the surface feature and the deformation feature are extracted from the feature map by using the surface feature extraction branch and the deformation feature extraction branch, respectively. Since this method shares one backbone network, compared with the traditional method of extracting the surface feature and the deformation feature, at least one feature extraction process is reduced, thereby reducing the running time of the model.

[0165] For example, the present application provides a process diagram for extracting the surface feature and the deformation feature, as shown in FIG. 13. Figure 13 The input image is a photo 13-1 of a container panel, which is first input into an overall feature extraction model 13-2, i.e., a backbone network, and then starts from the output end of the backbone network, and is divided into a surface feature extraction branch 13-3 and a deformation feature extraction branch 13-4. The surface feature of the input image is extracted by the surface feature extraction branch 13-3, and the deformation feature of the input image is extracted by the deformation feature extraction branch 13-4.

[0166] In the present application, the feature map of the input image is first extracted by using the backbone network in the target feature extraction model, and then the surface feature and the deformation feature are extracted from the feature map by using the surface feature extraction branch and the deformation feature extraction branch, respectively. Therefore, the running time of the model can be reduced by sharing the backbone network, thereby reducing the feature extraction time and further improving the efficiency of deformation detection.

[0167] In some embodiments of the present application, after the surface type of the surface of the to-be-detected box contained in the input image is classified by using the image classifier, that is, after S1021, the method can further include: S1023-S1024, as follows:

[0168] S1023, the surface type is used to select a matching feature extraction model corresponding to the input image from a plurality of preset feature extraction models.

[0169] S1024, the surface features and the deformation features are extracted from the input image in parallel by using the matching feature extraction model.

[0170] Since the input image can contain the surface of the to-be-detected box, however, the differences between different surfaces of the to-be-detected box are large (for example, the difference between the door and the side surface is large), if the feature extraction model adapted to different surfaces is used for feature extraction, the surface features and the deformation features extracted will be more accurate. Therefore, in the embodiments of the present application, after the surface type is determined by the deformation detection device, the feature extraction model corresponding to the surface type is found from a plurality of preset feature extraction models according to the surface type. Since there is a corresponding relationship between the surface type and the input image, the feature extraction model is a matching feature extraction model corresponding to the input image. Then, the deformation detection device uses the matching feature extraction model to extract the deformation features and the surface features from the input image in parallel, so as to extract more accurate deformation features and surface features by using the feature extraction model adapted to the input image.

[0171] In the embodiments of the present application, the surface of the to-be-detected box contained in the input image is classified by the deformation detection device to obtain the surface type, so that the most suitable matching feature extraction model is selected for the input image according to the surface type, so as to extract more accurate deformation features and surface features by using the target feature extraction model, so as to further improve the accuracy of deformation detection in the subsequent process.

[0172] In some embodiments of the present application, after the surface features are extracted from the feature map by the surface feature extraction branch in the target feature extraction model, that is, after S1022b, the method can further include: S1022c-S1022d, as follows:

[0173] S1022c, the surface features are used to segment the box surface region of the to-be-detected box from the input image.

[0174] The deformation detection device classifies each pixel in the input image into two categories, i.e., a box surface region of the to-be-detected box and a box surface region not of the to-be-detected box, according to the surface feature, and then segments the box surface region of the to-be-detected box from the input image according to the category of each pixel.

[0175] Exemplarily, Figure 14 is a schematic diagram of a box surface region provided by an embodiment of the present application. Referring to Figure 14 , the box surface region in the input image is the black region in the image 14-1.

[0176] S1022d, the box surface region is labeled from the input image to obtain a box surface detection result of the input image.

[0177] The deformation detection device labels the box surface region in the input image to obtain a box surface detection result, so that the staff can more easily distinguish the box surface region in the input image from other background regions.

[0178] Exemplarily, the present application provides a schematic diagram of a box surface detection result, as shown in Figure 15 The deformation detection device labels the box surface in the to-be-detected box in the input image with highlighted lines to obtain a box surface detection result 15-1, so that the staff can distinguish the to-be-detected box and the box surface from the background and other boxes.

[0179] In the embodiment of the present application, after the surface feature is extracted, the deformation detection device can segment the box surface region from the input image according to the surface feature, and label the box surface region from the input image, so as to obtain a box surface detection result that is easier for the staff to view.

[0180] In some embodiments of the present application, after the deformation region is labeled to obtain a deformation detection result of the input image, i.e., after S104, the method can further include S105, as follows:

[0181] S105, classifying the deformation detection result of the input image to obtain a deformation degree of the deformation detection result.

[0182] After obtaining the deformation detection result, the deformation detection device can obtain a preset deformation classifier, and then input the deformation detection result into the preset deformation classifier to identify the deformation degree of the deformation detection result by using the deformation classifier. It can be understood that the deformation degree can be represented by deformation levels, such as mild deformation, moderate deformation and severe deformation, etc.; or the deformation degree can be represented by deformation percentages, such as 100%, 50%, etc., which are not limited in the present application.

[0183] Exemplarily, the embodiments of the present application provide a diagram of the degree of formation, see Figure 16 The recognition result of the deformation detection result 16-1 by the deformation detection device is slight deformation 16-11, the recognition result of the deformation detection result 16-2 is moderate deformation 16-21, and the recognition result of the deformation detection result 16-3 is severe deformation 16-31.

[0184] It can be understood that the deformation detection result can include multiple labeled deformation sub-regions, and the deformation detection device determines the corresponding deformation level or deformation percentage for each labeled deformation sub-region, thereby obtaining the deformation degree of the deformation detection result.

[0185] In the embodiments of the present application, the deformation detection device can also identify the deformation degree of the deformation detection result, so that the staff can simply and directly judge the deformation condition of the to-be-detected box.

[0186] In some embodiments of the present application, after determining the deformation region from the input image based on the surface feature and the deformation feature, that is, after S103, the method can further include: S106, as follows:

[0187] S106, denoising the deformation region to obtain a denoised deformation region.

[0188] Since the deformation region is determined from the input image based on the surface feature and the deformation feature, pixel by pixel, it is very likely that the determined deformation region contains pixel regions that are too small, that is, only a few or tens of pixels. However, these small pixel regions actually do not constitute deformation, and therefore, in the deformation region, there are some noise. In the embodiments of the present application, the deformation detection device will denoise the deformation region after determining the deformation region, thereby removing the pixel regions that do not constitute deformation, to obtain a denoised deformation region, thereby reflecting a more realistic deformation condition to the staff.

[0189] Correspondingly, after obtaining the denoised deformation region, the deformation detection device subsequently labels the deformation region to obtain the deformation detection result of the input image, that is, the specific implementation of S104 will change to: S1041, as follows:

[0190] S1041, labeling the denoised deformation region to obtain the deformation detection result of the input image.

[0191] In the embodiments of the present application, the deformation detection device can remove the small pixel regions that do not constitute deformation from the deformation region, to realize denoising of the deformation region, obtain a denoised deformation region, and thereby obtain a more realistic deformation detection result, further improving the accuracy of the deformation detection result.

[0192] In some embodiments of this application, the deformed region includes multiple deformed sub-regions. Each deformed sub-region is a pixel region determined based on surface features and deformation features. In this case, denoising the deformed region to obtain the denoised deformed region, i.e., the specific implementation process of S106, may include: S1061-S1063, as follows:

[0193] S1061. For each of the multiple deformable sub-regions, calculate the area of ​​the region.

[0194] The deformation detection device calculates the area of ​​each deformation sub-region to obtain the area of ​​each sub-region. It can be understood that the deformation detection device can use the total number of pixels contained in each deformation sub-region as the area of ​​that sub-region, or it can use an area calculation function to calculate the area through integration; this application does not specify a particular limitation.

[0195] S1062. From multiple deformable sub-regions, filter the deformable sub-regions whose area is greater than or equal to the area threshold to obtain at least one candidate deformable sub-region.

[0196] S1063. Use at least one candidate deformation sub-region as the denoised deformation region.

[0197] The deformation detection device, after obtaining the sum of the areas corresponding to each deformation sub-region, acquires an area threshold. It then compares the area of ​​each deformation sub-region with the acquired area threshold, filtering out deformation sub-regions whose areas are greater than or equal to the area threshold, thus obtaining at least one candidate deformation sub-region. Next, the deformation detection device uses at least one candidate deformation sub-region to form a set, which is the denoised deformation region.

[0198] In this embodiment of the application, the deformation detection device can use the area of ​​each deformation sub-region to screen out at least one candidate deformation sub-region, so as to use at least one candidate deformation sub-region to form a denoised deformation region, thereby achieving denoising of the deformation region.

[0199] It should be noted that, in some embodiments of this application, in order to obtain a more accurate deformation area, the deformation detection device can also collect the depth data of the surface of the box to be detected after determining the deformation area based on the input image, verify the deformation area, obtain the verified deformation area, and then mark the verified deformation area to obtain the final deformation detection result, thereby obtaining a more accurate deformation detection result.

[0200] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0201] The embodiments of this application are implemented in the scenario of deformation detection of the surface of a container. Figure 17 This is a schematic diagram illustrating the process of deformation detection of a container panel according to an embodiment of this application. See also... Figure 17 The process includes:

[0202] S301, taking pictures.

[0203] Staff first use their mobile phones (terminals) to take photos of the container panel surface 17-1, and then upload the photos to the cloud (server; the equipment cluster consisting of the server and the terminal is a deformation detection device) as input for the container surface to be inspected (input image containing the surface of the container to be inspected).

[0204] S302, Classification of boxes to be inspected.

[0205] A multi-classifier (image classifier) ​​based on a residual network structure is designed in the cloud to classify the surfaces of container boxes in photos (classifying the surfaces of the container to be detected contained in the input image). The output of the multi-classifier includes left and right sides, front, top, bottom, door, and interior. This is to perform deformation detection processing only on the front, left, back, and top surfaces (surface types) of the container, while temporarily ignoring deformation detection on the doors and floor. The multi-classifier is designed using ResNet18 as its backbone network.

[0206] It should be noted that the container facade is defined by six sides based on the container's rectangular structure: the front, left, right, door, and floor. Because the left and right sides of the container are extremely similar, especially when photographed due to the angle, they are often indistinguishable. Therefore, the left and right sides are grouped together. Furthermore, since the deformation characteristics of the door and floor differ significantly from those on other panels, only the door and floor are distinguished, but deformation detection is not performed on them.

[0207] S303, Inspection of the panel surface of the box to be inspected and inspection of the deformation of the box to be inspected.

[0208] Based on a high-resolution network, a multi-branch network structure (target feature extraction model) is designed to extract features from images after classification, including any one or more sides of the board surface. Simultaneously, board surface features (surface features) and deformation features are obtained. Then, through an attention mechanism, the board surface features are used to eliminate interference from complex backgrounds (such as the bottom, sky, and other containers) in the photo, so as to facilitate the subsequent output of board surface region segmentation results and deformation region segmentation results.

[0209] For example, Figure 18is a schematic diagram of a multi-branch network structure provided by an embodiment of the present application. As shown in Figure 18 The cloud first extracts the overall features of the container panel photos 18-1 using the backbone network in the multi-branch network structure 18-2 to obtain an overall feature map (feature map), then extracts the panel features (surface features) using the panel branch 18-3 (surface feature extraction branch), and extracts the deformation features (deformation features) using the deformation branch 18-4 (deformation feature extraction branch). Then, the cloud strengthens the panel information in the deformation features and suppresses the background similar features based on the attention mechanism 18-5, thereby obtaining the deformation region segmentation result 18-6 (deformation detection result). At the same time, the cloud also segments the panel region segmentation result 18-7 (box surface detection result) using the panel features.

[0210] Further, the features in the panel region can effectively distinguish the difference between the panel and the background, so the features of the panel region have a certain promoting effect on deformation detection. In the attention mechanism, the cloud first performs maximum pooling and average pooling (multiple types of pooling operations) on the feature map extracted by the panel branch, i.e., the panel features, to obtain global features of different dimensions about the panel and the background (global feature maps corresponding to multiple types of pooling operations), then further fuses (obtains a surface global feature vector) through a shared fully connected layer (fully connected layer), and then fuses with the weight of the transformed panel and background features (the weight of the surface of the to-be-detected box in the input image), and finally obtains a channel attention weight vector (attention weight vector). Finally, the vector is point multiplied with the feature map of the deformation branch, i.e., the deformation features (obtaining the to-be-detected deformation features), thereby realizing the strengthening of the panel information in the deformation branch and the suppression of the background similar features.

[0211] In some cases, there may be very small deformation regions in the deformation detection result, or there may be very small holes in the panel detection result, so the cloud will denoise the output segmentation result (denoise the deformation region to obtain the denoised deformation region). At this time, the cloud can obtain all the regions in the segmentation map, calculate the area of each region (calculate the area of each sub-deformation region), and then remove the regions smaller than a certain threshold (equivalent to screening the deformation sub-regions with an area greater than or equal to the area threshold).

[0212] S304, deformation degree classification.

[0213] The serious deformation of the container panel has certain influence on the loading and stacking of the container, but the slight deformation has little influence, and therefore the classification of the deformation degree is very important for the delivery and repair and use of the container. The cloud can design a deformation degree classifier (preset deformation classifier) based on the residual network structure, classify the deformation area on the container panel (classify the deformation detection result of the input image), and obtain the deformation degree of each deformation area, i.e. slight, moderate and severe (deformation degree of the deformation detection result).

[0214] In the above manner, the deformation detection of the container can be intelligently realized, the labor cost is reduced, the detection efficiency is improved, meanwhile, the influence of the background and other containers on the deformation detection can be excluded, the error caused by human factors is reduced, and the precision of the deformation detection is improved.

[0215] The following continues to illustrate an exemplary structure of the implementation of the deformation detection device 555 provided in the embodiments of the present application as a software module. In some embodiments, as shown in Figure 5 The software module stored in the deformation detection device 555 of the memory 550 can include:

[0216] An image acquisition module 5551 is configured to acquire an input image containing a surface of a container to be detected; the container to be detected is a container waiting for deformation detection;

[0217] A feature extraction module 5552 is configured to perform parallel extraction of surface features and deformation features on the input image, to obtain surface features and deformation features corresponding to the input image; the surface features represent appearance information of an object in the input image, and the deformation features represent bending and wrinkle degree of the texture of the object in the input image;

[0218] A deformation determination module 5553 is configured to determine a deformation area from the input image based on the surface features and the deformation features; the deformation area is a region of the surface of the container to be detected where deformation occurs;

[0219] A result labeling module 5554 is configured to label the deformation area to obtain a deformation detection result of the input image.

[0220] In some embodiments of the present application, the deformation determination module 5553 is further configured to control attention of the deformation features according to the surface features to obtain a to-be-detected deformation feature; the to-be-detected deformation feature represents the deformation feature with attention focused on the bending and wrinkle degree of the surface of the container to be detected; and the deformation area is detected from the input image according to the to-be-detected deformation feature.

[0221] In some embodiments of the present application, the deformation determination module 5553 is further configured to calculate, by using the surface feature, an image weight occupied by the surface of the to-be-detected box in the input image; extract a global feature of the surface feature to obtain a surface global feature vector; calculate attention of the surface global feature vector by using the image weight to obtain an attention weight vector; and point-multiply the attention weight vector and the deformation feature to obtain the to-be-detected deformation feature.

[0222] In some embodiments of the present application, the deformation determination module 5553 is further configured to perform a plurality of types of pooling operations on the surface feature respectively to obtain a global feature map corresponding to each of the plurality of types of pooling operations; the global feature map includes features of the surface of the to-be-detected box and features of a background region; and fuse the global feature map corresponding to each of the plurality of types of pooling operations by a fully connected layer to obtain the surface global feature vector.

[0223] In some embodiments of the present application, the feature extraction module 5552 is further configured to classify, by using an image classifier, a surface of the to-be-detected box included in the input image to obtain a surface type; and when the surface type belongs to a preset type, perform, by using a target feature extraction model, parallel extraction of the surface feature and extraction of the deformation feature on the input image to obtain the surface feature and the deformation feature.

[0224] In some embodiments of the present application, the feature extraction module 5552 is further configured to perform, by using a backbone network in the target feature extraction model, image feature extraction on the input image to obtain a feature map of the input image; extract, by using a surface feature extraction branch in the target feature extraction model, the surface feature from the feature map; and extract, by using a deformation feature extraction branch in the target feature extraction model, the deformation feature from the feature map.

[0225] In some embodiments of the present application, the feature extraction module 5552 is further configured to segment, by using the surface feature, a box surface region of the to-be-detected box from the input image; and label the box surface region from the input image to obtain a box surface detection result of the input image.

[0226] In some embodiments of the present application, the deformation detection apparatus 555 further includes a deformation classification module 5555.

[0227] The deformation classification module 5555 is configured to classify the deformation detection result of the input image to obtain a deformation degree of the deformation detection result.

[0228] In some embodiments of the present application, the deformation detection apparatus 555 further comprises a deformation denoising module 5556.

[0229] The deformation denoising module 5556 is configured to denoise the deformation region to obtain a denoised deformation region.

[0230] The result labeling module 5554 is further configured to label the denoised deformation region to obtain the deformation detection result of the input image.

[0231] In some embodiments of the present application, the deformation region comprises a plurality of deformation sub-regions, and the deformation denoising module 5556 is further configured to calculate a region area for each deformation sub-region in the plurality of deformation sub-regions, select a deformation sub-region with a region area greater than or equal to an area threshold from the plurality of deformation sub-regions to obtain at least one candidate deformation sub-region, and take the at least one candidate deformation sub-region as the denoised deformation region.

[0232] In some embodiments of the present application, the feature extraction module 5552 is further configured to select a matching feature extraction model corresponding to the input image from a plurality of preset feature extraction models by using the surface type, and perform parallel extraction of surface features and deformation features on the input image by using the matching feature extraction model to obtain the surface features and the deformation features.

[0233] The embodiment of the present application provides a computer program product or a computer program, which comprises computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device executes the deformation detection method provided in the embodiment of the present application.

[0234] The embodiment of the present application provides a computer readable storage medium storing executable instructions, wherein the computer readable storage medium stores executable deformation detection instructions. When the executable deformation detection instructions are executed by a processor, the processor will execute the deformation detection method provided in the embodiment of the present application, for example, the method shown in the figure. Figure 6

[0235] In some embodiments, the computer readable storage medium can be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disc, or CD-ROM; or various devices comprising one or any combination of the above memories.

[0236] ​In some embodiments, the executable deformation detection instructions can take the form of a program, software, software modules, scripts, or code, written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and they can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0237] By way of example, the executable deformation detection instructions can, but need not necessarily, correspond to a file in a file system, can be stored in a portion of a file that holds other programs or data, for example, one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, for example, files that store one or more modules, sub programs, or portions of code.

[0238] By way of further example, the executable deformation detection instructions can be deployed for execution on one computer, or on multiple computers of a computing environment that span multiple locations, or multiple computers that are distributed among multiple locations and that are interconnected by a communication network.

[0239] The above description is intended to enable any person skilled in the art to make and use the present application, and is provided in the context of a particular application and its requirements. Various modifications to the embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein can be applied to other embodiments. Thus, the present application is not intended to be limited to the embodiments described herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A deformation detection method, characterized by, The method comprises the following steps: collecting an input image containing a surface of a box to be detected, wherein the box to be detected is a box waiting for deformation detection; extracting surface features and deformation features from the input image in parallel to obtain surface features and deformation features corresponding to the input image; the surface features represent appearance information of an object in the input image, and are used to distinguish a box region of the box to be detected and a background; the deformation features represent the bending and wrinkling degrees of the texture of the object in the input image; determining an image weight of the surface of the box to be detected in the input image by using a proportion of the surface of the box to be detected in the input image; extracting global features from the surface features to obtain a surface global feature vector; and calculating attention by using the image weight on the surface global feature vector to obtain an attention weight vector; point-multiplying the attention weight vector and the deformation features to obtain a deformation feature to be detected; wherein the attention weight vector is used to concentrate the attention on the feature part corresponding to the surface of the box to be detected, so as to suppress the interference term corresponding to the background; and the deformation feature to be detected represents the deformation feature with the attention concentrated on the bending and wrinkling degrees of the surface of the box to be detected; detecting a deformation region from the input image according to the deformation feature to be detected, wherein the deformation region is a region of the surface of the box to be detected that has been deformed; denoising the deformation region, and labeling the denoised deformation region to obtain a deformation detection result of the input image.

2. The method of claim 1, wherein, The method of extracting global features from the surface features to obtain a surface global feature vector comprises the following steps: performing a plurality of types of pooling operations on the surface features respectively to obtain a global feature map corresponding to each of the plurality of types of pooling operations; the global feature map comprises features of the surface of the box to be detected and features of a background region; fusing the global feature maps corresponding to the plurality of types of pooling operations by a fully connected layer to obtain the surface global feature vector.

3. The method according to claim 1 or 2, characterized in that, The method of extracting surface features and deformation features from the input image in parallel to obtain surface features and deformation features corresponding to the input image comprises the following steps: classifying the surface of the box to be detected contained in the input image by using an image classifier to obtain a surface type; when the surface type belongs to a preset type, extracting surface features and deformation features from the input image in parallel by using a target feature extraction model to obtain the surface features and the deformation features.

4. The method of claim 3, wherein, The method of extracting surface features and deformation features from the input image in parallel by using the target feature extraction model to obtain the surface features and the deformation features comprises the following steps: extracting image features of the input image by using a backbone network in the target feature extraction model to obtain a feature map of the input image; extract the surface feature from the feature map through a surface feature extraction branch in the target feature extraction model, and extract the deformation feature from the feature map through a deformation feature extraction branch in the target feature extraction model.

5. The method of claim 4, wherein, After the surface feature is extracted from the feature map through the surface feature extraction branch in the target feature extraction model, the method further includes: segmenting a box surface region of the box to be detected from the input image using the surface feature; annotating the box surface region from the input image to obtain a box surface detection result of the input image.

6. The method according to any one of claims 1 to 2, 4 or 5, characterized in that, After the de-noised deformation region is annotated to obtain the deformation detection result of the input image, the method further includes: classifying the deformation detection result of the input image to obtain a deformation degree of the deformation detection result.

7. The method of claim 1, wherein, The deformation region includes a plurality of deformation sub-regions, and the de-noising of the deformation region includes: calculating a region area for each deformation sub-region in the plurality of deformation sub-regions; from the plurality of deformation sub-regions, filtering deformation sub-regions with a region area greater than or equal to an area threshold to obtain at least one candidate deformation sub-region; the at least one candidate deformation sub-region is taken as the de-noised deformation region.

8. The method of claim 3, wherein, After the surface of the box to be detected contained in the input image is classified using the image classifier to obtain a surface type, the method further includes: using the surface type to select a matching feature extraction model corresponding to the input image from a plurality of preset feature extraction models; using the matching feature extraction model to perform parallel extraction of surface features and deformation features on the input image to obtain the surface features and the deformation features.

9. A deformation detection apparatus, characterized by comprising: includes: an image acquisition module configured to acquire an input image containing a surface of a box to be detected, the box to be detected being a box waiting for deformation detection; a feature extraction module configured to perform parallel extraction of surface features and deformation features on the input image to obtain surface features and deformation features corresponding to the input image; the surface features represent appearance information of an object in the input image, and are used to distinguish a box region of the box to be detected from a background; the deformation features represent a bending and wrinkle degree of a texture of the object in the input image; a deformation determination module configured to determine an image weight of the surface of the box to be detected in the input image using a proportion of a part of the surface of the box to be detected occupying the input image; extract global features from the surface features to obtain a surface global feature vector; and calculate attention of the surface global feature vector using the image weight to obtain an attention weight vector. Point-multiply the attention weight vector and the deformation feature to obtain a to-be-detected deformation feature; wherein the attention weight vector is used to concentrate the attention on the feature part corresponding to the surface of the to-be-detected box, so as to suppress the interference term corresponding to the background; the to-be-detected deformation feature represents the deformation feature of the bending and wrinkling degree of the surface of the to-be-detected box; According to the to-be-detected deformation feature, a deformation region is detected from the input image, wherein the deformation region is a region of the surface of the to-be-detected box that has been deformed; A result labeling module is configured to denoise the deformation region, label the denoised deformation region, and obtain a deformation detection result of the input image.

10. The apparatus of claim 9, wherein, The deformation determination module is specifically configured to: perform a plurality of types of pooling operations on the surface feature respectively to obtain a global feature map corresponding to each of the plurality of types of pooling operations; the global feature map includes the feature of the surface of the to-be-detected box and the feature of the background region; fuse the global feature map corresponding to each of the plurality of types of pooling operations through a fully connected layer to obtain the surface global feature vector.

11. The apparatus of any one of claims 9 to 10, wherein, The deformation determination module is specifically configured to: classify the surface of the to-be-detected box contained in the input image using an image classifier to obtain a surface type; when the surface type belongs to a preset type, perform parallel extraction of the surface feature and the deformation feature on the input image using a target feature extraction model to obtain the surface feature and the deformation feature.

12. A deformation detection apparatus, characterized by comprising: comprise: a memory configured to store executable deformation detection instructions; a processor configured to execute the executable deformation detection instructions stored in the memory to implement the method of any one of claims 1 to 8.

13. A computer-readable storage medium, characterized in that, executable deformation detection instructions are stored, and when executed by a processor, the method of any one of claims 1 to 8 is implemented.

14. A computer program product comprising computer programs or instructions, characterized in that, The computer program or instructions are executed by the processor to implement the deformation detection method of any one of claims 1 to 8.

Images