Container visual inspection system

Abstract

This system provides a container appearance inspection system that can perform accurate inspections even when sufficient training data is not available. [Solution] The system includes an imaging unit 2 that images the container 100, an abnormality inspection unit 41 that performs image processing on the captured image to emphasize the contours of abnormal areas to determine the presence and location of abnormal areas, a first determination unit 42 that uses a first AI to determine whether the captured image contains damage based on the results of the abnormality inspection unit 41, and a second determination unit 43 that uses a second AI to determine whether the captured image contains damage, regardless of the results of the abnormality inspection unit 41.

Description

Technical Field

[0001] The present invention relates to an appearance inspection system for cargo containers.

Background Art

[0002] Non-Patent Document 1 describes efforts to automate the damage check of containers that have conventionally been performed visually. Specifically, it is described that the state of the container is photographed with a camera, the obtained image is screened by a damage check system using AI, and if it is determined that there is damage, a separate visual confirmation is made. In addition to the determination based on the image, it is also described that the presence or absence of damage is determined based on three-dimensional data measured by a laser scanner.

Prior Art Documents

Patent Documents

[0003]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Generally, in a damage check system using AI, when the learning data is not sufficiently available immediately after the system is introduced, etc., the AI is immature and it is difficult to perform inspections with high accuracy.

[0005] An object of the present invention is to provide a container appearance inspection system that can perform inspections with high accuracy even when the learning data is not sufficiently available.

Means for Solving the Problems

[0006] ] The container appearance inspection system of the present invention is characterized by comprising: an imaging unit 2 that images a container 100; an abnormality inspection unit 41 that performs image processing on the captured image to emphasize the contours of abnormal areas to determine the presence and location of abnormal areas; a first determination unit 42 that uses a first AI to determine whether the captured image contains damage based on the results of the abnormality inspection unit 41; and a second determination unit 43 that uses a second AI to determine whether the captured image contains damage, without relying on the results of the abnormality inspection unit 41.

[0007] In the above-described container appearance inspection system, it is preferable that the first AI comprises multiple models, each of which has learned different types of damage from one another.

[0008] Furthermore, it is preferable that the first AI comprises multiple models trained using an ensemble method.

[0009] Furthermore, it is preferable to further include a result reflection unit 60 that reflects the results obtained from the first determination unit 42 and the second determination unit 43 into a single image.

[0010] Furthermore, it is preferable to further include a point cloud data acquisition unit 3 that acquires point cloud data of the three-dimensional coordinates of the container 100, and a three-dimensional determination unit 50 that determines whether or not there is damage based on the point cloud data.

[0011] In this case, it is preferable to further include a result reflection unit 60 that reflects the results obtained from the first determination unit 42, the second determination unit 43, and the three-dimensional determination unit 50 into a single image. [Effects of the Invention]

[0012] The container appearance inspection system of the present invention includes an abnormality inspection unit that performs image processing on an captured image to emphasize the contours of abnormal areas to determine the presence and location of abnormal areas; a first determination unit that uses a first AI to determine whether the captured image contains damage based on the results of the abnormality inspection unit; and a second determination unit that uses a second AI to determine whether the captured image contains damage, regardless of the results of the abnormality inspection unit. Therefore, it is possible to perform inspections with high accuracy even when sufficient training data is not available.

[0013] In the above container appearance inspection system, if the first AI has multiple models and each model has learned different types of damage from each other, it can determine not only the presence or absence of damage, but also the type of damage.

[0014] If the first AI has multiple models trained using an ensemble method, it can suppress misclassifications due to catastrophic forgetting, for example.

[0015] If the system also includes a result reflection unit that incorporates the results obtained from the first and second determination units into a single image, the determination results are easier to verify.

[0016] If the system further includes a point cloud data acquisition unit that acquires point cloud data of the container's 3D coordinates, and a 3D determination unit that determines whether or not there is damage based on the point cloud data, inspection can be performed with greater accuracy.

[0017] If the system also includes a result reflection unit that incorporates the results obtained from the first judgment unit, the second judgment unit, and the 3D judgment unit into a single image, it becomes easier to confirm the judgment results. [Brief explanation of the drawing]

[0018] [Figure 1] This is a block diagram of a container visual inspection system. [Figure 2] A flowchart of the image determination unit. [Figure 3] This is an explanatory diagram from the discomfort testing section to the first judgment section. [Figure 4] It is a flowchart of the three-dimensional determination unit. [Figure 5] It is a conceptual diagram showing the cleaning process of point cloud data. [Figure 6] It is a conceptual diagram showing the simplification process and the merging process of overlapping points. [Figure 7] It is a conceptual diagram showing the process of extracting the frontmost point. [Figure 8] It is a conceptual diagram showing a part of the process of calculating the average value. [Figure 9] It is a conceptual diagram showing the continuation of FIG. 8. [Figure 10] It is a conceptual diagram showing a reference line and an assumed damage. [Figure 11] It is a flowchart of the result determination unit and the display unit. [Figure 12] It is an image showing the determination result.

Mode for Carrying Out the Invention

[0019] Next, the container appearance inspection system 1 according to an embodiment of the present invention will be described. As shown in FIG. 1, the container appearance inspection system 1 includes an imaging unit 2 that images the container 100, a point cloud data acquisition unit 3 that acquires point cloud data of the three-dimensional coordinates of the container 100, a storage unit 20 that stores the imaged image and the point cloud data, and a central processing unit 30 that processes the imaged image and the point cloud data. The analysis unit 10. This container appearance inspection system 1 may include a display unit 70 that displays the result analyzed by the analysis unit 10.

[0020] The container 100 to be inspected is, for example, an international shipping container standardized by ISO standards. Specifically, it is a dry container. The overall shape is a roughly rectangular parallelepiped that is long from front to back, and the dimensions in the longitudinal direction are, for example, approximately 45 feet, approximately 40 feet, and approximately 20 feet. When the longitudinal direction is considered as the front-to-back direction, the surfaces to be inspected are, for example, the top surface 101 and both sides 102. The top surface 101 and both sides 102 are corrugated sheets with grooves 100a and protrusions 100b alternating in the front-to-back direction. The direction in which the grooves 100a and protrusions 100b extend is left-to-right on the top surface 101 and up-to-down on the sides 102. Also, the depth direction on the top surface 101 is up-to-down, and the depth direction on the sides 102 is left-to-right.

[0021] The imaging unit 2 is equipped with imaging means for color imaging the surface of the container 100. The imaging means is, for example, a two-dimensional color camera. A line sensor camera is preferred as the two-dimensional color camera. However, it is not limited to a line sensor camera; various still image cameras can be used. Furthermore, it is not limited to a still image camera; a video camera may also be used. In the case of a video camera, still images can be extracted from the video. It is preferable to provide three imaging means so that the top surface and both sides of the container can be imaged individually. Four or more imaging means may be provided, or there may be one or two.

[0022] The point cloud data acquisition unit 3 is equipped with a 3D measurement means for acquiring point cloud data of the 3D coordinates of the container surface. The 3D measurement means is, for example, LiDAR (Light Detection and Ranging). However, any known 3D measurement means that can acquire point cloud data of 3D coordinates can be used, not limited to LiDAR, but also including stereo cameras and other types of devices. It is preferable to provide three 3D measurement means so that the top surface and both sides of the container can be measured individually. Four or more units may be provided, or one or two units may be used.

[0023] The imaging and 3D measuring means are mounted, for example, on a gate-type gate 110. By passing the container 100 through this gate 110, imaging is performed by the imaging unit 2 and point cloud data is acquired by the point cloud data acquisition unit 3. When imaging or acquiring point cloud data, the container 100 is moved relative to the imaging and 3D measuring means in the front-to-back direction. For example, the container 100 is placed on a transport means such as a truck and passed through the gate 110. However, the container 100 may be fixed and the imaging and 3D measuring means may be moved. Of course, imaging and point cloud data acquisition may also be performed without moving the container 100, imaging, or 3D measuring means. Furthermore, imaging and point cloud data acquisition may be performed with multiple containers 100 lined up in the front-to-back direction. In this case, it may be possible to select only one of the containers 100 as the inspection target.

[0024] An entry sensor, such as a photoelectric sensor, may be provided so that imaging and point cloud data acquisition can begin when the container 100 reaches a predetermined position. Similarly, an exit sensor, such as a photoelectric sensor, may be provided so that imaging and point cloud data acquisition can end when the container 100 moves away from the predetermined position. The entry and exit sensors may be combined into a single sensor. Additionally, lighting may be provided to illuminate the inspection surface of the container 100.

[0025] The captured images and measured point cloud data are sent to the analysis unit 10. The analysis unit 10 is equipped with a so-called server. The server is equipped with a central processing unit (processor such as a CPU or GPU) 30 and a storage unit (storage medium such as an SSD, HDD, or memory) 20. There may be one central processing unit 30 or multiple storage units 20. That is, each of the processes described later may be processed by one central processing unit 30, or the processing may be divided among multiple central processing units 30. In addition, the captured images and measured point cloud data, as well as processed images obtained by applying some processing to the captured images and processed data obtained by applying some processing to the point cloud data, may be stored in one common storage unit 20, or they may be stored separately in multiple storage units 20.

[0026] The central processing unit 30 includes an image determination unit 40 that determines the presence or absence of damage based on the captured image, a 3D determination unit 50 that determines the presence or absence of damage based on point cloud data, and a result reflection unit 60 that reflects the determination results obtained from the image determination unit 40 and the 3D determination unit 50 into a single image. The image determination unit 40, the 3D determination unit 50, and the result reflection unit 60 are programs that perform analysis in cooperation with the central processing unit 30.

[0027] The image determination unit 40 includes an abnormality inspection unit 41 that performs image processing on the captured image to emphasize the contours of abnormal areas to determine the presence and location of abnormal areas; a first determination unit 42 that uses a first AI (image recognition AI) to determine whether the captured image contains damage and what type of damage it is, based on the results of the abnormality inspection unit 41; and a second determination unit 43 that uses a second AI (object detection AI) to determine whether the captured image contains damage and what type of damage it is, independently of the results of the abnormality inspection unit 41 (i.e., regardless of the results of the abnormality inspection unit 41). AI stands for Artificial Intelligence.

[0028] The 3D determination unit 50 includes a reference line formation unit 52 that sets a reference line extending in the vertical axis direction based on each point constituting the point cloud data, and a concave / concave determination unit 53 that determines whether the damage is convex or concave based on the reference line. The 3D determination unit 50 also includes a 3D CAD (Computer Aided Design) 51 for inputting and rendering the point cloud data.

[0029] The analysis results performed by the analysis unit 10 are displayed on the display unit 70. The display unit 70 is equipped with a display means, such as a monitor. The display program for displaying the analysis results on the display means may be executed in the analysis unit 10, or a separate computer may be provided in the display unit 70 and the program may be executed by that computer. The computer may be, for example, a desktop computer, a laptop computer, or a mobile computing device such as a smartphone or tablet. Alternatively, projection or printing may be used instead of displaying on the display unit 70.

[0030] The imaging unit 2 and the analysis unit 10 can be connected in any way that allows the captured images to be sent to the analysis unit 10. For example, they may be connected by a communication cable or by a network such as an internet connection. In the case of an internet connection, it may be wired or wireless. The same applies to the point cloud data acquisition unit 3 and the analysis unit 10, and the analysis unit 10 and the display unit 70. In other words, any connection that allows the necessary information to be transmitted and / or received is acceptable.

[0031] Next, we will explain each process of the container appearance inspection system 1. In this container appearance inspection system 1, as shown in Figure 1, the process is divided into judgment by the image determination unit 40 and judgment by the 3D determination unit 50. Furthermore, the image determination unit 40 is divided into a route that passes through the first determination unit 42 and a route that passes through the second determination unit 43. Therefore, we will first explain the route that passes through the first determination unit 42 among the judgments made by the image determination unit 40.

[0032] In the route passing through the first determination unit 42, as shown in Figure 2, first, an image of the surface to be inspected is acquired by the imaging unit 2 (S1). Next, the acquired image is stored in the storage unit 20 (S2). Next, the abnormality inspection unit 41 of the image determination unit 40 emphasizes the contours of the abnormal areas (S3). Abnormal areas are areas that are different from normal areas (abnormal areas), and their contours can be emphasized by applying automatic gain adjustment, gamma correction, and saturation correction. In the abnormality inspection unit 41, as shown in Figure 3, one original image is duplicated into three copies: one for automatic gain adjustment, one for gamma correction, and one for saturation correction. Different adjustments and corrections are applied to each original image to emphasize the contours of the abnormal areas.

[0033] Automatic gain adjustment automatically corrects only the luminance component of the color representation. Specifically, it first converts the original RGB image to Lab format. Next, it applies auto-gain control to the L channel, which is the luminance component, to flatten the histogram (adjust the overall luminance balance of the image). After that, it recombines the L channel with the a and b channels and converts it back from Lab format to RGB format to return to the original color system. Automatic gain adjustment emphasizes differences in luminance and contrast, making abnormal areas visually more prominent.

[0034] Gamma correction applies gamma correction to the entire original image. Gamma correction enhances the dark and bright areas of the image, making abnormal areas visually more prominent. By creating a lookup table (conversion table) for gamma correction in advance, the value of each pixel in the image can be corrected efficiently, speeding up processing and allowing correction to be applied quickly even to large numbers of images. The lookup table is created by calculating the inverse gamma value and then calculating the gamma correction value for each pixel value. Specifically, the reciprocal gamma value used for gamma correction is calculated, this value is used to perform a non-linear transformation of the pixel value, and then the result of gamma correction is calculated for pixel values ​​from 0 to 255, and the result is reflected (stored) in the lookup table.

[0035] Saturation correction first involves converting the color system to adjust the saturation (degree of color saturation), one of the elements that represent color. Specifically, the original image is converted from RGB format to HSV format. Next, gain is applied to the saturation (degree of color saturation). After that, it is converted back to RGB format.

[0036] To explain in more detail, first, the original image is converted from RGB format to HSV format, and then the brightness contrast of the original image is adjusted. Specifically, the brightness and constructor are adjusted to reduce the effect of shadows. More specifically, the brightness is divided by the corresponding amount of blur (contrast adjustment by blurring). In other words, local brightness fluctuations are smoothed out and divided by the original brightness. This makes it easier to detect anomalies even in environments where the lighting effect is unstable. Also, because it is converted to HSV format, only the brightness can be adjusted independently without affecting the hue or saturation.

[0037] Next, the color with the largest number of occurrences in the hue histogram is selected as the container color. Since the surface of container 100 is generally monochromatic, the container color can be obtained by taking the maximum value in the hue histogram. Next, the hue is shifted so that the container color is close to yellow. Specifically, although the rust on the container is close to red, in the color wheel, red is located at the upper and lower limits of the numerical values, so the values ​​are shifted so that red is located near the center. Due to the characteristics of container 100, damaged areas are prone to peeling paint and rust. Therefore, emphasizing the rust makes it easier to detect damage.

[0038] Next, H, S, and V are merged (recombined). Then, the position of the maximum value is obtained from the hue histogram. This allows the container color to be selected. Next, the next most frequent color is searched for. Specifically, the lower limit is searched in reverse order from the maximum value. More specifically, the left side of the histogram (towards 0 degrees of hue) from the container color is searched. Also, the upper limit is searched sequentially from the maximum value, for example, up to 360 degrees. Specifically, the right side of the histogram (towards 360 degrees of hue) from the container color is searched for. Note that the above 360 ​​degrees can be arbitrarily determined within the hue definition range (0 to 360). The searched color is then determined to be the color of the discrepancy. Note that in the case of extremely bright color schemes, there is a possibility of overexposure, so in this case, this color is excluded and the next most frequent color is considered the color of the discrepancy.

[0039] Next, we search for a lower limit from the mean value of the brightness histogram. We specify the brightness at the same coordinates as the discrepancy detected by hue. Brightness represents the shape of an object in an image, and it is possible to extract the contour of the discrepancy.

[0040] Whether there are any unnatural areas in images that have undergone automatic gain adjustment, gamma correction, or saturation correction is determined by analyzing the hue histogram. Specifically, using the container color as a reference, it is checked whether there are any colors (abnormal colors) on the histogram that have peaks that are clearly different from the container color. If such abnormal colors are detected, they are determined to be unnatural areas.

[0041] If no anomalies are found, that is, if no large peaks other than the container color are detected on the histogram, the first judgment unit will not perform a judgment, and the process will terminate. Note that the above series of steps are performed by a program, not by a human.

[0042] Next, the location and size of the abnormal area are determined from the image in which the abnormal area is identified, and based on this location and size information, the abnormal area is cut out from the captured image (i.e., the original image that has not undergone image processing to emphasize the outline of the abnormal area). In other words, an abnormal image containing the abnormal area is extracted (S4). When extracting the abnormal area, the abnormal area is cut out, for example, by cutting it into a rectangle, but if the cut-out area overlaps with other cut-out areas, or if a cut-out area contains another cut-out area, it may be combined into one. In addition, the image may be processed separately (resizing, contrast enhancement, noise reduction, normalization (adjustment of brightness and saturation)) to make it easier for the first AI of the first determination unit 42 to make a determination.

[0043] Then, the first determination unit 42 determines this abnormal image (S5). The damage determined here includes, for example, bending, holes, cuts, scratches, breakage, rust, cracks, improper repairs, and others. The first AI used for determination is equipped with multiple models (trained models), and each model has learned different types of damage from one another.

[0044] The determination process involves having a rust model, trained only on rust, analyze a given image to determine if it has rust, and then having a hole model, trained only on holes, analyze the same image to determine if it has holes. In other words, a model trained on only one type of damage is prepared for each type of damage, and the same image is analyzed by each model to accurately classify the type of damage.

[0045] Furthermore, even if any type of damage is detected, the detection process does not end there; instead, it is continued using other (preferably all) models. In other words, the presence or absence of multiple (preferably all) types of damage is checked. This ensures that even if multiple types of damage are present in one location, they can be detected without fail.

[0046] If the damage does not fall into any category, it is determined to be an abnormality area (an area where some kind of damage is suspected, but the type of damage is not clear). The determination result (type of damage, location, size, location and size of the abnormality area) is sent to the result reflection unit 60 (S7).

[0047] Next, the route through the second determination unit 43 will be explained. First, the imaging unit 2 acquires an image of the surface to be inspected (S1). Next, the acquired image is stored in the storage unit 20 (S2). Then, the original image is determined by the second determination unit 43 (S6). The second AI used for determination, like the first AI in the first determination unit 42, is equipped with multiple models (trained models), and each model has learned different types of damage from one another.

[0048] The determination is made by having a rust model, trained to recognize only rust, examine the original image to determine if rust is present, and then having a perforation model, trained to recognize only holes, examine the same original image to determine if there are holes. In other words, a model trained to recognize only one type of damage is prepared for each type of damage, and each model is made to examine the same original image to accurately classify the type of damage. This allows for accurate classification of the type of damage, similar to the first AI. The types of damage determined here are the same as those determined by the first determination unit 42. In short, a double check is performed between the first AI and the second AI.

[0049] Furthermore, even if any type of damage is detected, the detection process does not end there; instead, it is continued using other (preferably all) models. In other words, the presence or absence of multiple (preferably all) types of damage is checked. This ensures that even if multiple types of damage are present in one location, they can be detected without fail.

[0050] The judgment result (type, location, and size of damage) is sent to the result reflection unit 60 (S7).

[0051] Next, the determination by the 3D determination unit 50 will be explained. The damage determined by the 3D determination unit 50 is convex damage (e.g., bulges) and concave damage (e.g., indentations), which are different from the damage that the image determination unit 40 determines. However, it is of course possible for the image determination unit 40 to determine the presence or absence of convex and concave damage.

[0052] In the 3D determination unit 50, as shown in Figure 4, first, the point cloud data acquisition unit 3 acquires point cloud data of the 3D coordinates (vertical axis, horizontal axis, depth) of the surface to be inspected (S11). The vertical axis is an axis substantially parallel to the direction in which the grooves 100a and protrusions 100b extend, the horizontal axis is an axis substantially parallel to the direction in which the grooves 100a and protrusions 100b alternately continue, and the depth is a direction substantially parallel to the direction of the recess of the grooves 100a and the direction of the protrusions 100b, and is perpendicular to both the vertical axis and the horizontal axis. For example, if the side surface 102 of the container 100 is the surface to be inspected, the vertical axis is the up-down direction of the container 100, the horizontal axis is the front-to-back direction of the container 100, and the depth is the left-to-right direction of the container 100.

[0053] Next, the point cloud data is stored in the storage unit 20 (S12). Next, the point cloud data is input into the 3D CAD 51 (S13). Next, as shown in Figure 5, points that are not within the inspection range are deleted from among the points that make up the point cloud data (S14).

[0054] Next, as shown in Figure 6, all numerical values ​​for the vertical axis, horizontal axis, and depth of each point constituting the point cloud data are rounded or truncated to a predetermined number of decimal places to simplify the data (S15). For example, if the unit is mm, the ones digit is rounded or truncated. As a result, all points are located at the intersections of a 3D grid with a spacing of 10 mm. Next, points whose three numerical values ​​for the vertical axis, horizontal axis, and depth overlap after simplification are merged into a single point (S16).

[0055] Next, as shown in Figure 7, among the multiple points where the two numerical values ​​on the vertical and horizontal axes overlap due to simplification, the foremost point with the lowest depth value is extracted (S17).

[0056] In addition to the corrugated sheet metal panels that are the surface to be inspected, container 100 also has other components such as posts (columns) and rails (beams) arranged around the outer perimeter of the panels. However, if the point cloud includes those of these other components, it can cause misjudgments, so these points are deleted at this stage. Furthermore, when using LiDAR as a 3D measurement method, due to the characteristics of LiDAR, a shift in the depth direction occurs at points at the same height as the LiDAR installation height (height on the vertical axis), so these points are deleted.

[0057] Next, among the frontmost points, those with overlapping values ​​on the horizontal axis are grouped together (S18). Then, as shown in Figures 8 to 10, the grouped points are divided into a group with low values ​​on the vertical axis and a group with high values ​​on the vertical axis. The average value of the depth is calculated for each group, and the point closest to this average is placed at the lower limit of the inspection range for the group with low values ​​on the vertical axis, and at the upper limit of the inspection range for the group with high values ​​on the vertical axis. A baseline is then created by connecting these two points (S19). In grouping, instead of dividing all points arranged vertically into one group or the other, a predetermined number of points are obtained, for example, up to 5 points above the lowest point and up to 5 points below the highest point. Also, when calculating the average value, outliers are excluded. Here, an outlier is a point with the same value on the horizontal axis but whose depth value does not overlap with any other point.

[0058] Next, the type of damage is determined to be convex and / or concave based on the reference line. Specifically, points that are more than a predetermined standard away from the reference line on the near side in the depth direction are assumed to be convex damage, and points that are more than a predetermined standard away from the reference line on the far side in the depth direction are assumed to be concave damage (S20). The assumed points are distinguished from other points by coloring them or by other means.

[0059] Next, the inspection area is divided into a grid, and if the proportion of points assumed to have convex damage among all points in one cell of the grid is above a predetermined standard, the entire cell is determined to have convex damage. Similarly, if the proportion of points assumed to have concave damage among all points in one cell of the grid is above a predetermined standard, the entire cell is determined to have concave damage (S21).

[0060] The grid size is arbitrary, but it should be large enough to contain multiple points in at least one grid. That is, it should be at least larger than the number of digits that can be grouped by simplification (S15). However, if it is made too large, the proportion of points assumed to be damage will become small, making it impossible to detect damage. Therefore, it is preferable to make it the same size as or smaller than the dimensions of the damage to be detected.

[0061] If adjacent cells have the same damage (for example, convex damage), they are merged as a single damage. The judgment result includes 3D coordinates, but after being converted to 2D coordinates by removing the depth coordinate, it is sent to the result reflection unit 60 (S7). Note that the above series of steps are performed by a program, not by a person.

[0062] The results may include whether the inspection was successful or unsuccessful, the container size (45ft, 40ft, or 20ft), the type of damage (concave or convex), the coordinates of the damage, the size of the damage, the maximum depth / height of the damage, and the average depth / height of the damage. The results may also be output in a file format for CAD software.

[0063] The judgment results obtained by the first judgment unit 42, the second judgment unit 43, and the 3D judgment unit 50 are sent to the result reflection unit 60 along with the type of damage, the size of the damage, the location information of the damage, and the size and location information of the abnormality area, as shown in Figure 11, and are reflected in a single image (S7). The single image is the original image used for the inspection. The method for reflecting the judgment results is a marking process that surrounds the areas determined to have damage or areas identified as abnormality (see Figure 12). The color, shape, and size of the marks may be changed to distinguish between damage and abnormality areas, or between different types of damage. It is preferable to adjust the size of the marking frames and marks so that they do not overlap with the damage.

[0064] The image into which the judgment result has been reflected by the result reflection unit 60 is displayed on the display unit 70 (S8). The image into which the judgment result has been reflected may also be stored in the storage unit 20.

[0065] Next, the AI ​​of the image classification unit 40 will be described. The first AI is equipped with multiple models (trained models) that have been trained using an ensemble method. Specifically, in order to suppress misclassification due to catastrophic forgetting, when additional training is performed on an existing model, a new model is created that has been trained only on the additional training portion, and the classification is performed using two models: the existing model and the new model. It is preferable that all models that have learned different types of damage have both old and new models, but it is also acceptable for only one of the models to have both old and new models. Note that the number of these old and new models is not limited to two, but may be three or more. If the classification results differ among multiple models, the classification result may be decided by majority vote. However, if one model determines that there is damage, it may be decided that there is damage even if other models determine that there is no damage. The second AI is also equipped with multiple models that have been trained using an ensemble method.

[0066] The training data used to train the first AI is for image recognition purposes, and uses data to determine whether or not there is damage, such as scratches, cuts, and holes. For this reason, the first AI is trained using supervised learning, and is learned based on two types of images: those that show damage and those that do not.

[0067] On the other hand, the training data used to train the second AI is for object recognition purposes, and uses training data in which the type and location of damage are annotated on container images. This data accurately annotates abnormal areas such as rust and scratches. Therefore, the second AI is undergoing so-called supervised learning.

[0068] One example of a machine learning model is a neural network model. The first AI uses a neural network-based model to perform image recognition, while the second AI uses an advanced object recognition model to recognize the location and type of detected damage in detail.

[0069] In the container appearance inspection system 1 with the above configuration, the first AI can classify the type of damage, and for example, even if scratches and rust are mixed in the same image, it can classify the scratches and rust separately. In addition, the second AI can quickly determine the presence or absence of damage. Furthermore, the abnormality inspection unit 41 can determine the presence or absence of abnormalities, so inspections can be performed even in situations where there is insufficient training data or the AI ​​training period is not adequately secured, such as in the initial stages of system introduction. Moreover, in the 3D determination unit 50, a reference line extending in the vertical axis direction is provided, and the presence or absence of convex or concave damage is determined based on this reference line, so the influence of grooves 100a and protrusions 100b on the surface of the container 100 can be reduced, and inspection can be performed with high accuracy.

[0070] Although embodiments of this invention have been described above, this invention is not limited to the embodiments described above, and can be implemented with various modifications within the scope of this invention.

[0071] If the captured image contains objects that are not subject to inspection, such as the background, a cleaning step to remove these objects may be included, for example, between the storage step S2 and the contour enhancement step S3. Similarly, if the point cloud data contains objects that are not subject to inspection, a cleaning step to remove these objects may be included, for example, between the storage step S12 and the 3D CAD input step S13.

[0072] In addition to extracting images of anomalies (S4), the AI ​​can also be given only the location of the anomaly, and based on that location, the AI ​​can determine whether or not there is damage and what type of damage it is.

[0073] The inspection target of the image determination unit 40 is not limited to the corrugated sheet metal panel, but may also include the four posts, the rails spanning between the posts (bottom rail, upper rail), the corner castings provided at the four corners, the ventilators provided at the upper corners of the panel, and the fork pockets provided on the bottom rail. However, the inspection target of the 3D determination unit is limited to the panel only. If multiple containers 100 are loaded on the truck bed, it may be possible to specify which container 100 to inspect. [Explanation of Symbols]

[0074] 1. Container Visual Inspection System 2 Imaging Unit 3. Point cloud data acquisition unit 10 Analysis Department 20 Memory section 30 Central processing unit 40 Image determination unit 41 Discomfort Examination Department 42 1st Judgment Section 43 Second Judgment Section 50 3D judgment section 51 3D CAD 52 Reference line forming section 53 Unevenness determination section 60 Result reflection section 70 Display section 100 containers 100a groove 100b protrusion 101 Top surface 102 Side view Gate 110

Claims

1. An imaging unit that images the container, An abnormality detection unit performs image processing on the captured image to enhance the contours of areas of abnormality, thereby determining the presence and location of such areas. Based on the results of the abnormality examination unit, the first determination unit uses the first AI to determine whether the captured image contains damage, A second determination unit, which uses a second AI to determine whether the captured image contains damage, is not based on the results of the abnormality examination unit. A container visual inspection system equipped with the following features.

2. The container appearance inspection system according to claim 1, wherein the first AI comprises multiple models, each model learning different types of damage from one another.

3. The container appearance inspection system according to claim 1, wherein the first AI comprises multiple models learned by an ensemble method.

4. The container appearance inspection system according to claim 1, further comprising a result reflection unit that reflects the results obtained from a first determination unit and a second determination unit into a single image.

5. A point cloud data acquisition unit that acquires point cloud data of the container's 3D coordinates, The container appearance inspection system according to claim 1, further comprising a three-dimensional determination unit that determines the presence or absence of damage based on point cloud data.

6. The container appearance inspection system according to claim 5, further comprising a result reflection unit that reflects the results obtained from the first determination unit, the second determination unit, and the three-dimensional determination unit into a single image.

Images