Visual inspection instrument, method, device, electronic device and storage medium

Abstract

The embodiments of the present invention disclose a visual inspection instrument, method, device, electronic device and storage medium. In the visual inspection instrument, a multi-eye camera is used to take pictures of the device to be inspected at a preset frequency when the visual inspection instrument is moving and all first light sources are turned on, so as to obtain multiple first inspection images, so as to perform first category defect detection on the device to be inspected based on each first inspection image; a photometric camera is used to take pictures of the device to be inspected when the visual inspection instrument is stationary and all second light sources are turned on and triggered, so as to obtain a full-brightness image; the photometric camera is also used to take pictures of the device to be inspected once when the visual inspection instrument is stationary and all second light sources are turned on and triggered in sequence, so as to obtain multiple second inspection images, so as to perform second category defect detection on the device to be inspected based on each second inspection image and the full-brightness image. The visual inspection instrument can take multiple types of pictures of the surface of the device to realize multi-type defect detection.

Description

Technical Field

[0001] The present invention relates to the field of machine vision technology, and in particular to a visual inspection instrument, method, device, electronic equipment and storage medium. Background Art

[0002] For large equipment, such as aircraft, surface defects can pose a fatal threat. Furthermore, large equipment often has complex surface textures, component contours with diverse geometric structures, and a wide variety of products, making surface defect detection on large equipment very inconvenient.

[0003] Currently, surface quality inspection for large equipment relies primarily on traditional manual visual inspection. However, visual inspection has limitations and cannot be used for some surface quality inspection projects that require quantitative analysis. Furthermore, manual visual inspection is affected by factors such as the inspector's individual experience, mental state, and fatigue level, lacking stability and objectivity, and is prone to missed inspections. Summary of the Invention

[0004] The present invention provides a visual inspection instrument, method, device, electronic equipment and storage medium for performing defect detection on the surface of a device.

[0005] According to one aspect of the present invention, a visual inspection instrument is provided, comprising: a multi-camera, a photometric camera, at least one first light source, and at least one second light source; wherein:

[0006] The multi-lens camera is configured to photograph the device to be inspected at a preset frequency when the visual inspection instrument is moving and all the first light sources are turned on, thereby obtaining a plurality of first inspection images, and performing first category defect inspection on the device to be inspected based on the first inspection images;

[0007] The photometric camera is used to take a picture of the device to be inspected to obtain a fully illuminated image when the visual inspection instrument is stationary and all the second light sources are turned on and triggered;

[0008] The photometric camera is also used to take a picture of the device to be inspected once when the visual inspection instrument is stationary and each of the second light sources is turned on and triggered in sequence, so as to obtain multiple second inspection images, so as to perform second category defect detection on the device to be inspected based on each of the second inspection images and the full-brightness image.

[0009] According to another aspect of the present invention, a visual inspection method is provided. The method is applied to the visual inspection instrument provided by any embodiment of the present invention, comprising: using a multi-lens camera, while the visual inspection instrument is moving and one or more first light sources are all turned on, photographing a device to be inspected at a preset frequency to obtain multiple first inspection images;

[0010] When detecting, based on each of the first detection images, that a position arrangement of a plurality of target detection objects in the image is offset, determining that a first category defect exists in the device to be detected;

[0011] When the visual inspection instrument is stationary and the plurality of second light sources are all turned on and triggered, a photometric camera is used to take a picture of the device to be inspected to obtain a fully illuminated image;

[0012] When the visual inspection instrument is stationary and each of the second light sources is sequentially turned on and triggered, a photometric camera is used to take a picture of the device to be inspected to obtain a plurality of second inspection images;

[0013] When a target shadow is detected in the image according to the full-bright image and each of the second detection images, it is determined that the device to be inspected has a second category defect.

[0014] According to another aspect of the present invention, a visual inspection device is provided. The device is applied to the visual inspection instrument provided in any embodiment of the present invention, comprising: a first detection image acquisition module, configured to use a multi-lens camera to photograph a device to be inspected at a preset frequency while the visual inspection instrument is moving and one or more first light sources are all turned on, thereby obtaining a plurality of first detection images;

[0015] A first category defect determination module is configured to determine that a first category defect exists in the device to be inspected when, based on each of the first inspection images, a positional arrangement of multiple target inspection objects in the image is offset;

[0016] a full-brightness image acquisition module, configured to take a photo of the device to be inspected by using a photometric camera when the visual inspection instrument is stationary and the plurality of second light sources are all turned on and triggered, thereby obtaining a full-brightness image;

[0017] A second detection image acquisition module is configured to use a photometric camera to take a picture of the device to be detected once when the visual detection instrument is stationary and each of the second light sources is triggered in sequence, thereby obtaining a plurality of second detection images;

[0018] The second category defect acquisition module is used to determine that the device to be inspected has a second category defect when a target shadow is detected in the image based on the full-bright image and each of the second inspection images.

[0019] According to another aspect of the present invention, an electronic device is provided, comprising:

[0020] A multi-camera, a photometric camera, at least one first light source, and at least one second light source;

[0021] at least one processor; and

[0022] a memory communicatively connected to the at least one processor; wherein,

[0023] The memory stores a computer program that can be executed by the at least one processor. The computer program is executed by the at least one processor to enable the at least one processor to perform the visual inspection method described in any embodiment of the present invention.

[0024] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the visual inspection method according to any embodiment of the present invention when executed.

[0025] The technical solution of an embodiment of the present invention is to arrange a visual inspection instrument including a multi-eye camera, a photometric camera, at least one first light source, and at least one second light source, wherein: the multi-eye camera is used to take pictures of the device to be inspected at a preset frequency when the visual inspection instrument is moving and all the first light sources are turned on, so as to obtain multiple first inspection images, so as to perform first category defect detection on the device to be inspected based on each first inspection image; the photometric camera is used to take pictures of the device to be inspected when the visual inspection instrument is stationary and all the second light sources are turned on and triggered, so as to obtain a full-bright image; the photometric camera is also used to take pictures of the device to be inspected once when the visual inspection instrument is stationary and all the second light sources are turned on and triggered in sequence, so as to obtain multiple second inspection images, so as to perform second category defect detection on the device to be inspected based on each second detection image and the full-bright image, thereby solving the problem of device surface defect detection. Multiple types of pictures can be taken on the device surface to realize multi-type defect detection on the device surface.

[0026] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0028] Figure 1a This is a structural diagram of a visual inspection instrument provided according to the first embodiment of the present invention;

[0029] Figure 1b 1 is a schematic diagram of the layout of a visual inspection instrument provided according to the first embodiment of the present invention;

[0030] Figure 2 is a flow chart of a visual inspection method provided according to the second embodiment of the present invention;

[0031] Figure 3 2 is a schematic structural diagram of a visual inspection device provided according to a third embodiment of the present invention;

[0032] Figure 4 3 is a schematic diagram of the structure of an electronic device for implementing the visual detection method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0033] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0034] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0035] Example 1

[0036] Figure 1a FIG. 1 is a schematic diagram of the structure of a visual inspection instrument according to the first embodiment of the present invention. This embodiment is applicable to the case where a visual inspection instrument is used to detect defects on the surface of an aircraft. Figure 1aAs shown, the visual inspection instrument includes: a multi-eye camera 110, a photometric camera 120, at least one first light source 130, and at least one second light source 140; wherein:

[0037] The multi-eye camera 110 is used to take pictures of the device to be inspected at a preset frequency when the visual inspection instrument is moving and all the first light sources 130 are turned on, to obtain multiple first inspection images, so as to perform first category defect inspection on the device to be inspected based on each first inspection image.

[0038] Among them, such as Figure 1a As shown, the multi-camera 110 can be a triangular camera. The multi-camera 110 can be arranged in an equilateral triangle, with the three cameras equidistant from each other. The multi-camera 110 can take pictures of the surface of the device while the visual inspection instrument is moving.

[0039] Specifically, for large equipment, such as aircraft, a single panoramic image is difficult due to its large surface area. Therefore, multiple inspection points can be pre-set, and the visual inspection instrument can move between these points for photography, or remain stationary at each inspection point to capture the entire surface of the equipment. Furthermore, between adjacent inspection points, as the visual inspection instrument moves, a multi-lens camera can capture images of the equipment surface at a preset frequency. At each inspection point, while the visual inspection instrument remains stationary, a photometric camera can capture images of the equipment surface.

[0040] The preset frequency can be a relatively high value to ensure that overlapping areas exist in the first inspection images captured by the multi-camera system. During the multi-camera system's capture, a first light source can illuminate the captured area. The first light source can be a bar-shaped light source. The first inspection image captured by the multi-camera system can be an image of the common area captured by the three cameras, thereby improving the accuracy of defect detection.

[0041] Category 1 defect detection involves checking the position of target objects within the equipment. For example, this could be detecting whether rivets in an aircraft are misaligned. Multi-lens cameras have 3D positioning capabilities, enabling them to measure the distance between target images, thereby determining whether a Category 1 defect exists within the equipment.

[0042] like Figure 1a As shown, the photometric camera 120 is configured to photograph the device under inspection when the visual inspection instrument is stationary and all second light sources 140 are activated and triggered, thereby obtaining a fully illuminated image. The photometric camera 120 is also configured to photograph each device under inspection when the visual inspection instrument is stationary and all second light sources 140 are activated and triggered sequentially, thereby obtaining multiple second inspection images. The device under inspection is then inspected for defects of the second category based on the second inspection images and the fully illuminated image.

[0043] The number of the second light sources may be three, four, or six, etc. The second light sources may be point light sources. The second light sources may be evenly distributed around the photometric camera, illuminating the photometric camera at a certain tilt angle. When acquiring the second detection image, the second light sources may be activated in sequence, with only one second light source activated at a time.

[0044] Second-category defect detection can include detection of bumps, paint peeling, scratches, corrosion, or dents on the surface of the device. When second-category defects exist on the surface of the device to be inspected, shadows will appear in the second inspection image captured by the photometric camera due to the height differences between these defects. To prevent objects on the surface of the device to be inspected from appearing like shadows in the captured image and affecting the inspection results, a fully illuminated image can be captured when all second light sources are fully illuminated. When each second light source is illuminated in sequence, the photometric camera can capture a second inspection image. Second-category defect detection is achieved through the second inspection image and the fully illuminated image.

[0045] At each inspection point, the photometric camera captures a full-brightness image and multiple secondary inspection images. The wide coverage of the photometric camera ensures that there is some overlap between the images captured at each inspection point, ensuring that the entire device under inspection is captured, facilitating a complete inspection of the device's surface.

[0046] On the basis of the above implementation, optionally, Figure 1a As shown, the visual inspection instrument also includes: a depth camera 150; the depth camera 150 is used to take pictures of the equipment to be inspected when the visual inspection instrument is stationary, and obtain at least one third inspection image, so as to perform a second category defect inspection on the equipment to be inspected based on each second inspection image, the full-bright image and each third inspection image.

[0047] A photometric camera can only perceive the intensity of reflected light but lacks depth information, making it impossible to calculate the incident direction of light from a point source. The visual inspection instrument provided by the embodiments of the present invention can be equipped with a depth camera to compensate for the shortcomings of a photometric camera. Based on the third detection image obtained by the depth camera and the second detection image obtained by the photometric camera, depth information can be added to the second detection image, better determining the actual length of the shadow in the image and more accurately detecting defects in the second category.

[0048] Based on the above implementation, the visual detection instrument may optionally further include: an anti-collision sensor; an anti-collision sensor for sensing the target distance between the visual detection instrument and surrounding objects, and issuing a distance warning when the target distance is less than a preset distance threshold to control the visual detection instrument to stop moving.

[0049] The cameras in this visual inspection instrument require calibration before use. Any changes in the relative positions of the cameras or the light source will affect the final inspection results. Therefore, an anti-collision sensor can be installed at the end of the visual inspection instrument. This sensor detects the distance between the visual inspection instrument and surrounding objects. If the distance is too close, it will issue a distance warning, stopping the visual inspection instrument to prevent a collision.

[0050] The images captured by the aforementioned cameras and the distance information obtained by the collision avoidance sensor can be transmitted to the processor via communication for defect detection. Specifically, the processor can be installed in the visual inspection instrument, or the processor can also be installed in the host computer of the visual inspection instrument. This embodiment of the present invention is not specifically limited to this.

[0051] Figure 1b FIG. 1 is a schematic diagram of the layout of a visual inspection instrument provided according to the first embodiment of the present invention. Figure 1b As shown, the multi-camera 110 can be arranged in an equilateral triangle, with the center of the multi-camera 110 aligned with the center of the depth camera 150. The photometric camera 120 can be positioned between the two triangular cameras. The first light source 130 can be positioned around the multi-camera 110. The second light source 140 can be positioned at a predetermined height around the photometric camera 120. The anti-collision sensor 160 can be mounted at the end of the visual inspection instrument.

[0052] The technical solution of an embodiment of the present invention is to arrange a visual inspection instrument including a multi-eye camera, a photometric camera, at least one first light source, and at least one second light source, wherein: the multi-eye camera is used to take pictures of the device to be inspected at a preset frequency when the visual inspection instrument is moving and all the first light sources are turned on, so as to obtain multiple first inspection images, so as to perform first category defect detection on the device to be inspected based on each first inspection image; the photometric camera is used to take pictures of the device to be inspected when the visual inspection instrument is stationary and all the second light sources are turned on and triggered, so as to obtain a full-bright image; the photometric camera is also used to take pictures of the device to be inspected once when the visual inspection instrument is stationary and all the second light sources are turned on and triggered in sequence, so as to obtain multiple second inspection images, so as to perform second category defect detection on the device to be inspected based on each second detection image and the full-bright image, thereby solving the problem of device surface defect detection. Multiple types of pictures can be taken on the device surface to realize multi-type defect detection on the device surface.

[0053] Example 2

[0054] Figure 2This is a flow chart of a visual inspection method according to a second embodiment of the present invention. This embodiment is applicable to the use of visual inspection equipment to detect defects on aircraft surfaces. The method can be performed by a visual inspection device, which can be implemented in hardware and / or software and configured in an electronic device. The visual inspection method provided in this embodiment of the present invention can be applied to the visual inspection equipment provided in any embodiment of the present invention.

[0055] Step 210: Using a multi-lens camera, when the visual inspection instrument is moving and one or more first light sources are all turned on, take pictures of the device to be inspected at a preset frequency to obtain multiple first inspection images.

[0056] Step 220: When it is detected based on each first detection image that there is a shift in the position arrangement of multiple target detection objects in the image, it is determined that the device to be inspected has a first category defect.

[0057] For example, the device to be inspected may be an aircraft. The target inspection object may be a rivet. The positional arrangement of rivets on the aircraft surface is typically regular, for example, evenly distributed in rows and columns. The multi-lens camera has a positioning function, and the first inspection image obtained may include image information and position information of the rivets. Image processing can be performed on the first inspection image to identify the position of the rivets and determine whether there is an offset in the rivet position arrangement. If there is an offset in the rivet position, it can be determined that the aircraft has a first category defect, namely, a rivet position deviation defect.

[0058] Step 230: When the visual inspection instrument is stationary and the plurality of second light sources are all turned on and triggered, a photo of the device to be inspected is taken by a photometric camera to obtain a fully bright image.

[0059] Step 240: When the visual inspection instrument is stationary and each second light source is sequentially turned on and triggered, the photometric camera takes a picture of each device to be inspected, thereby obtaining a plurality of second inspection images.

[0060] Step 250: When a target shadow is detected in the image based on the full-bright image and each second detection image, it is determined that the device to be inspected has a second category defect.

[0061] Exemplarily, the surface quality defects of an aircraft mainly include: bumps, paint peeling, scratches, corrosion, dents, etc. These types of defects have a common feature, which is the existence of height differences. When the aircraft is photographed by a photometric camera, if there are bumps, paint peeling, scratches, corrosion, or dents on the surface of the aircraft, there will be shadows in the second detection image. Since there may be some objects on the surface of the aircraft that cast shadows when photographed under light (such as black objects), the full bright image and the second detection image can be comprehensively processed to eliminate the impact of non-defective objects, determine the target shadow, and improve the accuracy of the second category defect detection. Among them, the target shadow can be understood as the shadow formed in the image when there are bumps, paint peeling, scratches, corrosion, or dents on the surface of the aircraft.

[0062] In an optional implementation of an embodiment of the present invention, when a target shadow is detected in the image based on the full-bright image and each second detection image, it is determined that a second category defect exists in the device to be inspected, including: obtaining a difference map corresponding to each second detection image based on the full-bright image and each second detection image; determining the position information of multiple target detection objects in each difference map based on the difference in shadow degree; performing shadow enhancement on each difference map, and after eliminating the target detection objects in the map based on the position information of multiple target detection objects, when a target shadow is detected in the map, it is determined that a second category defect exists in the device to be inspected.

[0063] The difference image can be understood as an image formed based on the difference between the full-brightness image and the second detection image. For example, the difference image can be an image formed by removing objects in the second detection image whose positions match the shadow positions in the full-brightness image. The rivet is raised on the surface of the aircraft, so the shadow formed by the rivet in the second detection image is very obvious. The shadow corresponding to the rivet can be determined based on the degree of shadow visibility in the difference image. That is, the obvious shadow in the difference image can be determined as the rivet. The shadow corresponding to the rivet can be eliminated in all difference images.

[0064] All difference images can be reconstructed using photometric stereo, enhancing shadows within the images. If the target shadow remains after enhancing the shadows and removing the shadows corresponding to the rivets, it can be determined that the device under inspection has a Category II defect. Specifically, the target shadow in the difference image can be detected by determining the degree of shadow visibility, identifying prominent shadows as target shadows and ignoring less prominent shadows. This allows for minor scratches on the aircraft surface to be detected, preventing minor issues that pose no threat from being detected as defects.

[0065] Among them, the photometric stereo method can be to reconstruct the normal and calculate the gradient of each difference map, and combine multiple difference maps into a gradient map.

[0066] Furthermore, based on the above embodiment, optionally, the method also includes: taking a picture of the device to be inspected by a depth camera when the visual inspection instrument is stationary, to obtain at least one third inspection image; determining the actual length of the target shadow based on each second inspection image and each third inspection image; in the target shadow, eliminating shadows whose actual length is less than a preset length threshold, and updating the target shadow.

[0067] The target shadow determined by the second inspection image and the full-brightness image lacks length. This means that the photometric camera lacks depth information and may determine that the target shadow is too shallow to be a defect. To further improve the accuracy of defect determination, a third inspection image can be acquired using a depth camera to add depth information to the second inspection image.

[0068] Specifically, the depth camera can collect the distance (depth) information between the device to be detected and the depth camera. By performing coordinate system conversion on the second detection image and the third detection image, the depth value of the second detection image can be determined. Furthermore, based on the depth value of the second detection image and the length of the target shadow in the second detection image, the actual length of the target shadow can be determined. Target shadows with smaller actual lengths can be eliminated, that is, very shallow scratches on the fuselage can be ignored, thereby improving the reliability of defect determination.

[0069] In an optional implementation of an embodiment of the present invention, the actual length of the target shadow is determined according to each second detection image and each third detection image, including: determining the position information and the first pixel coordinates corresponding to the target shadow in each second detection image; determining the second pixel coordinates at the position matching the position information corresponding to the target shadow in the third detection image; determining the target depth value corresponding to the target shadow according to the first pixel coordinates, the second pixel coordinates, the depth value of the third detection image, the camera parameters of the photometric camera, and the camera parameters of the depth camera; and determining the actual length of the target shadow according to the target depth value and the length of the target shadow in the image.

[0070] The photometric camera and the depth camera can be installed very close together, so the effective areas of the second and third detection images are theoretically the same. Alternatively, the second and third detection images can be calibrated based on the rivet positions in the second and third detection images. When the position information of the target shadow is determined in the second detection image, the corresponding matching position can be determined in the third detection image.

[0071] Determine the pixel coordinates corresponding to the target shadow in the second detection image and the third detection image respectively, that is, obtain the first pixel coordinates and the second pixel coordinates. The depth camera may have a depth value of the third detection image (which can be understood as the depth between the depth camera and the device to be detected). The photometric camera and the depth camera may have respective camera parameters. The camera parameters may include but are not limited to the intrinsic parameter matrix, pose matrix, etc. of the camera. The camera parameters may be determined by camera calibration during the preparatory work.

[0072] Based on the first pixel coordinate, the second pixel coordinate, the depth value of the third detection image, the camera parameters of the photometric camera, and the camera parameters of the depth camera, a matrix calculation can be performed to determine the target depth value corresponding to the first pixel coordinate. Furthermore, based on the length of the target shadow in the image and the target depth value, a matrix calculation can be performed to determine the actual length of the target shadow, accurately determining the actual size of the target shadow and improving the reliability of defect determination. The length of the target shadow in the image can be determined based on the first pixel coordinate.

[0073] Based on the above embodiment, optionally, the method further includes: stitching the first detection images according to the position arrangement of multiple target detection objects in each first detection image to obtain a first stitching image; determining the shooting position of each second detection image according to the first stitching image; stitching the second detection images according to the shooting position and the position arrangement of multiple target detection objects in the second detection image to obtain a second stitching image; when it is determined that the equipment to be inspected has a second category defect, performing position calibration in the second stitching image according to the position of the target shadow in the second detection image.

[0074] Among them, the first detection image is captured by a multi-eye camera in a scanning form during the movement of the visual inspection instrument. Therefore, there will be overlapping areas in each first detection image. By comparing the position arrangement of the rivets in the first detection image, the splicing relationship of the multiple first detection images obtained before and after can be determined. The multiple first detection images can be spliced to obtain a first spliced image. The surface of the device to be inspected can be fully presented in the first spliced image. If a first category defect is detected in the device to be inspected, the defect can be marked in the first spliced image, which makes it easier for quality inspectors to promptly discover equipment defects, perform repairs, and ensure equipment safety.

[0075] The second detection image is captured when the visual inspection instrument is stationary. For example, during the visual inspection instrument's inspection process, a detection path is pre-set from detection point A to detection point B to detection point C. The first detection image can be captured while the visual inspection instrument is moving from detection point A to detection point B, and from detection point B to detection point C; the second detection image can be captured while the visual inspection instrument is at detection point A, detection point B, and detection point C.

[0076] The multi-eye camera captures images at the transition between two detection points, and the detection points before and after can be known. Therefore, the shooting position of the second detection image can be determined based on the first mosaic. For example, the first first detection image in the first mosaic corresponds to detection point A, and the tenth first detection image corresponds to detection point B. The positional relationship between detection points A and B can be known based on the first mosaic. For example, with detection point A as a reference, the distance offset to the left is detection point B. The second detection image taken at detection point A can be placed at the corresponding position of the first first detection image in the first mosaic; the second detection image taken at detection point B can be placed at the corresponding position of the tenth first detection image in the first mosaic. By analogy, a rough stitching of the second detection image can be achieved. Furthermore, the stitching of the second detection images can be adjusted based on the overlapping rivet position information in each second detection image to achieve fine stitching and generate a second mosaic.

[0077] When a second category defect is detected in the device to be inspected, the defect position can be marked in the second splicing image, so that quality inspectors can find the device defect in time, perform repairs, and ensure the safety of the equipment.

[0078] Furthermore, the third detection image can be spliced with reference to the method for generating the second spliced image, and a third spliced image can be generated accordingly. Defects can also be calibrated in the third spliced image.

[0079] Based on the above implementation, optionally, the method further includes: sensing the target distance between the visual detection instrument and surrounding objects through an anti-collision sensor, and issuing a distance warning when the target distance is less than a preset distance threshold; and controlling the visual detection instrument to stop moving according to the distance warning.

[0080] The technical solution of this embodiment is to use a multi-eye camera to take pictures of the device to be inspected at a preset frequency when the visual inspection instrument is moving and one or more first light sources are all turned on to obtain multiple first inspection images; when it is detected based on each first inspection image that there is an offset in the position arrangement of multiple target inspection objects in the image, it is determined that the device to be inspected has a first category defect; when the visual inspection instrument is stationary and multiple second light sources are all turned on and triggered, the photometric camera is used to take pictures of the device to be inspected to obtain a full-brightness image; when the visual inspection instrument is stationary and each second light source is turned on and triggered in sequence, the photometric camera is used to take pictures of the device to be inspected once to obtain multiple second inspection images; when it is detected based on the full-brightness image and each second inspection image that there is a target shadow in the image, it is determined that the device to be inspected has a second category defect, which solves the problem of device surface defect detection, can take multiple types of pictures of the device surface, realize effective detection of multiple types of defects on the device surface, and ensure a wide range of use of the visual inspection instrument.

[0081] Example 3

[0082] Figure 3 Schematic diagram of the structure of a visual inspection device provided according to the third embodiment of the present invention. The device can be applied to the visual inspection instrument provided by any embodiment of the present invention. Figure 3 As shown, the device includes: a first detection image acquisition module 310, a first category defect determination module 320, a full bright image acquisition module 330, a second detection image acquisition module 340 and a second category defect acquisition module 350.

[0083] The first detection image acquisition module 310 is configured to use a multi-lens camera to take pictures of the device to be inspected at a preset frequency while the visual inspection instrument is moving and one or more first light sources are all turned on, thereby obtaining a plurality of first detection images;

[0084] A first category defect determination module 320 is configured to determine, based on each first inspection image, that a first category defect exists in the device to be inspected when a positional arrangement of multiple inspection target objects in the image is offset;

[0085] The full-brightness image acquisition module 330 is used to take a photo of the device to be inspected by using a photometric camera when the visual inspection instrument is stationary and the multiple second light sources are all turned on and triggered, thereby obtaining a full-brightness image;

[0086] The second detection image acquisition module 340 is used to use a photometric camera to take a picture of each device to be detected when the visual detection instrument is stationary and each second light source is triggered in sequence, thereby obtaining multiple second detection images;

[0087] The second category defect acquisition module 350 is configured to determine that the device to be inspected has a second category defect when a target shadow is detected in the image based on the full-bright image and each second inspection image.

[0088] Optionally, the device further includes:

[0089] A third detection image acquisition module is used to take a picture of the device to be detected by using a depth camera when the visual detection instrument is stationary, to obtain at least one third detection image;

[0090] an actual length determination module, configured to determine an actual length of a target shadow based on each second detection image and each third detection image;

[0091] The target shadow update module is used to eliminate shadows whose actual length is less than a preset length threshold in the target shadow and update the target shadow.

[0092] Optionally, the second category defect acquisition module 350 includes:

[0093] a difference map determining unit, configured to obtain, based on the full-brightness image and each second detection image, a difference map corresponding to each second detection image;

[0094] a position information determining unit, configured to determine position information of a plurality of target detection objects in each difference map based on differences in shadow levels;

[0095] The second category defect detection unit is used to enhance the shadows of each difference image, and after eliminating the target detection objects in the image based on the position information of multiple target detection objects, determine that the second category defect exists in the device to be inspected when a target shadow is detected in the image.

[0096] Optional, actual length determination module, including:

[0097] a first pixel coordinate determining unit, configured to determine position information and first pixel coordinates corresponding to a target shadow in each second detection image;

[0098] a second pixel coordinate determining unit, configured to determine a second pixel coordinate at a position matching the position information corresponding to the target shadow in the third detection image;

[0099] a target depth value determining unit, configured to determine a target depth value corresponding to the target shadow according to the first pixel coordinate, the second pixel coordinate, the depth value of the third detection image, camera parameters of the photometric camera, and camera parameters of the depth camera;

[0100] The actual length determining unit is used to determine the actual length of the target shadow according to the target depth value and the length of the target shadow in the image.

[0101] Optionally, the device further includes:

[0102] a first mosaic image acquisition module, configured to mosaic the first detection images according to the position arrangement of the plurality of target detection objects in the first detection images to obtain a first mosaic image;

[0103] a shooting position determination module, configured to determine the shooting position of each second detection image according to the first spliced image;

[0104] a second mosaic image determining module, configured to mosaic the second detection images according to the shooting position and the position arrangement of the plurality of target detection objects in the second detection image to obtain a second mosaic image;

[0105] The position calibration module is used to perform position calibration in the second spliced image according to the position of the target shadow in the second detection image when it is determined that the second category defect exists in the device to be inspected.

[0106] Optionally, the device further includes:

[0107] A distance warning module is used to sense the target distance between the visual inspection instrument and surrounding objects through an anti-collision sensor, and to issue a distance warning when the target distance is less than a preset distance threshold;

[0108] The control module is used to control the visual inspection instrument to stop moving according to the distance warning.

[0109] The visual inspection device provided by the embodiment of the present invention can execute the visual inspection method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0110] Example 4

[0111] Figure 4 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0112] like Figure 4As shown, the electronic device 10 includes at least one processor 11 and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by the at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. Various programs and data required for the operation of the electronic device 10 can also be stored in the RAM 13. The processor 11, ROM 12 and RAM 13 are connected to each other via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14. The electronic device may also include a multi-lens camera 110, a photometric camera 120, at least one first light source 130, and at least one second light source 140.

[0113] Multiple components in the electronic device 10 are connected to the I / O interface 15, including an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0114] The processor 11 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the visual inspection method.

[0115] In some embodiments, the visual inspection method can be implemented as a computer program that is tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the visual inspection method described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to perform the visual inspection method in any other suitable manner (e.g., by means of firmware).

[0116] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0117] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the computer program is executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer program may be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0118] In the context of the present invention, computer-readable storage media can be tangible media that can contain or store a computer program for use with an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Computer-readable storage media can include but are not limited to electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, computer-readable storage media can be machine-readable signal media. More specific examples of machine-readable storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0119] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0120] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0121] A computing system may include clients and servers. The clients and servers are typically remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host. This server is a hosting product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosting and VPS services.

[0122] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present invention can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved. This is not limited herein.

[0123] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A visual inspection instrument, characterized in that: include: A multi-camera, a photometric camera, at least one first light source, and at least one second light source; wherein: The multi-lens camera is configured to photograph the device to be inspected at a preset frequency when the visual inspection instrument is moving and all the first light sources are turned on, thereby obtaining a plurality of first inspection images, and performing first category defect inspection on the device to be inspected based on the first inspection images; The photometric camera is used to take a picture of the device to be inspected to obtain a fully illuminated image when the visual inspection instrument is stationary and all the second light sources are turned on and triggered; The photometric camera is also used to take a picture of the device to be inspected once when the visual inspection instrument is stationary and each of the second light sources is turned on and triggered in sequence, so as to obtain multiple second inspection images, so as to perform second category defect detection on the device to be inspected based on each of the second inspection images and the full-brightness image.

2. The visual inspection instrument according to claim 1, characterized in that: Also includes: Depth camera; The depth camera is used to take pictures of the device to be inspected when the visual inspection instrument is stationary to obtain at least one third inspection image, so as to perform a second category defect inspection on the device to be inspected based on each of the second inspection images, the full-brightness image and each of the third inspection images.

3. The visual inspection instrument according to claim 1, characterized in that: Also includes: Anti-collision sensor; The anti-collision sensor is used to sense the target distance between the visual detection instrument and surrounding objects, and to issue a distance warning when the target distance is less than a preset distance threshold, so as to control the visual detection instrument to stop moving.

4. A visual inspection method, characterized in that: The visual inspection instrument according to any one of claims 1 to 3 comprises: Using a multi-lens camera, while the visual inspection instrument is moving and one or more first light sources are all turned on, photographing the device to be inspected at a preset frequency to obtain a plurality of first inspection images; When detecting, based on each of the first detection images, that a position arrangement of a plurality of target detection objects in the image is offset, determining that a first category defect exists in the device to be detected; When the visual inspection instrument is stationary and the plurality of second light sources are all turned on and triggered, a photometric camera is used to take a picture of the device to be inspected to obtain a fully illuminated image; When the visual inspection instrument is stationary and each of the second light sources is sequentially turned on and triggered, a photometric camera is used to take a picture of the device to be inspected to obtain a plurality of second inspection images; When a target shadow is detected in the image according to the full-bright image and each of the second detection images, it is determined that the device to be inspected has a second category defect.

5. The method according to claim 4, characterized in that Also includes: When the visual inspection instrument is stationary, taking a picture of the device to be inspected by using a depth camera to obtain at least one third inspection image; determining an actual length of the target shadow according to each of the second detection images and each of the third detection images; Among the target shadows, the shadows whose actual lengths are less than a preset length threshold are eliminated, and the target shadows are updated.

6. The method according to claim 4, characterized in that When a target shadow is detected in the image according to the full-brightness image and each of the second detection images, determining that the device to be inspected has a second category defect includes: Obtaining, based on the full-brightness image and each of the second detection images, a difference image corresponding to each of the second detection images; In each of the difference images, determining position information of multiple target detection objects based on differences in shadow levels; After shadow enhancement is performed on each of the difference images and target detection objects in the image are eliminated according to position information of multiple target detection objects, when target shadows are detected in the image, it is determined that the device to be inspected has a second category defect.

7. The method according to claim 5, characterized in that Determining an actual length of the target shadow according to each of the second detection images and each of the third detection images includes: Determining position information and first pixel coordinates corresponding to the target shadow in each of the second detection images; Determine a second pixel coordinate at a position in the third detection image that matches the position information corresponding to the target shadow; Determining a target depth value corresponding to the target shadow according to the first pixel coordinates, the second pixel coordinates, the depth value of the third detection image, camera parameters of the photometric camera, and camera parameters of the depth camera; The actual length of the target shadow is determined according to the target depth value and the length of the target shadow in the image.

8. The method according to claim 4, characterized in that Also includes: splicing the first detection images according to the position arrangement of the plurality of target detection objects in the first detection images to obtain a first spliced image; determining, according to the first mosaic image, a shooting position of each of the second detection images; splicing the second detection images according to the shooting position and the position arrangement of the plurality of target detection objects in the second detection image to obtain a second spliced image; When it is determined that the device to be inspected has a second category defect, position calibration is performed in the second spliced image according to the position of the target shadow in the second inspection image.

9. The method according to claim 4, characterized in that Also includes: The anti-collision sensor senses the target distance between the visual inspection instrument and surrounding objects, and issues a distance warning when the target distance is less than a preset distance threshold; According to the distance warning, the visual inspection instrument is controlled to stop moving.

10. A visual inspection device, characterized in that: The visual inspection instrument according to any one of claims 1 to 3 comprises: A first detection image acquisition module is configured to use a multi-lens camera to take pictures of the device to be detected at a preset frequency while the visual detection instrument is moving and one or more first light sources are all turned on, thereby obtaining a plurality of first detection images; A first category defect determination module is configured to determine that a first category defect exists in the device to be inspected when, based on each of the first inspection images, a positional arrangement of multiple target inspection objects in the image is offset; a full-brightness image acquisition module, configured to take a photo of the device to be inspected by using a photometric camera when the visual inspection instrument is stationary and the plurality of second light sources are all turned on and triggered, thereby obtaining a full-brightness image; A second detection image acquisition module is configured to use a photometric camera to take a picture of the device to be detected once when the visual detection instrument is stationary and each of the second light sources is triggered in sequence, thereby obtaining a plurality of second detection images; The second category defect acquisition module is used to determine that the device to be inspected has a second category defect when a target shadow is detected in the image based on the full-bright image and each of the second inspection images.

11. An electronic device, characterized in that: The electronic device comprises: A multi-camera, a photometric camera, at least one first light source, and at least one second light source; at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor. The computer program is executed by the at least one processor to enable the at least one processor to perform the visual inspection method according to any one of claims 4 to 9.

12. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the visual inspection method according to any one of claims 4 to 9 when executed.

Images