Measurement optimization method for multiple scenes in machine vision gray scale instrument application

Abstract

The invention relates to the technical field of data processing, and discloses a measurement optimization method for multiple scenes in machine vision gray scale instrument application, which comprises the following steps: erecting a 3D camera; executing a calibration process, and obtaining a reference 2D color image, a reference light reflectivity intensity image, a reference 3D height image and background plane data; each frame of camera snapshot data in the operation process is compared with reference data, and reference package shapes under the 2D color image, the luminous reflectance intensity image and the 3D height image are obtained respectively; overlapping the obtained reference parcel shapes to obtain a complete parcel shape, and cutting out the complete parcel shape under the 3D height image to obtain 3D height data of the complete parcel; the 3D height data of the complete package and the background plane data obtained by the calibration process are utilized to calculate the measurement data of the complete package and judge whether the hyperedge difference exists or not. According to the application, the problems encountered by using a traditional 2D camera as related applications of a gray scale instrument in industries such as express logistics can be solved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method for optimizing the measurement of various scenes in the application of a machine vision grayscale instrument. Background Art

[0002] In the logistics industry, there are several typical machine vision applications, such as using industrial cameras to scan codes, using laser radar or 3D vision technology to measure the size of packages, etc. Grayscale meters are widely used in the logistics industry, usually consisting of one or more 2D industrial cameras and supporting light sources. Grayscale meters are mainly used in the logistics industry to determine whether there are packages on conveyor belts or cross-belt sorters, whether the packages are centered or beyond the conveyor belt boundary, etc. However, the traditional use of 2D industrial cameras to implement grayscale meter functions will face the following problems: when the color of the package is close to the color of the conveyor belt, it is difficult for the camera to use color or grayscale images to confirm the existence and boundary position of the package; the camera must be installed vertically on the logistics line to shoot downward. When subject to height restrictions, a wide-angle lens must be used to ensure the width of the field of view, but it is still a conical field of view. Therefore, when shooting higher packages, due to the characteristics of the conical field of view that is larger near and smaller far away, it is easy to misjudge that the package is beyond the edge. Summary of the invention

[0003] The purpose of this application is to provide a measurement optimization method for multiple scenes in the application of a machine vision grayscale meter to solve the technical problems raised in the above background.

[0004] To achieve the above purpose, the present application discloses the following technical solution: a method for optimizing the measurement of multiple scenes in the application of a machine vision grayscale instrument, the method comprising the following steps:

[0005] Set up a 3D camera;

[0006] Execute the calibration process, obtain the reference 2D color image, reference light reflectance intensity image, reference 3D height image and background plane data, and save them;

[0007] Compare each frame of camera snapshot data in the running process with the reference data of the calibration process to obtain the reference package shape in 2D color image, light reflectance intensity image and 3D height image respectively;

[0008] Overlapping the reference package shapes under the 2D color image, the light reflectance intensity image and the 3D height image to obtain the complete package shape, and cutting out the complete package shape under the 3D height image to obtain the 3D height data of the complete package;

[0009] Using the 3D height data of the complete package and the background plane data obtained by the calibration process, the measurement data of the complete package is calculated and whether there is any super-edge anomaly is determined.

[0010] Preferably, the data stream processing and operation mode of the 3D camera includes:

[0011] Obtain a light reflectance intensity image, a 2D color image, and a 3D height image from a 3D camera;

[0012] Use 2D color images to identify objects with different colors from the background, use light reflectance intensity images to identify objects with different materials from the background, and use 3D height images to identify objects with different heights from the background;

[0013] Identify background planes using 3D height images.

[0014] Preferably, the calibration process specifically includes:

[0015] The 3D camera is mounted and fixed so that its position and angle will not change in subsequent processes;

[0016] Make sure that there is no object being measured within the current camera field of view;

[0017] Obtain a frame of snapshot data under the current camera field of view, the snapshot data including a 2D color image, a light reflectance intensity image and a 3D height image;

[0018] The obtained 2D color image, light reflectance intensity image and 3D height image are stored in the 3D camera, and the 2D color image, light reflectance intensity image and 3D height image are defined as reference background images in the operation process;

[0019] intercepting height data of a range where a reference plane is located in the 3D height image in the reference background image, fitting the plane using a plane search algorithm to obtain a fitting plane, and calculating the deflection coordinates of the fitting plane relative to the 3D camera;

[0020] The deflection coordinates of the fitting plane relative to the 3D camera are stored in the 3D camera.

[0021] Preferably, the obtaining of the complete package 3D height data specifically includes:

[0022] Using a visual algorithm, the areas where the obtained 2D color image, light reflectance intensity image, and 3D height image are different from the reference background image stored in the camera are identified as parcel areas;

[0023] The parcel regions found in the 2D color image, the light reflectance intensity image and the 3D height image are processed respectively to generate a complete parcel region;

[0024] In the 3D height image, the height data in the complete package area is intercepted, the calculation of the area where the package is located and the separation of the height data of the package are completed, and the 3D height data of the complete package is obtained.

[0025] Preferably, the calculation of the complete package measurement data and the determination of whether there is a super-edge anomaly specifically include:

[0026] The height data of the separated parcel is offset based on the deflection coordinates of the fitting plane obtained in the calibration process relative to the camera to obtain the height data of the parcel based on the reference plane coordinate system;

[0027] Using 3D point cloud algorithm and height data based on reference plane coordinate system, the length, width, height, position, rotation angle and volume of the package are measured in the reference plane coordinate system.

[0028] Using the 3D point cloud algorithm and the height data based on the reference plane coordinate system, all the height data of the package in the space is projected onto the reference plane to quickly determine whether the package in the space exceeds the boundary range set on the reference plane.

[0029] Beneficial effects: The measurement optimization method for multiple scenes in the machine vision grayscale application of this application, compared with the increased data types of traditional 2D cameras, solves the problems encountered in using traditional 2D cameras for grayscale related applications in express logistics and other industries. Specifically, when the 2D camera grayscale is installed at an angle, it cannot accurately realize super-edge judgment, but this problem can be solved by using a 3D camera combined with a projection algorithm in this application; when a 2D camera grayscale is faced with an object with a similar color to the background, it cannot accurately separate the object, but the light reflection intensity map contained in the 3D camera used in this application can ignore the color of the object and use the different reflectivity of light on the surface of different materials to realize accurate identification of the object. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0031] Figure 1 A system architecture diagram of a measurement optimization method for multiple scenes in a machine vision grayscale instrument application provided in an embodiment of the present application;

[0032] Figure 2 A flowchart of the operation of the measurement optimization method for multiple scenes in the application of the machine vision grayscale instrument provided in the embodiment of the present application;

[0033] Figure 3 A schematic diagram of a visual algorithm used in an embodiment of the present application;

[0034] Figure 4 A schematic diagram of a calculation algorithm for the area where a package is located used in an embodiment of the present application;

[0035] Figure 5 A schematic diagram of the 3D point cloud algorithm used in an embodiment of the present application. DETAILED DESCRIPTION

[0036] The technical solutions in the embodiments of the present application are described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present application.

[0037] In this article, the term "comprising" is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of more restrictions, the elements defined by the sentence "comprising..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0038] See also Figure 1-2

[0039] This embodiment discloses a method for optimizing the measurement of multiple scenes in a machine vision grayscale instrument application. Specifically, the method includes a hardware setup process and an operation process. The hardware setup process and the operation process are introduced below.

[0040] Hardware installation process:

[0041] Set up a 3D camera. Among them, 3D industrial cameras can better make up for the several major pain points caused by using 2D industrial cameras in grayscale applications:

[0042] 1. 3D industrial cameras can retain 2D lenses and have all the excellent features of 2D industrial cameras.

[0043] 2. Compared with 2D industrial cameras, 3D industrial cameras have additional light reflectance intensity map and height map data, which increases the dimension of the data and has greater scalability.

[0044] 3. As 3D industrial cameras have light reflectivity intensity map data, the light reflectivity intensity is determined by the surface material of the object being photographed and is not affected by color. This can better solve the problem of using 2D cameras as grayscale meters, where the color of the package is similar to that of the conveyor belt surface, leading to incorrect judgment.

[0045] 4. As 3D industrial cameras have height map data, they can identify the upper surface of the logistics line, fit the reference plane, and project the identified package height data onto the reference plane. This method can avoid problems such as misjudgment of package over-edge due to conical field of view. At the same time, due to the use of projection algorithms, the camera does not need to be set up directly above the conveyor belt to shoot downwards, and a certain angle of inclination can be allowed, so that the deployment of the camera is no longer restricted by height and is more flexible.

[0046] In the hardware setup process, the data stream processing and operation mode of the 3D camera include:

[0047] Obtain a light reflectance intensity image, a 2D color image, and a 3D height image from a 3D camera;

[0048] Use 2D color images to identify objects with different colors from the background, use light reflectance intensity images to identify objects with different materials from the background, and use 3D height images to identify objects with different heights from the background;

[0049] Identify background planes using 3D height images.

[0050] Through the above 3D camera settings, the following functions can be achieved:

[0051] Combine the object recognition results of different images to determine whether there is a package in the camera's field of view;

[0052] Project the 3D image of the package onto the background plane, calculate the length, width, height and volume of the package, and determine whether it exceeds the set boundaries.

[0053] Operation process:

[0054] After setting up the 3D camera, execute the calibration process to obtain and save the reference 2D color image, reference light reflectance intensity image, reference 3D height image and background plane data;

[0055] Compare each frame of camera snapshot data in the running process with the reference data of the calibration process (i.e., reference 2D color image, reference light reflectance intensity image, reference 3D height image and background plane data) to obtain the reference package shape under the 2D color image, light reflectance intensity image and 3D height image respectively;

[0056] Overlapping the reference package shapes under the 2D color image, the light reflectance intensity image and the 3D height image to obtain the complete package shape, and cutting out the complete package shape under the 3D height image to obtain the 3D height data of the complete package;

[0057] Using the 3D height data of the complete package and the background plane data obtained by the calibration process, the measurement data of the complete package is calculated and whether there is any super-edge anomaly is determined.

[0058] In the operation process, the calibration process specifically includes:

[0059] The 3D camera is set up and fixed so that its position and angle will not change in the subsequent process; that is, it is ensured that the camera has been set up and its position and angle will not change in the subsequent operation process;

[0060] Make sure that there is no object to be measured in the current camera field of view; that is, make sure that the current camera field of view is all background and the object to be measured does not exist in the camera field of view;

[0061] Obtain a frame of snapshot data under the current camera field of view, the snapshot data including a 2D color image, a light reflectance intensity image and a 3D height image;

[0062] The obtained 2D color image, light reflectance intensity image and 3D height image are stored in the 3D camera, and the 2D color image, light reflectance intensity image and 3D height image are defined as reference background images in the operation process;

[0063] intercepting height data of a range where a reference plane is located in the 3D height image in the reference background image, fitting the plane using a plane search algorithm to obtain a fitting plane, and calculating the deflection coordinates of the fitting plane relative to the 3D camera;

[0064] The deflection coordinates of the fitting plane relative to the 3D camera are stored in the 3D camera.

[0065] In the operation process, the 3D height data of the complete package is obtained, specifically including:

[0066] Using a visual algorithm, the areas where the obtained 2D color image, light reflectance intensity image, and 3D height image are different from the reference background image stored in the camera are identified as parcel areas; the visual algorithm uses the following Figure 3 The SSIM (Structural Similarity Index of Two Images) algorithm shown;

[0067] The parcel regions found in the 2D color image, the light reflectance intensity image and the 3D height image are processed respectively to generate a complete parcel region;

[0068] In the 3D height image, the height data in the complete package area is intercepted, the calculation of the area where the package is located and the separation of the height data of the package are completed, and the 3D height data of the complete package is obtained. Among them, the calculation of the area where the package is located is as follows Figure 4 The Graham scanning method in the prior art is shown.

[0069] In the operation process, the calculation of the measurement data of the complete package and the determination of whether there is an over-edge anomaly specifically include:

[0070] The height data of the separated parcel is offset based on the deflection coordinates of the fitting plane obtained in the calibration process relative to the camera to obtain the height data of the parcel based on the reference plane coordinate system;

[0071] Using the 3D point cloud algorithm, combined with the height data based on the reference plane coordinate system, the length, width, height, position, rotation angle and volume of the package in the reference plane coordinate system are measured; the 3D point cloud algorithm uses the following Figure 5 The OBB (minimum bounding box) algorithm in the prior art shown;

[0072] Using the 3D point cloud algorithm and the height data based on the reference plane coordinate system, all the height data of the package in the space is projected onto the reference plane to quickly determine whether the package in the space exceeds the boundary range set on the reference plane.

[0073] Based on the above, the measurement optimization method for multiple scenes in the machine vision grayscale application disclosed in this embodiment, compared with the increased data types of traditional 2D cameras, solves the problems encountered in using traditional 2D cameras for grayscale related applications in express logistics and other industries. Specifically, when the 2D camera grayscale is installed at an angle, it cannot accurately realize super-edge judgment, but the 3D camera combined with the projection algorithm in this application can solve this problem; when the 2D camera grayscale is faced with an object with a similar color to the background, it cannot accurately separate the object, but the light reflection intensity map contained in the 3D camera used in this application can ignore the color of the object and use the different reflectivity of light on the surface of different materials to realize accurate identification of the object.

[0074] In the embodiments provided in the present application, it should be understood that the embodiments described herein can be implemented in hardware, software, firmware, middleware, code or any appropriate combination thereof. For hardware implementation, the processor can be implemented in one or more of the following units: application specific integrated circuit (ASIC), digital signal processor (DSP), digital signal processing device (DSPD), programmable logic device (PLD), field programmable gate array (FPGA), processor, controller, microcontroller, microprocessor, other electronic units designed to implement the functions described herein or their combination. For software implementation, part or all of the flow of the embodiment can be completed by instructing the relevant hardware through a computer program. When implemented, the above program can be stored in a computer-readable storage medium or transmitted as one or more instructions or codes on a computer-readable storage medium. Computer-readable storage media include computer storage media and communication media, wherein the communication medium includes any medium that is convenient for transmitting a computer program from one place to another. The storage medium can be any available medium that a computer can access. The computer-readable storage medium can include but is not limited to RAM, ROM, EEPROM, CD-ROM or other optical disk storage, disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of an instruction or data structure and can be accessed by a computer.

[0075] Finally, it should be noted that the above is only a preferred embodiment of the present application and is not intended to limit the present application. Although the present application has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A measurement optimization method for multiple scenes in the application of machine vision grayscale instrument, characterized in that: The method comprises the following steps: Set up a 3D camera; Execute the calibration process, obtain the reference 2D color image, reference light reflectance intensity image, reference 3D height image and background plane data, and save them; Compare each frame of camera snapshot data in the running process with the reference data of the calibration process to obtain the reference package shape in 2D color image, light reflectance intensity image and 3D height image respectively; Overlapping the reference package shapes under the 2D color image, the light reflectance intensity image and the 3D height image to obtain the complete package shape, and cutting out the complete package shape under the 3D height image to obtain the 3D height data of the complete package; Using the 3D height data of the complete package and the background plane data obtained by the calibration process, the measurement data of the complete package is calculated and whether there is any super-edge anomaly is determined.

2. The method for optimizing the measurement of multiple scenes in the application of a machine vision grayscale instrument according to claim 1, characterized in that: The data stream processing and operation mode of the 3D camera includes: Obtain a light reflectance intensity image, a 2D color image, and a 3D height image from a 3D camera; Use 2D color images to identify objects with different colors from the background, use light reflectance intensity images to identify objects with different materials from the background, and use 3D height images to identify objects with different heights from the background; Identify background planes using 3D height images.

3. The measurement optimization method for multiple scenes in the application of the machine vision grayscale instrument according to claim 1 is characterized in that: The calibration process specifically includes: The 3D camera is mounted and fixed so that its position and angle will not change in subsequent processes; Make sure that there is no object being measured within the current camera field of view; Obtain a frame of snapshot data under the current camera field of view, the snapshot data including a 2D color image, a light reflectance intensity image and a 3D height image; The obtained 2D color image, light reflectance intensity image and 3D height image are stored in the 3D camera, and the 2D color image, light reflectance intensity image and 3D height image are defined as reference background images in the operation process; intercepting height data of a range where a reference plane is located in the 3D height image in the reference background image, fitting the plane using a plane search algorithm to obtain a fitting plane, and calculating the deflection coordinates of the fitting plane relative to the 3D camera; The deflection coordinates of the fitting plane relative to the 3D camera are stored in the 3D camera.

4. The method for optimizing the measurement of multiple scenes in the application of a machine vision grayscale instrument according to claim 3, characterized in that: The obtaining of the complete 3D height data of the package specifically includes: Using a visual algorithm, the areas where the obtained 2D color image, light reflectance intensity image, and 3D height image are different from the reference background image stored in the camera are identified as parcel areas; The parcel regions found in the 2D color image, the light reflectance intensity image and the 3D height image are processed respectively to generate a complete parcel region; In the 3D height image, the height data in the complete package area is intercepted, the calculation of the area where the package is located and the separation of the height data of the package are completed, and the 3D height data of the complete package is obtained.

5. The method for optimizing the measurement of multiple scenes in the application of a machine vision grayscale instrument according to claim 4, characterized in that: The calculation of the complete package measurement data and the determination of whether there is an over-edge anomaly specifically include: The height data of the separated parcel is offset based on the deflection coordinates of the fitting plane obtained in the calibration process relative to the camera to obtain the height data of the parcel based on the reference plane coordinate system; Using 3D point cloud algorithm and height data based on reference plane coordinate system, the length, width, height, position, rotation angle and volume of the package are measured in the reference plane coordinate system. Using the 3D point cloud algorithm and the height data based on the reference plane coordinate system, all the height data of the package in the space is projected onto the reference plane to quickly determine whether the package in the space exceeds the boundary range set on the reference plane.

Images