Colorless and transparent object recognition method and system applied to assembly line
By acquiring the RGB and depth images of colorless transparent objects in the pipeline, aligning them and globally optimizing them, the problem of insufficient recognition of colorless transparent objects in the prior art is solved, and a higher recognition accuracy is achieved.
Patent Information
- Application Number
- CN202210146548.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-02-17
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2042-02-17
AI Technical Summary
The identification of colorless transparent objects in the prior art is not accurate enough, especially in a cluttered environment, the boundaries of multiple colorless transparent objects cannot be accurately identified.
By obtaining the RGB image and depth image of the colorless transparent object in the pipeline, aligning it, and globally optimizing the depth image according to the RGB image, outputting the optimized depth image, and finally colorless transparent object based on the optimized depth image Perform identification and positioning.
The recognition accuracy of colorless transparent objects is improved, and the boundaries of colorless transparent glass bottles can be accurately identified in a cluttered environment.
Smart Images

Figure CN114529524B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of image recognition and processing, and in particular to a method and system for identifying colorless and transparent objects applied to an assembly line. Background Art
[0002] Occasionally, colorless transparent glass bottles may fall over or break on the medicine can production line. At this time, the operating staff need to use plastic gloves to handle the failure on site, which requires the operators to work in shifts 24 hours a day. At the same time, since the medicine can production line has high requirements for cleanliness, personnel entering and leaving the operating room need to undergo strict cleaning treatment, which takes up a lot of manpower and material resources. In the prior art, an optical 3D sensor can be used to obtain the depth image of the colorless glass bottles on the production line, and then the colorless glass bottles can be identified, and their position and posture can be located at the same time. The robot maintenance system can replace the employees to realize the identification and processing of the fallen and broken glass bottles on the production line. However, the optical 3D sensor used in the prior art is not accurate enough in identifying colorless transparent objects. For example, in a cluttered environment, it is impossible to accurately identify the boundaries of colorless transparent glass bottles for multiple colorless transparent objects that are mutually occluded and clustered. Summary of the invention
[0003] In order to overcome, at least to a certain extent, the problem of insufficient accuracy in identifying colorless and transparent objects in the related art, the present application provides a method and system for identifying colorless and transparent objects applied to an assembly line.
[0004] The scheme of this application is as follows:
[0005] According to a first aspect of an embodiment of the present application, a colorless and transparent object recognition method applied to an assembly line is provided, comprising:
[0006] Get the RGB image and depth image of the colorless and transparent objects in the pipeline;
[0007] Aligning the RGB image and the depth image;
[0008] Performing global optimization on the depth image according to the RGB image, and outputting an optimized depth image;
[0009] The colorless transparent object is identified and located according to the optimized depth image.
[0010] Preferably, in an achievable manner of the present application, the step of acquiring the RGB image and the depth image of the colorless and transparent object in the pipeline includes:
[0011] Obtain depth images of colorless and transparent objects in the pipeline based on the depth camera;
[0012] The RGB images of colorless and transparent objects in the production line are obtained based on dual telecentric cameras.
[0013] Preferably, in an implementable manner of the present application, the globally optimizing the depth image according to the RGB image includes:
[0014] Inputting the RGB image into a pre-trained deep convolutional network to obtain multiple optimization dimensions and optimization data corresponding to each optimization dimension;
[0015] The depth image is globally optimized according to each of the optimization dimensions and the optimization data corresponding to each of the optimization dimensions.
[0016] Preferably, in an implementable manner of the present application, the optimization dimensions include at least: surface normal, transparent surface mask and occlusion boundary.
[0017] Preferably, in an implementable manner of the present application, there are multiple deep convolutional networks, and each of the deep convolutional networks corresponds to each of the optimization dimensions.
[0018] Preferably, in an implementable manner of the present application, the globally optimizing the depth image according to the RGB image further includes:
[0019] The depth image is corrected and depth-adjusted according to each of the optimization dimensions and the optimization data corresponding to each of the optimization dimensions.
[0020] According to a second aspect of an embodiment of the present application, a colorless and transparent object recognition system applied to an assembly line is provided, comprising:
[0021] A first camera group, a second camera group, a processor, a power supply and a communication module;
[0022] The power supply is respectively connected to the first camera group, the second camera group, the processor and the communication module, and is used to provide working power;
[0023] The processor is respectively connected to the first camera group, the second camera group and the communication module;
[0024] The communication module is also connected to the host computer;
[0025] The first camera group is used to obtain RGB images of colorless and transparent objects in the production line;
[0026] The second camera group is used to obtain depth images of colorless and transparent objects in the pipeline;
[0027] The processor is used to align the RGB image and the depth image; also globally optimize the depth image according to the RGB image and output the optimized depth image; also identify and locate the colorless transparent object according to the optimized depth image, and send the identification and positioning result to the host computer through the communication module.
[0028] Preferably, in an implementable manner of the present application, the first camera group includes: a depth camera;
[0029] The second camera group includes: a dual telecentric camera;
[0030] The communication module comprises: a GPRS (General packet radio service) wireless communication module, and the communication module communicates based on the TCP / IP (Transmission Control Protocol / Internet Protocol) protocol.
[0031] The technical solution provided by the present application may include the following beneficial effects: the colorless transparent object recognition method applied to the pipeline in the present application not only obtains the depth image of the colorless transparent object, but also obtains the RGB image of the colorless transparent object, and aligns the RGB image and the depth image. In the present application, the depth image is globally optimized according to the RGB image, the optimized depth image is output, and the colorless transparent object is identified and located according to the optimized depth image, thereby improving the recognition accuracy of the colorless transparent object.
[0032] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0034] Figure 1 It is a flowchart of a method for identifying a colorless and transparent object applied to an assembly line provided by an embodiment of the present application;
[0035] Figure 2 It is a structural schematic diagram of a colorless and transparent object recognition system applied to an assembly line provided by an embodiment of the present application.
[0036] Figure numerals: first camera group-1; second camera group-2; processor-3; power supply-4; communication module-5. DETAILED DESCRIPTION
[0037] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.
[0038] Embodiment 1
[0039] A colorless and transparent object recognition method applied to assembly lines, referring to Figure 1 ,include:
[0040] S11: Obtain RGB images and depth images of colorless and transparent objects in the pipeline;
[0041] Specifically, the RGB image and depth image of the colorless and transparent objects in the pipeline are obtained, including:
[0042] Obtain depth images of colorless and transparent objects in the pipeline based on the depth camera;
[0043] The RGB images of colorless and transparent objects in the production line are obtained based on dual telecentric cameras.
[0044] The 3D vision system uses ToF optical geometry or physical principles and non-contact optical imaging technology to identify the spatial position of the object being measured. The object being measured will be converted into a 3D point set. The 3D vision detection system makes judgments based on the 3D point set, thereby avoiding the problem of two-dimensional machine vision being affected by external factors and affecting the reliability of detection. Therefore, in this embodiment, a mainstream depth camera is used as a 3D vision detection to obtain the depth image of colorless and transparent objects in the assembly line.
[0045] Compared with traditional RGB cameras, dual telecentric cameras have unique optical characteristics such as high resolution, ultra-wide depth of field, ultra-low distortion, and unique parallel light design. It brings a qualitative leap to machine vision precision detection and is used in various detection applications that traditional lenses cannot meet. In order to solve the problems of colorless transparent objects being light-transmissive and reflective, and being easily disturbed by external ambient light, light from all angles will cause occlusion or whitening, affecting the posture detection of colorless transparent objects, etc., this embodiment uses dual telecentric cameras to obtain RGB images of colorless transparent objects in the assembly line.
[0046] S12: align the RGB image and the depth image;
[0047] Alignment means aligning the images in the RGB image and the depth image. In this embodiment, the RGB image and the depth image are aligned to prevent deviations from occurring when the depth image is subsequently optimized.
[0048] S13: globally optimize the depth image according to the RGB image, and output the optimized depth image;
[0049] Specifically, the depth image is globally optimized according to the RGB image, including:
[0050] Input the RGB image into a pre-trained deep convolutional network to obtain multiple optimization dimensions and the optimization data corresponding to each optimization dimension;
[0051] The depth image is globally optimized according to each optimization dimension and the optimization data corresponding to each optimization dimension.
[0052] Among them, the optimization dimensions include at least: surface normal, transparent surface mask and occlusion boundary.
[0053] In this embodiment, the RGB image is input into a pre-trained deep convolutional network, and the surface normal, transparent surface mask, and occlusion boundary are obtained according to the deep convolutional network.
[0054] Preferably, there are multiple deep convolutional networks, each of which corresponds to each optimization dimension.
[0055] Correspondingly, the global optimization of the depth image based on the RGB image also includes:
[0056] The depth image is corrected and the depth is adjusted according to each optimization dimension and the optimization data corresponding to each optimization dimension.
[0057] In this embodiment, three neural networks are used: the first network is used to obtain surface normals, the second is used to obtain occlusion boundaries to deal with depth discontinuities in depth images, and the third is used to obtain occlusion boundaries, which are used to delete all pixels related to transparent objects in the depth image in order to fill in their correct depth.
[0058] S14: Identify and locate the colorless and transparent object according to the optimized depth image.
[0059] Finally, in this embodiment, the colorless transparent object is identified and located according to the optimized depth image, thereby improving the recognition accuracy of the colorless transparent object.
[0060] Embodiment 2
[0061] A colorless and transparent object recognition system applied to an assembly line, comprising:
[0062] A first camera group 1, a second camera group 2, a processor 3, a power supply 4 and a communication module 5;
[0063] The power supply 4 is respectively connected to the first camera group 1, the second camera group 2, the processor 3 and the communication module 5, and is used to provide a working power supply 4;
[0064] The processor 3 is respectively connected to the first camera group 1, the second camera group 2 and the communication module 5;
[0065] The communication module 5 is also connected to the host computer;
[0066] The first camera group 1 is used to obtain RGB images of colorless and transparent objects in the pipeline;
[0067] The second camera group 2 is used to obtain the depth image of the colorless and transparent objects in the pipeline;
[0068] Processor 3 is used to align the RGB image and the depth image; it also globally optimizes the depth image based on the RGB image and outputs the optimized depth image; it also identifies and locates the colorless transparent object based on the optimized depth image, and sends the identification and positioning results to the host computer through the communication module 5.
[0069] Preferably, the first camera group 1 comprises: a depth camera;
[0070] The second camera group 2 includes: a dual telecentric camera;
[0071] The communication module 5 includes: a GPRS wireless communication module 5, and the communication module 5 communicates based on the TCP / IP protocol.
[0072] GPRS is a new data bearing service developed on the existing GSM system, supports TCP / IP protocol, and can directly communicate with packet data network (Internet, etc.). The application scope of GPRS wireless transmission system is very wide, which can cover almost all medium and low-end services and low-speed data transmission, and is especially suitable for sudden small-flow data transmission services. Therefore, in this embodiment, GPRS wireless communication module 5 is selected, with TCP / IP protocol embedded, and industrial-grade GPRS module is used, which is suitable for the situation that the single-chip data acquisition and transmission system does not have TCP / IP protocol stack, but uses serial communication.
[0073] It can be understood that the same or similar parts of the above embodiments can be referenced to each other, and the contents not described in detail in some embodiments can refer to the same or similar contents in other embodiments.
[0074] It should be noted that, in the description of this application, the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, in the description of this application, unless otherwise specified, the meaning of "plurality" refers to at least two.
[0075] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.
[0076] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.
[0077] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.
[0078] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.
[0079] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.
[0080] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.
[0081] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.
Claims
1. A colorless and transparent object recognition method applied to an assembly line, characterized in that: include: Get the RGB image and depth image of the colorless and transparent objects in the pipeline; Aligning the RGB image and the depth image; Performing global optimization on the depth image according to the RGB image, and outputting an optimized depth image; Identify and locate the colorless and transparent object according to the optimized depth image; The globally optimizing the depth image according to the RGB image comprises: Inputting the RGB image into a pre-trained deep convolutional network to obtain multiple optimization dimensions and optimization data corresponding to each optimization dimension; Performing global optimization on the depth image according to each of the optimization dimensions and the optimization data corresponding to each of the optimization dimensions; Correcting and depth-adjusting the depth image according to each of the optimization dimensions and the optimization data corresponding to each of the optimization dimensions; The optimization dimensions include at least: surface normal, transparent surface mask and occlusion boundary; There are multiple deep convolutional networks, and each of the deep convolutional networks corresponds to each of the optimization dimensions.
2. The method according to claim 1, characterized in that The step of obtaining the RGB image and the depth image of the colorless and transparent object in the pipeline includes: Obtain depth images of colorless and transparent objects in the pipeline based on the depth camera; The RGB images of colorless and transparent objects in the production line are obtained based on dual telecentric cameras.
3. A colorless and transparent object recognition system applied to an assembly line, characterized in that: include: A first camera group, a second camera group, a processor, a power supply and a communication module; The power supply is respectively connected to the first camera group, the second camera group, the processor and the communication module, and is used to provide working power; The processor is respectively connected to the first camera group, the second camera group and the communication module; The communication module is also connected to the host computer; The first camera group is used to obtain RGB images of colorless and transparent objects in the production line; The second camera group is used to obtain depth images of colorless and transparent objects in the pipeline; The processor is used to align the RGB image and the depth image; further globally optimize the depth image according to the RGB image and output the optimized depth image; further identify and locate the colorless transparent object according to the optimized depth image, and send the identification and positioning result to the host computer through the communication module; The globally optimizing the depth image according to the RGB image comprises: Inputting the RGB image into a pre-trained deep convolutional network to obtain multiple optimization dimensions and optimization data corresponding to each optimization dimension; Performing global optimization on the depth image according to each of the optimization dimensions and the optimization data corresponding to each of the optimization dimensions; Correcting and depth-adjusting the depth image according to each of the optimization dimensions and the optimization data corresponding to each of the optimization dimensions; The optimization dimensions include at least: surface normal, transparent surface mask and occlusion boundary; There are multiple deep convolutional networks, and each of the deep convolutional networks corresponds to each of the optimization dimensions.
4. The colorless and transparent object recognition system applied to the assembly line according to claim 3 is characterized in that: The first camera group includes: a depth camera; The second camera group includes: a dual telecentric camera; The communication module includes: a GPRS wireless communication module, and the communication module communicates based on the TCP / IP protocol.
Citation Information
Patent Citations
Training method and device for transparent object recognition, storage medium and terminal
CN109903323A
Object recognition method and device, equipment and medium
CN113065521A
