Image processing apparatus and method, production system, product manufacturing method and medium

Abstract

The present invention relates to an image processing apparatus and method, production system, product manufacturing method, and medium. In order to provide an image processing apparatus capable of supporting installation of an imaging unit, the image processing apparatus includes: an image acquisition unit configured to acquire an image; a feature information extraction unit configured to extract feature information from the image; and a first generation unit configured to generate installation support feature information for supporting installation of the imaging unit based on the feature information extracted by the feature information extraction unit.

Description

Technical Field

[0001] The present invention relates to an image processing device and the like capable of supporting the installation of an imaging unit. Background Art

[0002] Technologies for controlling various robotic devices such as gripping devices, robotic arms, other actuators, and conveying devices are known for purposes such as production, quality inspection, and transportation of products. Furthermore, technologies for capturing images of objects (hereinafter referred to as workpieces) using imaging devices such as cameras, monitoring device and operating conditions by performing image processing on the recorded image data, measuring the position of the workpiece, and performing inspections are also known.

[0003] By using the above-described imaging device and image processing device, measurement and inspection by visual observation, manual assembly, manual alignment, etc. become unnecessary. In addition, switching between control instructions and operation corrections of the robot device can be performed, thereby constructing a system that performs more diverse desired operations.

[0004] Furthermore, by switching between image processing details, different processing can be performed on a single image data set, enabling measurement and inspection, image processing, and monitoring by analyzing multiple different areas. When using imaging devices such as cameras and image processing equipment to implement visual functions in production and conveyor systems, it is important to install the cameras in the appropriate position and orientation within the system, ensuring the desired composition for the workpiece and the relative positional relationship with other equipment.

[0005] Furthermore, there are cases where, for example, due to an increase in product distribution volume or production volume, the conveying and production systems described above cannot be processed using only the originally constructed systems, and it is necessary to replicate (additionally install) a system of equivalent specifications within the same or another production plant. In such a replicated system, the processing speed, accuracy, and reliability of the replicated conveying and production systems must be equal to or greater than the processing speed, accuracy, and reliability of the originally constructed conveying and production systems. Thus, complex adjustments such as adjusting the camera's position and posture, verifying the operation of the image processing device, and verifying the operation of the system are repeated as many times as desired until the desired processing speed, accuracy, and reliability are achieved.

[0006] For example, Japanese Patent Application Laid-Open No. 2020-109915 includes a determination unit for determining the type of subject within an image. Based on the determination result, the unit recommends to the user suitable imaging parameters and compositions for capturing the subject by referencing a history of past imaging methods. This helps the user successfully capture images with similar compositions by referencing images captured by professional photographers and past images.

[0007] However, while Japanese Patent Application Laid-Open No. 2020-109915 can recommend the next imaging parameters on a GUI by referencing the history of imaging parameters used in previous imaging of a subject, it is difficult to apply this technology to, for example, manufacturing systems that require highly accurate image measurement and image processing. In other words, it is difficult to apply this technology to systems that require camera installation and adjustment of position and orientation to accurately reproduce imaging positions in units of sub-pixels to several pixels.

[0008] For example, consider the process of removing workpieces such as electronic circuits measuring several centimeters from a pallet using a robot, or the process of aligning workpieces to be aligned on a discharge device in a conveyor system. In such cases, the accuracy required of the image measuring device is typically several microns to several millimeters.

[0009] In this case, without properly installed cameras, image measurement results can include large measurement errors, making it impossible for the robot to accurately grasp the workpiece and poorly perform palletizing (alignment) on the conveyor. Furthermore, there is a risk of the workpiece falling, failing to grasp the workpiece, or colliding with nearby conveying equipment and being damaged due to an undesirable grasping posture.

[0010] For this reason, in such a conveying system or production system, the relative positional relationship and posture of the camera and various peripheral devices (robot, conveying device, work table, etc.) are important, and complicated adjustment operations as described above are required.

[0011] Furthermore, not only in the copied system but also in the initially constructed system, the camera may need to be replaced due to, for example, a camera position deviation caused by some problem or human error, or a camera failure during system operation. In this case as well, there is the problem of requiring complicated adjustment operations as described above.

[0012] In the present technical field, there is a need to provide an image processing device that can solve the above-mentioned problems and support the installation of an imaging unit. Summary of the Invention

[0013] According to one aspect of the present disclosure, in order to solve the above-mentioned problem, an image processing device includes at least one processor or circuit, and the at least one processor or circuit is configured to serve as: an image acquisition unit, which is configured to acquire an image; a feature information extraction unit, which is configured to extract feature information from the image; and a first generation unit, which is configured to generate installation support feature information for supporting the installation of a camera unit based on the feature information extracted by the feature information extraction unit.

[0014] According to another aspect of the present disclosure, an image processing device includes at least one processor, which is configured to serve as: an information acquisition unit, which is configured to acquire installation support feature information for supporting the installation of a camera unit; a search unit, which is configured to search for the installation support feature information acquired by the information acquisition unit from an image acquired by the camera unit; and an installation support unit, which is configured to support the installation of the camera unit based on the result of the search performed by the search unit.

[0015] According to another aspect of the present disclosure, a production system includes at least one processor, which is configured to serve as: an information acquisition unit, which is configured to acquire installation support feature information for supporting the installation of a camera unit; a search unit, which is configured to search for the installation support feature information acquired by the information acquisition unit from an image acquired by the camera unit; an installation support unit, which is configured to support the installation of the camera unit based on the result of the search performed by the search unit; a measuring unit, which is configured to acquire measurement feature information for measuring the position and posture of an object, and measure the position and posture of the object by searching for the measurement feature information from the image acquired by the camera unit; and a holding unit, which is configured to hold the object based on the result of the measurement performed by the measuring unit.

[0016] According to another aspect of the present disclosure, a product manufacturing method includes: obtaining installation support feature information for supporting the installation of a camera unit; searching for the installation support feature information obtained in the acquisition of the installation support feature information from an image obtained by the camera unit; supporting the installation of the camera unit based on the search results obtained in the search for the installation support feature information; obtaining measurement feature information for measuring the position and posture of an object, and measuring the position and posture of the object by searching for the measurement feature information from the image obtained by the camera unit; holding the object based on the measurement results obtained in the measurement of the position and the posture; and performing predetermined processing on the object held in the holding of the object.

[0017] According to yet another aspect of the present disclosure, an image processing method includes: acquiring an image; extracting feature information from the image; and generating installation support feature information for supporting installation of an imaging unit based on the extracted feature information.

[0018] According to another aspect of the present disclosure, an image processing method includes: obtaining installation support feature information for supporting the installation of a camera unit; searching for the installation support feature information obtained in the acquisition of the installation support feature information from an image obtained by the camera unit; and supporting the installation of the camera unit based on the search results obtained in the search for the installation support feature information.

[0019] According to another aspect of the present disclosure, a non-volatile computer-readable storage medium is configured to store a computer program for an image processing device to perform the following steps: acquiring an image; extracting feature information from the image; and generating installation support feature information for supporting the installation of a camera unit based on the extracted feature information.

[0020] Further features of the present disclosure will become apparent from the following description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 10 is an overall configuration diagram of a production system including the image processing apparatus 101 according to the embodiment.

[0022] Figure 2 is a flowchart illustrating in detail an example of a system operation set for the system control apparatus 102 according to the embodiment.

[0023] Figure 3 is a block diagram of the image processing apparatus 101 according to the embodiment.

[0024] Figure 4 3 is a diagram showing a flowchart generation screen for generating the image processing program 307 according to the embodiment.

[0025] Figure 5 is a diagram illustrating a search pattern registration screen according to the embodiment.

[0026] Figure 6 is a flowchart illustrating a sequence for generating installation support information according to an embodiment.

[0027] Figure 7 : is a diagram showing feature information used for camera installation support according to an embodiment.

[0028] Figure 8 is an overall configuration diagram of a replication system according to an embodiment.

[0029] Figure 9 It shows Figure 8 FIG. 8 is a diagram of a search pattern registration screen of a copy system shown in FIG.

[0030] Figure 10 2 is a diagram showing a camera installation support screen according to the embodiment.

[0031] Figure 11 is a flowchart illustrating a camera installation support process according to the embodiment.

[0032] Figure 12 is used in Figure 5 The verification screen of the position / phase measurement result of the workpiece when the test execution button 511 is clicked.

[0033] Figure 13 is a diagram illustrating a measurement result verification screen after camera position adjustment according to the embodiment. DETAILED DESCRIPTION

[0034] Hereinafter, with reference to the accompanying drawings, preferred modes of the present disclosure will be described using embodiments. In each figure, the same reference numerals are applied to the same members or elements, and repeated descriptions will be omitted or simplified.

[0035] Figure 1 10 is an overall configuration diagram of a production system including the image processing apparatus 101 according to the embodiment.

[0036] The system control device 102 is shown, and the overall operation of the production system is controlled by sending instructions from the system control device 102 to the image processing device 101, the loader 103, the robot 104, and the ejection device 105. In addition, the system control device 102 has a computer such as a CPU built therein, and has a memory not shown built therein, and controls the entire production system by reading a computer program from the above-mentioned memory.

[0037] Figure 1 The production system shown in the figure has a loading step of using a loading device 103 to load a pallet 109, where the pallet 109 is loaded with workpieces 110 as a plurality of objects. In addition, the position and posture of the workpiece 110 are measured using an image processing device 101 and a camera 106 as a first imaging unit. In other words, the image processing device 101 obtains measurement feature information for measuring the position and posture of the workpiece, and measures the position and posture of the workpiece by searching for the above-mentioned measurement feature information in the image obtained by the imaging unit. In addition, the workpiece 110 is grasped by a robot 104 as a grasping unit based on the measurement results, and is removed from the pallet 109. The workpiece 110 is aligned in the discharge device 105, and processed, etc., thereby manufacturing a product.

[0038] The image processing device 101 has a computer such as a CPU built therein, and has a memory not shown built therein, and the image processing device 101 functions as an image processing unit that reads a computer program from the above memory and sets or performs various types of image processing. The camera 106 functions as an image acquisition unit and acquires an image of an object by performing image shooting. Although Figure 1 In the embodiment, the camera 106 and the image processing apparatus 101 are arranged inside the system as separate bodies, but the camera 106 and the image processing apparatus 101 may be configured integrally.

[0039] Similarly, the image processing system is composed of the image processing device 101 and the system control device 102, and the image processing device 101 and the system control device 102 may be formed integrally. In addition, although the image processing device 101 and the system control device 102 may be housed in the same housing as the robot 104 or the camera 106, etc., they are described as separate bodies in this embodiment.

[0040] The robot 104 has a built-in robot control device that receives control instructions from the outside and controls the operation of the robot's axes and gripping mechanisms, such as the robot's hand (end effector). In this embodiment, the robot is shown as having an articulated robot configuration, but a single-axis drive device, a rotary table, or an orthogonal robot can also be used as a drive device. Furthermore, the robot can be a system device configured by combining a robot hand (end effector) including a gripping mechanism and a suction mechanism.

[0041] The stand 107 is used to fix the camera 106 and the camera position adjustment stage 108. The camera position adjustment stage 108 can adjust the spatial position and posture of the camera by performing axis adjustment according to an external signal or manually.

[0042] In this embodiment, the system control device 102 will be described as performing control instructions for the entire system. However, it may be possible to Figure 1 The production system can be operated by controlling an operating device external to the production system shown in FIG. , or the production system can be configured to be automatically operated based on a preset schedule. Furthermore, based on the results of processing performed by the image processing device 101 and the like, other image processing can be performed by other devices at a later stage, or the robot can be controlled by the aforementioned other devices based on the results of image processing.

[0043] The image processing device 101 and camera 106 are configured to capture images in any sequence using imaging parameters, process the captured images, and output the resulting images. The camera 106 may include a mechanism for changing various imaging parameters, such as pan, tilt, zoom factor, focal length, aperture, and signal amplification. Furthermore, based on instructions from the image processing device 101 or the system control device 102, the camera can control the imaging angle by controlling the pan and tilt, as well as change various imaging parameters, such as zoom factor, focal length, aperture, and signal amplification.

[0044] In the presence of industrial uses such as Figure 1 In such a production system with many repetitive operations, the image processing device is used for visual sensing of robots and conveyors, and the operation of the entire production system is preset by the system control device 102. In addition, the timing, parameters, and composition of the imaging that are preferred for image processing are also preset.

[0045] Figure 2 102 is a flowchart showing in detail an example of the system operation set for the system control apparatus 102 according to the embodiment. The system operation according to the present embodiment operates according to a computer program generated by the user using the flowchart of the system control apparatus 102.

[0046] However, through Figure 1 After a system control program generating device (not shown) generates a system control program, the program can be downloaded to the memory of the system control device 102. Alternatively, a packaged function of a computer program including a combination of flowcharts prepared for each function or for each purpose can be provided in a form in which a user selects on a GUI or adjusts parameters.

[0047] exist Figure 2 In the flowchart shown in FIG, the system control device 102 loads the pallet 109 (on which the workpiece 110 is loaded) by the loader 103, and measures the position and posture of the workpiece 110 using the image processing device 101 and the camera 106. This flowchart shows in detail an example of the flow of a process of extracting the workpiece 110 from the pallet 109 while being held by the robot 104 and aligning the workpiece in the discharge device 105.

[0048] In step S201, the system control device 102 starts the system operation. In step S202, the loader 103 moves the stage by a predetermined feed amount according to the control instruction sent from the system control device 102 to the loader 103. As a result, the pallet 109 loaded with the workpiece 110 is transported to a predetermined position.

[0049] In step S203, the system control device 102 sends an image processing instruction to the image processing device 101. The image processing device 101 performs image processing using the image of the object captured by the camera 106 and measures the position and posture of the workpiece 110. When the image processing device 101 determines the position and posture of the workpiece 110, it responds to the system control device 102 with the result. At this point, the image processing device 101 can also use the image captured by the camera 106 to determine the type of workpiece 110 and, if necessary, change the discharge position or sort the workpieces. Alternatively, image processing can be used to perform a quality inspection to determine whether the workpiece 110 contains any quality issues, such as defects.

[0050] Furthermore, in the case where the workpiece 110 cannot be detected, the system control device 102 may determine that the feed amount of the loader is insufficient and further feed the table of the loader 103. Furthermore, the condition of the workpiece 110 or the pallet 109 may be determined based on the processing result of step S203, and for example, in the case where the workpieces overlap each other or overflow from the pallet, the system control device 102 may change the control speed of the robot 104 or stop the operation of the system.

[0051] In step S204 , based on the restored information of the position and posture of the workpiece 110 , the system control device 102 performs a correction operation of moving the robot 104 to the upper side of the workpiece 110 or rotating the hand (end effector).

[0052] In step S205 , the system control device 102 moves the hand of the robot 104 to a position where the workpiece can be grasped by the hand (end effector), for example, to the upper right side of the workpiece 110 .

[0053] In step S206, the system control device 102 controls the opening / closing of the hand to hold (pick up) the workpiece 110. Figure 1 In step S207, the robot 104 moves the workpiece to the upper side of the discharge device 105. In step S208, the system control device 102 moves the robot 104 to the position directly above the predetermined placement position, in accordance with the number of repetitions determined in step S211 of the flowchart and the externally acquired load capacity.

[0054] In step S209, the system control device 102 sets (places) the workpiece 110 at a predetermined setting position by controlling the opening / closing of the hand. Then, in step S210, the system control device 102 moves (retracts) the robot to the upper side of the transport stage.

[0055] In step S211, the number of robot operations is less than the predetermined number N1 (in Figure 1 In the case shown, N1 = 2), and if the system control device 102 determines that the setting of all two workpieces 110 corresponding to a row of horizontal alignment has not been completed, the process returns to step S203. Then, the operations of steps S203 to S211 are performed again. On the other hand, if the number of operations reaches the predetermined number N1, the system control device 102 determines that the setting of the workpieces corresponding to a row has been completed, and the process proceeds to step S212.

[0056] In step S212, when the number of operations is less than the predetermined number N2 ( Figure 1 In the case shown in , N2=3), and if it is determined that all workpieces 110 corresponding to three rows on the pallet have not been set (No), the process returns to step S202, and the loader 103 advances the pallet by one row. Then, steps S202 to S212 are performed.

[0057] In step S212, when the number of operations has reached the predetermined number N2 (in Figure 1 In the case shown in , N2 = 3), and if it is determined that the setting of all the workpieces 110 corresponding to the three rows has been completed (Yes), the process proceeds to step S213. Accordingly, the gripping of the workpieces on one pallet and the setting operation of these workpieces are completed. For the next pallet, the process is repeated. Figure 2 The entire process shown in .

[0058] Furthermore, in such a production system, the camera and image processing device can be used not only for visual sensing of the robot as described above, but also for the purpose of monitoring the operating status of the system. For example, it is possible to continuously monitor whether the workpieces loaded onto the loader 103 or the pallet 109 are damaged, whether the robot 104 has made movements that deviate from the intended operation, or whether there are any obstacles to operation near the operating area.

[0059] Figure 3 1 is a block diagram of the image processing apparatus 101 according to the embodiment. The image processing apparatus 101 includes an input / output display device 301 as a display unit configured using a liquid crystal panel or the like, and an operation input device 302 configured for various operation inputs using a keyboard and mouse, a touch panel, an input operation controller, a gesture input device, or the like. The input / output display device 301 and the operation input device 302 mainly configure a user interface.

[0060] Furthermore, the image processing device 101 can be connected not only to the camera 106 but also to the camera position adjustment stage 108 and the like for control purposes. Furthermore, an illumination device composed of a halogen lamp or a light-emitting diode lamp, etc., can be additionally provided for image capture, and an external storage device can be connected to increase storage capacity. The aforementioned units are connected via an interface 303 disposed on the internal bus of the image processing device 101. The interface 303 is configured based on the specifications applicable to communication with the aforementioned units. For example, the interface may include a network interface, a serial communication interface, etc.

[0061] The image processing apparatus 101 includes a CPU, which is a computer composed of a general-purpose microprocessor, and an arithmetic operation unit 304, which is composed of an image processing processor, etc., as a control device (which is the main processing unit of image processing). The arithmetic operation unit 304 is connected to the storage unit 305 via an internal bus (data bus, address bus, and other control lines). The storage unit 305 is composed of, for example, a ROM, RAM, or a non-volatile storage device such as an E(E)PROM. Alternatively, the storage unit 305 can be configured using an external storage device (such as an HDD or a storage device composed of a semiconductor element (not shown), or an external storage device that can be connected to the interface 303).

[0062] The data storage area 306 of the storage unit 305 is composed of a RAM area in the storage unit 305, a file area of ​​an external storage device, a virtual storage area, and the like. The data storage area 306 is used to temporarily store processing data and to store image processing setting parameters, etc. Furthermore, an image processing program 307, which is a computer program for performing image processing according to this embodiment, is stored in the storage unit 305.

[0063] The image processing program 307 changes image processing settings and performs image processing according to various operations performed using the operation input device 302 and the like. Furthermore, the image processing program 307 can save the changed details in the data saving area 306 or delete the changed details. Furthermore, the image processing program 307 can transmit and receive data with respect to an external control device via the interface 303, and the image processing program 307 can connect to an external storage unit and the like and store data in the external storage unit and receive data as input from the external storage unit.

[0064] Image processing program 307 is composed of software that implements the following functions, for example. First, image processing 308 is the main component of the image processing program that implements the following image processing. Image processing 308 utilizes image processing library 309. Image processing library 309 is, for example, a statically or dynamically linked library and is installed in storage unit 305. Image processing settings 310 determine the behavior of image processing 308 based on various operations performed through input device 302 and the like.

[0065] Furthermore, the image processing program 307 includes input / output (I / O) routines that implement the following functions. Specifically, the I / O routines include external device control 311, saved data generation 312, instruction acceptance 313 from an external control device, and temporary storage processing 314 for temporarily storing data in the RAM area and the buffer area of ​​the arithmetic operation unit 304. Furthermore, the I / O routines include display screen generation 315, saved data output 316, and operation acceptance 317 from the operation input device 302. Each of these functions is installed in the storage unit 305 in the form of an application (utility) program or a subroutine composed of a statically or dynamically linked library.

[0066] By executing the image processing program 307, the image processing apparatus 101 can control the camera 106 and perform image processing using the arithmetic operation unit of the image processing apparatus.

[0067] Based on such operations or instructions, the arithmetic operation unit 304 can call various functions of the image processing program 307 or library, perform arithmetic operation processing, and transmit the image processing results to the system control device 102. In addition, the arithmetic operation unit 304 can accumulate (record) the image processing results in an external storage device. In addition, the arithmetic operation unit 304 can combine the screen configuration pre-stored in the program and the image processing results into a screen, and display the combined screen on the input / output display device 301.

[0068] Figure 4 1 is a diagram showing a flowchart generation screen for generating the image processing program 307 according to the embodiment. By executing the flowchart generated here, the image processing apparatus 101 realizes the process for performing Figure 2 In addition, the image processing step S203 in the system control can be displayed in the input / output display device 301. Figure 4 The image processing program 307 according to the present embodiment operates according to a flowchart generated by a user using the image processing apparatus 101 .

[0069] As another embodiment, the following form may be adopted: Figure 1 After an image processing program generating device (not shown) generates the image processing program 307, the image processing program 307 is copied to the storage unit 305 of the image processing apparatus 101. Alternatively, the image processing program may be provided in a form in which a user selects a packaged function including a combination of image processing flowcharts generally prepared in advance for various functions or purposes on a GUI, or adjusts parameters.

[0070] exist Figure 4 , a list 401 of processing parts of the flowchart is shown. The user drags and drops the parts from the list 401 to the flowchart area 402 using the mouse of the operation input device 302, and connects the parts with lines, thereby generating a flowchart.

[0071] An example of the image processing flowchart 403 is shown. Here, the internal processing of the image processing 203 in the operation flow of the system control will be described as an example.

[0072] In step S404, the image processing apparatus 101 accepts an image processing request from the system control apparatus 102. At this time, a flowchart of image processing can be specified, and image processing parameters for execution and timing of image data to be used can be specified.

[0073] In the image data acquisition process of step S405, the image processing apparatus 101 acquires image data of the workpiece, which is the object imaged by the camera 106. Alternatively, the image to be processed may be acquired from the recorded data accumulated by the camera. For example, the acquired data may be the most recent image data in the accumulated recorded data or image data at a measurement time specified by the system control apparatus.

[0074] In step S406, the generated flowchart checks (searches) for the presence or absence of artifacts in the acquired image data. For example, a presence / absence checking method can be obtained from the image processing library 309, and this presence / absence checking method can be, for example, a method that uses brightness information and color information to extract the artifact and then determines the presence / absence based on whether the area of ​​the artifact is equal to or greater than a predetermined value. Alternatively, the presence / absence can be calculated through pattern matching using shape feature information such as concentration distribution and brightness gradient, and pixels with a brightness gradient equal to or greater than a predetermined gradient can be extracted as edge information, and the position, center of gravity, and inclination of the edge information can be used to check for the presence / absence.

[0075] In step S407, the presence or absence of a workpiece is determined. If the inspection result obtained in step S406 determines that the workpiece is not present (No), this result is recorded in step S409, and a reply is sent to the system control device 102 using this result in step S410. If it is determined that the workpiece is present (Yes) in step S407, the process proceeds to step S408.

[0076] In step S408, the position and phase of the workpiece are measured. A method for measuring the position and phase of the workpiece can be used that reads and obtains measurement feature information, such as a local pattern image of the workpiece's concentration distribution, pre-recorded and stored on a recording medium, and then searches for positions and phases with a strong correlation coefficient with the measurement feature information from the input image. Alternatively, a unit can be installed that is configured to extract feature information, such as the shape of the workpiece's brightness gradient, from an image and generate measurement feature information for measuring the object's position and posture based on the shape feature information. The measurement feature information can then be used to search for positions and phases with a strong correlation coefficient on the input image.

[0077] In step S409, which records results, data such as the execution results and judgment results obtained in steps S404 through S408 is recorded. The start time of step S404, which receives a request and details about a specific execution option, may also be stored. Furthermore, image data obtained in step S405, which records images, image capture time, image size, camera parameters, and camera information, may be stored as metadata. Furthermore, workpiece extraction area information, density information and gradient information used for inspection, and color information obtained in step S406 may be stored. As the execution result of step S406, the workpiece's detected position and detection phase, as well as correlation values, may also be recorded.

[0078] The execution results of the flowchart generated in this manner are temporarily stored via temporary storage processing 314 performed by the image processing apparatus 101, or stored in the data storage area 306. Furthermore, the execution results can be referenced or deleted as needed, transmitted or received via the interface 303, or stored or input by connecting to an external storage unit, etc. In step S410, the results of step S409, or some of them, are output, and the flowchart ends. When the results of step S409 are output, they can be transmitted to the system control apparatus 102, stored in the data storage area 306 of the image processing apparatus, or output to and displayed on the input / output display device 301.

[0079] Furthermore, when the user clicks button 411, the image processing apparatus 101 generates an image processing program 307 for executing the flowchart displayed in the flowchart area 402. Furthermore, the user can add a new flowchart to the system by entering a new flowchart name in list box 412. Furthermore, the user can use list box 412 to select a flowchart to be displayed in the flowchart area 402. When button 413 is clicked, the generation of the image processing program is completed, and the flowchart area 402 is closed.

[0080] In addition, when the user double-clicks any one of the steps that are part of the image processing flowchart 403, a transition to a setting screen for setting detailed processing of the part can be made. Figure 5 A method for registering a characteristic pattern for realizing the position / phase measurement function is described using the setting screen of step S408 for measuring the position and phase as an example.

[0081] In addition, a method will be described for generating support data for supporting installation operations in a case where a user installs an initial (first) system and then installs an equivalent system a second or subsequent time, or in a case where a user installs an initial system and then changes the installation status of a camera, etc.

[0082] Figure 5 is a diagram showing a search pattern registration screen according to an embodiment, and will refer to Figure 5 A user interface for performing settings for image processing in step S408 as a position / phase measurement step (part) of the workpiece will be described.

[0083] In the position / phase measurement step S408, a search is performed on the input image for a position and phase with a strong correlation coefficient, or a local pattern image of the concentration distribution of the workpiece as the object is used to extract shape feature information of the brightness gradient of the workpiece. By using such feature information, a search is performed on the input image for a position and phase with a strong correlation coefficient. Figure 5 GUI for registering a pattern image (measurement feature information) for searching will be described with reference to the screen shown in FIG.

[0084] exist Figure 5 5 shows a button 502 of a pull-down menu for selecting an input candidate of a “setting image” for setting (an image that becomes a reference for setting image processing). Figure 5By selecting camera 106 as the "Camera to Use" in the image processing apparatus 101, the "Camera Image" is displayed as the "Setting Image." When the user clicks button 502, the captured image acquired by camera 106 is displayed as the "Camera Image." Additionally, the file names of images previously saved in the data storage area 306 of the image processing apparatus 101 can be displayed as options in a list format, allowing the user to select one of these file names. When the user selects a character string from this list, the selected image is registered as the setting image.

[0085] exist Figure 5 , "camera 106" is selected as the "camera to be used" and "camera image" is selected as the "setting image", and the captured image acquired by the camera 106 is used as the setting image. When the button 502 is clicked, an image acquisition step for acquiring an image is performed, and the captured image (camera image) acquired by the camera 106 is displayed in the image display area 503 as the setting image.

[0086] Next, the user sets a search pattern setting area 504 as an area for setting a search pattern (measurement feature information). For this setting, the area shape is first selected using the button 505.

[0087] As the area shape, a shape such as a rectangle, circle, ellipse, or ring can be selected from a pull-down menu using button 505. Furthermore, area addition and area removal using a combination of these shapes can be performed. Furthermore, a free shape can be set using the operation input device 302 such as a mouse. In this embodiment, a case where a rectangular search pattern setting area 504 is set will be described as an example.

[0088] As setting item 506 for a rectangular area, the upper left x-coordinate and lower right x-coordinate are set. Here, an image coordinate system is used, with the upper left side of the captured image as the origin, and a horizontal x-axis and a vertical y-axis. As a method for specifying setting item 506 for the area, in addition to directly entering a numerical value, a function can also be implemented, such as providing increment / decrement buttons and allowing adjustment of the setting item by clicking the increment / decrement buttons.

[0089] At this time, the search pattern setting area 504 is displayed in a form superimposed on the image display area 503 and is updated and displayed at any time. According to this, the user can set the search pattern setting area 504 while checking the setting image in the image display area 503. Here, using Figure 5 The above-described setting step of the GUI shown in serves as a second generating unit (second generating step) for generating measurement feature information used to measure the position and orientation of the object based on the above-described extracted feature information.

[0090] Next, a sequence for setting a search method using the search pattern setting area 504 will be described.

[0091] Here, for example, "shape feature" is selected as the search method using the search method 507. The search method based on "shape feature" is a method in which shape feature information of the brightness gradient of the workpiece is extracted and, using this feature information, a search is performed for a position and phase on the input image that has a strong correlation coefficient with the search pattern setting area 504.

[0092] In addition to these, at least one of "normalized correlation" and "matching" can be selected as a search method. "Normalized correlation" is a method in which local pattern information, such as the workpiece's concentration distribution, is normalized and a search is performed for positions and phases that have a strong correlation coefficient with the search pattern setting area 504. Furthermore, "matching" is a method in which, for example, if the camera 106 can acquire three-dimensional information about the workpiece, a three-dimensional shape is extracted as a model, a three-dimensional shape is searched within a three-dimensional space, and the distance between the two shapes is minimized. Furthermore, at least one of the search methods, such as "shape feature," "normalized correlation," and "matching," can be configured to be selectable based on the model type of the camera serving as the imaging unit.

[0093] In addition, Figure 5 In the example shown in FIG, "Shape Feature" is selected as the search method, and when the correlation value between the search pattern setting area 504 and the workpiece exceeds a predetermined threshold, it is determined that the shape feature has been detected. For example, when the correlation value exceeds the value set in the pre-specified detection threshold 508, it is determined that the shape feature has been detected.

[0094] In addition, when the search pattern setting area 504 can be changed to a specific zoom-in / zoom-out range 509 or a perspective distortion (swing and pitch distortion) range 510, a search for artifacts can be performed inside the image, and artifacts entering the threshold range can be detected. Figure 5 In the example shown in , the enlargement / reduction range 509 is set to 0.9 times to 1.1 times, whereby similar workpieces can be detected from inside the image while allowing a size difference from the set search pattern to be less than 10 percent.

[0095] This allows robust detection of workpieces even when the distance to the workpiece is unclear or when individual workpieces vary. Furthermore, because the perspective distortion range 510 is set to ±5°, similar workpieces can be detected from within the image while allowing for ±5° of pan and tilt (rotation around the horizontal axis, vertical axis, or image rotation) relative to the set search pattern. This allows robust detection of workpieces even when the workpiece is tilted.

[0096] After the settings of 505 to 510 are completed, when the test execution button 511 is clicked, the camera 106 selected as the "camera to be used" is used to take pictures, the position and phase of the workpiece are measured for the acquired image, and the measurement results of the position and phase of the workpiece can be checked.

[0097] Figure 12 Is when Figure 5 5. The inspection screen of the position / phase measurement result of the workpiece when the test execution button 511 is clicked in FIG. An example is shown in which similar feature information is detected in the image of the image display area 503 using the search pattern set in the search pattern setting area 504 and the result is displayed.

[0098] Workpieces 1201 to 1206 for which shape features similar to those of the search pattern have been detected are highlighted and displayed. While IDs are assigned in order from highest similarity (correlation) to lowest similarity, IDs may be assigned in ascending or descending order of detection coordinates, or based on a detection sequence, etc. The number 1207 of workpieces for which shape features similar to those of the search pattern have been detected is shown.

[0099] Window 1208 is used to select the ID assigned to each inspection workpiece and to display detailed inspection information for the selected ID. Window 1209 displays detailed inspection information related to the ID selected in window 1208. Figure 12 "1" is selected in window 1208 in , thereby displaying the following result, which indicates that the search has been performed based on "shape feature" as the search method, the correlation value as the score is 0.99, and the difference in position and rotation phase is almost 0.

[0100] When the cancel button 513 is clicked in this state, the settings set as described above and the setting change portion are discarded.

[0101] On the other hand, when the OK button 512 is clicked, the settings of the search pattern setting area 504, the settings of the buttons 505 to the perspective distortion range 510 related to position / phase measurement, and the like are all set as described above. Figure 12The detection results shown in , etc. are saved in the data saving area 306 in association with the image processing program 307, etc. These pieces of information are read when the program is executed or when a transition is made to the image processing setting screen.

[0102] Furthermore, when the camera installation support information save button 514 is checked when the OK button is clicked, a process of generating installation support information for supporting camera installation is performed, and the result of the process is also saved in association with the image processing program 307. This will be described below.

[0103] Figure 6 is a flowchart illustrating a sequence for generating camera installation support information (support information for installation) according to the embodiment. Figure 6 The sequence shown in the figure shows the Figure 5 Flow that the arithmetic operation unit 304 of the image processing apparatus 101 performs when the OK button 512 is clicked with the camera installation support information save button 514 in the state shown in FIG.

[0104] In addition, it is possible to input a camera image. Figure 6 The process shown in FIG. 5 or the process can be performed in the background during the adjustment of the search pattern setting area 504 Figure 6 By such a configuration, preprocessing such as outline extraction, shadow calculation, and normalization can be performed in parallel, reducing the user's operation standby time and enabling smooth generation of camera installation support information.

[0105] The arithmetic operation unit 304 of the image processing device 101 begins the process in step S601 and extracts peripheral features in step S602. In step S602, feature information corresponding to shape features is extracted from the area surrounding the search pattern setting area 504, where the search pattern is set in the camera input image. The reason for extracting feature information from the peripheral area is that when not only the features of the workpiece but also the features of its surrounding environment are used as information for camera installation, the accuracy is likely to be higher.

[0106] As the range of the surrounding area, if there is room for the capability and time of the arithmetic operation unit 304, feature information can be extracted from the area including the search pattern setting area 504 (for example, the entire image display area 503). In addition, feature information can be extracted within a range corresponding to a preset number of pixels.

[0107] In other words, feature information for generating the mounting support feature information is extracted from a first region of the image, and feature information for generating the measurement feature information is extracted from a second region of the image that is different from the first region.

[0108] Furthermore, feature quantities may be extracted from the surrounding area excluding the search pattern setting area 504. Here, step S602 functions as a feature information extraction unit (feature information extraction step) that extracts feature information of an object or its surroundings from an image, for example.

[0109] Furthermore, if a feature is found within a predetermined range of the periphery, the region may be widened while widening the feature extraction region until the feature is found in the periphery.

[0110] Figure 7 70 is a diagram illustrating feature information for camera installation support according to the embodiment, and illustrates an example of shape features (feature information) 701 extracted from the entire image in which a search pattern is set.

[0111] Then, in Figure 6 In step S603 shown in FIG, the extracted features 701 are separated. Separation methods can be considered based on various types of information such as position and shape, gradient intensity, shadow average, and shadow deviation, and such options can be configured to be selectable in the GUI or setting file. Here, for simplicity of explanation, it is assumed that separation is performed based on the connectivity (continuity) of the extracted features 701. For example, the column "Feature Information" 702 is the separation result of the features 701 based on connectivity.

[0112] In step S604, the separated features 701 are evaluated. Information contributing to the adjustment of the camera position (in other words, the relative position between the camera and the workpiece) is used as an evaluation metric. For example, the evaluation can be based on various types of information, such as the size of the feature shape, the magnitude of the shadow deviation, and distance information relative to the search pattern setting area 504 on the image where the workpiece model was generated. Selection of such information can be configured in a GUI or a settings file. Here, step S604 serves as an evaluation unit for evaluating the feature information extracted by the extraction unit.

[0113] Here, it is assumed that the evaluation ranking (priority) is determined based on the image size of the separated features 701 as the above information. In other words, the size of the separated features 701 is evaluated in step S604, and then the features are sorted based on this size in step S605, and the sequence to be prioritized for camera adjustment support is determined. Based on this, installation support feature information is generated, with the priority assigned according to the evaluation.

[0114] Here, step S605 functions as a first generating unit (a first generating step) that generates mounting support feature information for supporting mounting of the imaging unit based on the extracted feature information.

[0115] The image graph shows the installation support feature information 702 generated by separating the feature information based on connectivity and sorting it using size information as an evaluation index. In the image graph, IDs are sorted in order of priority.

[0116] Step S606 is a process for saving the generated installation support feature information. In other words, the search information, such as the settings of the search pattern setting area 504 set as described above, and the settings of the buttons 505 to the perspective distortion range 510 related to position and phase measurement, is saved. Furthermore, by clicking the OK button 512, the search information, such as the installation support information extracted and calculated in steps S602 to S605, is also saved in the data storage area 306 of the image processing device 101, in association with the image processing program 307. The process then ends in step S607.

[0117] Furthermore, the installation support information includes feature information represented by feature 701 and feature information 702, the size, position, phase, intensity gradient information, color information, evaluation value, priority for camera installation support, and the original image displayed in image display area 503. Furthermore, the installation support information is used as search information for searching for an object.

[0118] Furthermore, the above-described setting information and installation support information can be stored in the temporary storage process 314 and the data storage area 306, and can be sent / received to the outside through the interface 303. Furthermore, the above-described information can be stored in a connected external storage device or the like or can be input from the external storage device.

[0119] Next, refer to Figures 8 to 13 Describes a method for reproducing images obtained by using mounting support information and a camera and image processing. Figure 1 The system is equivalent to the production system shown in , and supports the method of installing the second camera and subsequent cameras in the same production system. In addition, support is provided by evaluating the reproducibility of the installation position and posture of the camera, outputting the evaluation results, and also outputting the equivalent image processing results.

[0120] Figure 8 is an overall configuration diagram of a replication system according to an embodiment, and shows Figure 1 A replication system for the production system shown in .

[0121] Figure 8 The image processing device 801 shown in FIG. 8 has a workpiece 810 with Figure 1 The image processing device 101 to the workpiece 110 shown in FIG. 1 have the same configuration, and thus description thereof will be omitted. Figure 8 The control of the system is carried out with Figure 2 The camera 806 serves as a second imaging unit for acquiring an image of an object.

[0122] In addition, the internal configuration and operation flow chart of the image processing device and the description of the screen generation and the method for registering the search pattern have the same meaning as Figures 3 to 5 The configuration is similar to the one shown in . Assuming that Figure 1 Various types of setting information set in the production system shown in and search information such as installation support information are saved by connection with an external storage unit or the like via the interface 303 .

[0123] In this state, Figure 8 The physical arrangement and features of the image processing device 801 to the workpiece 810 in the replication system shown in FIG. Figure 1 In the case of reproducing the characteristics and accuracy equivalent to those of the production system shown in FIG, the image processing of the image processing device 801 can be performed in the same manner as in FIG. Figure 4 The processing is similar to the processing of the flowchart 403 shown in FIG. In addition, the image processing flow can also be performed in this way. Figure 2 The process shown in , which can be easily copied (added) with Figure 1 A production system (transportation system) with an operating capacity equivalent to that shown in .

[0124] However, in reality, even when replicating a production system (conveyor system) by arranging and producing devices according to the system's plan, assembly errors may occur during each assembly process, or individual differences may exist between individual devices or components. Consequently, such errors accumulate throughout the system, making it extremely difficult for a replica system to reproduce similar system operation immediately after assembly. Consequently, after system assembly, numerous adjustments and operational verifications are required for each device and component, accompanied by trial and error in order to reproduce the intended system operation.

[0125] Figure 9 It shows Figure 8 The diagram of the search pattern registration screen of the copy system shown in FIG. The components 502 to 513 of the screen are the same as those of FIG. Figure 5 The constituent elements 502 to 512 shown in are the same, and thus their description will be omitted. In addition, various types of setting data of the image processing apparatus are reproduced by reading the processing parameters that have been set and saved in each processing block of the image processing flowchart 403 that has been saved when the initial system is set.

[0126] In addition, by reading the data saved as search information, the settings of the search pattern setting area 504 and the parameter settings 505 to 510 related to the measurement of position and phase, in other words, the measurement feature information for measuring the position and posture of the object, etc. are obtained.

[0127] The image processing device 801 then searches for the measurement feature information in the image obtained by the camera to measure the position and posture of the workpiece. Furthermore, the desired product is manufactured by performing predetermined processing based on the position and posture of the object as measured, such as using the robot 804 to grasp the object and transport it to a predetermined position such as a transport device 805, or performing predetermined processing.

[0128] However, at this time, in the camera 806 in the copy system, there is a high possibility that the relative positional relationship between the camera 806 and the workpiece 810 or between the pallet 809 on which the workpiece 810 is mounted and the loading device 803 is not physically reproduced with high accuracy. Figure 9 As shown, the position of the search pattern setting area 504 and the position of the workpiece 810 do not coincide with each other. Figure 9 In the state shown, appropriate settings cannot be made, and even when pattern search (measurement of position and phase) and image processing and system shown in the image processing flowchart 403 are performed, desired operation cannot be reproduced.

[0129] In this case, the system installer needs to make adjustments by adjusting the position and posture of camera 806 using camera adjustment stage 808, or by adjusting the layout of gantry 807 and loading device 803, etc., so that the desired image measurement can be performed. Furthermore, when making these adjustments, it is necessary to repeatedly verify the images captured by the camera, the operation of the image processing device, and the operation of the system while adjusting the relative positional relationships between the camera and various peripheral devices (robots, conveyors, work tables, etc.). Therefore, ensuring the required processing speed, accuracy, and reliability requires considerable effort and time.

[0130] In contrast, in this embodiment, efficient installation operation of such an imaging device can be achieved.

[0131] Figure 10 is a diagram showing a camera installation support screen according to the embodiment, and Figure 11 is a flowchart illustrating a camera installation support process according to the embodiment.

[0132] In the following, reference will be made to Figure 10 and Figure 11The operation of the program for supporting adjustment of the camera installation position according to the present embodiment will be described.

[0133] A screen 1001 is shown for supporting the installation of a camera, and the Figure 9 A button for starting the camera installation support processing program is provided on the setting screen 501 shown in FIG, and the camera installation support processing program can be configured to start in a case where the setting of the image processing is not well performed, etc. Figure 4 A button for starting the camera installation support processing program is provided on the screen of the flowchart of the image processing such as the flowchart area 402 shown in FIG. 1 , and a button for starting the camera installation support processing program can be provided. Figure 4 Before setting up each image processing step, adjust the camera's installation position and check the installation status.

[0134] A button 1002 is used to select an input candidate of an image as the installation support object, and the image is configured here as a camera image, and a stored image or the like that has been previously photographed by a camera can be read.

[0135] Button 1003 is used to designate a camera as an installation support target, and here, camera 806 is selected. In an image processing system to which a plurality of cameras are connected, the plurality of cameras may be configured to be selectable, and an information source for supporting the installation of the camera may be selected according to the selected camera (in Figure 7 Switch between pre-saved information in the

[0136] The display area 1004 is used to display the input image of the camera 806 and information for supporting the installation of the camera in a superimposed manner.

[0137] Check box 1005 indicates whether to perform camera installation support processing, and when the check box is checked, the camera installation support processing is performed. For example, in the case where the computational operation load of image processing for camera installation support is high and image processing causes hindrance in other image processing and operations, this check box may be turned on only when performing camera installation adjustment. Figure 11 The flowchart shown in describes the details of the camera installation support process in detail.

[0138] The button 1006 is used to select the feature ID used for the camera installation support process. In other words, the button 1006 serves as a setting unit for setting search information based on an image, and can set predetermined feature information as search information based on the result (priority) of the evaluation performed by the evaluation unit.

[0139] Here, as in Figure 7As described in the feature information 702 shown in FIG. 1 , the usage feature ID is an ID number associated with feature information to which a specific evaluation value is assigned.

[0140] The result 1007 is a result of searching for installation support feature information corresponding to the usage feature ID1 on the input image, and is displayed in a form superimposed on the input image.

[0141] The result 1008 of the search for the mounting support feature information on the input image is represented as a numerical value or the like. In this embodiment, Figure 5 As shown in the search method 507 shown in FIG, shape feature information is used for image processing to measure position and phase, and therefore the shape feature is selected as the installation support feature information for supporting camera installation. However, local shading information, etc., can be used as the installation support feature information, and when the camera uses an imaging element capable of obtaining a three-dimensional shape or a stereo camera, three-dimensional shape information can be used.

[0142] In addition, the image search result 1008 using shape feature information displays a search score (similarity, correlation value), the magnification / reduction ratio at the point of closest agreement, the horizontal position, the vertical position, and the rotation values ​​in each axis. The user adjusts the camera position while referring to these values. Generally, it is preferable to adjust the position so that the degree of agreement with the previously stored feature information for camera installation support is maximized. In other words, the search score is as high as possible, the magnification / reduction ratio is as close to 1.0 as possible, and the deviation amount in each axis is as close to zero as possible.

[0143] Furthermore, in this embodiment, although adjustment can be performed while referring to the search score, the magnification, and the deviation amount in each axial direction, at least one of these three pieces of information may be displayed, and adjustment may be performed while referring to the at least one piece of information.

[0144] The guide information 1009 defines the axes of the input image with the horizontal direction being set as the X axis, the vertical direction being set as the Y axis, and the depth direction of the image forming surface being set as the Z axis. However, the directions of the axes can be set according to the situation of the camera and image processing library used.

[0145] Visual support information 1010 and 1011 supports camera installation. For example, the direction and length of arrows can be used to display the direction of deviation in the X and Y axes, the direction of deviation in the rotational direction about the center of each axis, and the amount of these deviations, as in visual support information 1010. Alternatively, the position of feature information of the original image serving as the target can be displayed in a superimposed manner, as in visual support information 1011. This allows the user to visually obtain support information for adjusting camera installation. In this way, by displaying at least one of the adjustment direction and amount of the imaging unit based on the search results, the adjustment process can be shortened.

[0146] Checkbox 1012 is used to execute an automatic setup adjustment program for automatically performing camera setup. By checking checkbox 1012, the automatic setup function is enabled. This function can be used, for example, when camera 806 or camera adjustment stage 808 includes adjustment mechanisms (adjustment units) for pan, tilt, position adjustment, and zoom. Furthermore, the adjustment unit can be configured to adjust at least one of pan, tilt, and zoom.

[0147] While operating such an adjustment mechanism, automatic adjustment is performed so that the various numerical values ​​indicated in 1008 become appropriate. Furthermore, the image processing system may be configured to automatically detect whether the aforementioned adjustment function (adjustment mechanism) is present in the camera 806 or the camera's adjustment stage 808, and, if it is determined that the adjustment function is present, to display a checkbox 1012 so that it can be selected. On the other hand, if it is determined that the adjustment function is not present, the checkbox 1012 may be grayed out or may not be displayed so that it cannot be selected.

[0148] When making adjustments, the direction in which at least one of the values ​​of the search score, magnification, and deviation amount in each axis direction of 1008 reaches or is less than a predetermined value can be found by scanning the entire movable range of the adjustment mechanism. Alternatively, the direction in which the weighted sum of such values ​​reaches or is less than a preset value can be found. Alternatively, the adjustment amount of the adjustment mechanism and the various values ​​of 1008 that change with such adjustments can be used to perform proportional control or PID feedback control.

[0149] The target value 1015 of the error between the feature information set when performing camera installation support and the search result is shown, and for example, when the amount of position deviation in each axis is within 5 pixels, it is judged to be successful (adjustment completed).

[0150] The OK button 1013 is used to end the camera installation adjustment support and end this screen 1001 when preferable camera adjustment can be performed while referring to the display 1015 .

[0151] In the case of automatic adjustment, button 1013 can be configured to be automatically turned on when the error converges to target value 1015. In the case of manual adjustment, the user can check that the error converges to target value 1015 and end the adjustment by turning on button 1013. When the adjustment is completed, the usage feature ID, camera image, information related to the camera used, and various numerical information (installation support information) 1008 can be recorded.

[0152] The cancel button 1014 is used to terminate the camera installation adjustment support and close this screen 1001 when preferable camera adjustment cannot be performed even using pre-prepared feature information and search conditions.

[0153] Then, as mentioned above, Figure 11 This is a flowchart showing the camera installation support process according to the embodiment, and is performed by a computer inside the image processing system executing a computer program stored in a memory. Figure 11 The processing of each step is shown in FIG.

[0154] like Figure 11 As shown in FIG, the camera installation support process starts in step S1101. The start timing is Figure 10 In the example shown in , the timing at which the check box 1005 of the camera installation support process is clicked. However, in addition to this, the timing at which the screen 1001 is opened or the like may be set as the start timing.

[0155] In step S1102, the feature information used for the search is read. Figure 10 In the example, select ID1 as the feature ID by pressing button 1006. Figure 7 As shown in step 702, the feature information of ID1 having the largest size is read as the installation support feature information. In other words, step S1102 functions as an information acquisition unit that acquires the installation support feature information for supporting the installation of the imaging unit (information acquisition step).

[0156] In step S1103, an image to be processed is input. In other words, an image to be processed is input from the camera 806 or a storage device.

[0157] Step S1104 is a process of extracting features of the input image, and extracting feature information for search processing from the image input from a camera, etc. Here, because "shape feature" is selected as the search method, shape feature information is extracted using brightness and gradient information of the image, etc.

[0158] Step S1105 is a feature information search process, and based on the feature information extracted in step S1104, a search is performed for the position, phase, and magnification / reduction ratio, etc. of feature information having a strong correlation coefficient with the mounting support feature information read in step S1102. In other words, step S1105 functions as a search unit (search step) for the mounting support feature information acquired by the information acquisition unit from the image acquired by the self-photographing unit. When the search is performed, the image acquired by the self-photographing unit is detected. Figure 5 Similarly, when the search pattern setting area 504 is registered on the setting screen 501 for measuring the position and phase shown in FIG, parameter settings from the button 505 to the perspective distortion range 510 related to the measurement of the position and phase can be used.

[0159] Alternatively, a pattern search may be performed using a setting screen or setting file similar to 501. Furthermore, as described above, the search unit may perform a search using at least one search method among "shape feature", "normalized correlation", and "matching" based on the installation support feature information.

[0160] This allows for robust search for feature information, even when there are deviations in the input image, workpiece, or measurement environment. In other words, even with minor differences between the initial system and the environment, feature information can be detected and output as a numerical value and supporting information for correcting the camera's position.

[0161] Step S1106 is result output processing.

[0162] picture Figure 10 The search result obtained in step S1105 is output and displayed as a numerical value, as shown in the installation support information 1008 shown in FIG. At this time, step S1105 functions as a display unit for displaying the search result obtained by the search unit, and as an installation support unit for supporting the installation of the above-mentioned imaging unit based on the search result obtained by the search unit (installation support step).

[0163] In step S1107, the search results obtained in step S1106 are evaluated. If the result is favorable (for example, if the installation support information 1008 is within the range of the target value 1015), the judgment is yes, and the camera installation support process ends. On the other hand, if the evaluation is not favorable, the judgment is no, and the process proceeds to step S1109.

[0164] In step S1109, it is checked whether the check box 1005 for the camera installation support process is on. If the check box is on, the judgment is yes, and the process proceeds to step S1110. On the other hand, if the check box is off, the judgment is no, the process proceeds to step S1108, and the process flow ends.

[0165] In step S1110, it is detected whether the camera 806 or the camera adjustment platform 808 has an adjustment function that can adjust the direction, position and zoom of the camera, and if the above adjustment function exists, it is determined whether the check box 1012 for automatic adjustment of the camera installation is turned on.

[0166] Then, if it is determined to be YES in step S1110, the process proceeds to step S1111 of the camera position automatic adjustment process. On the other hand, if it is determined to be OFF, it is determined to be NO and the process proceeds to step S1112 of the manual position adjustment process.

[0167] In step S1111, the camera position is automatically adjusted. If the camera 806 or the camera adjustment stage 808 has an adjustment function (adjustment mechanism) for adjusting at least one of pan, tilt, and zoom of the camera, the amount of adjustment is instructed.

[0168] As described above, while scanning the entire movable range of the adjustment mechanism, an evaluation value is acquired while, for example, varying the indicated value using appropriate intervals, and adjustment can be terminated when the evaluation value reaches or falls below a predetermined value. Alternatively, adjustment can be controlled so that it continues until the optimal evaluation value is obtained. In other words, step S1111 functions as an automatic adjustment unit that adjusts the position or posture of the imaging unit based on the search results, adjusting at least one of pan, tilt, and zoom.

[0169] In the case of using feedback control or the like in step S1111, the next indicated value can be determined based on the relationship between the previous evaluation value and the indicated value, and the adjustment can be completed at the time when the evaluation value becomes a predetermined value or less than a predetermined value, or at the time when the evaluation value converges within a predetermined width. In step S1111, if an automatic adjustment instruction is given, the process returns to the image input in step S1103.

[0170] In step S1112, the camera position is manually adjusted. This adjustment is performed, for example, when the camera adjustment mechanism or the stage adjustment mechanism is a manual adjustment mechanism, or when adjustment is required outside the movable range of the automatic adjustment mechanism.

[0171] In addition, Figure 11In the flowchart shown in FIG, after the camera is adjusted (step S1111 or step S1112), the image input step S1103 is performed. However, instead of proceeding to step S1103, it may be determined whether the evaluation value has improved. If there is no improvement, the feature ID used for the search may be switched to another ID, and the reading step S1102 may be performed. In this case, it is preferable to change the ID number so that the size of the feature information is reduced.

[0172] In this manner, the installation positions of the cameras can be adjusted while attempting to switch the feature information supporting the installation of a plurality of cameras, whereby installation of a preferred camera can be performed.

[0173] Figure 13 This is the image measurement result verification screen. Figure 5 、 Figure 9 and Figure 12 1 and 2. In the drawings, the same reference numerals denote the same components, and thus description thereof will be omitted.

[0174] In order to perform image processing similar to that of the image processing apparatus 801 of the first system, the respective image processing apparatuses 801 in the second and subsequent (copy) systems call Figure 4 Then, by clicking on step S408 as the position / phase measurement processing unit within the flowchart area 402, the Figure 13 The setting screen 501 shown in FIG. Figure 10 On the screen 1001 supporting installation of the camera shown in FIG, setting can be made so that automatic transition to the next step is performed at the point in time when the installation target value 1015 is satisfied.

[0175] exist Figure 13 , the measurement results of the position and phase of the workpiece when the test execution button 511 is clicked are displayed in the image display area 503. In other words, a search is performed using the search pattern setting area 504 set for the image processing device 101 of the initial (first) system, and similar feature information is detected from within the captured image acquired by the camera 806, and the results are displayed.

[0176] 1301 to 1306 emphasize and display the artifacts whose shape features are similar to the search pattern and are detected. Figure 12Similarly, IDs are assigned in order from highest to lowest search score (similarity, correlation). Due to the fact that the workpiece is placed on pallet 809, which has weak physical positional constraints and the influence of individual differences, ID 1 is assigned to the workpiece. Based on the detection information 1309 at this time, it can be understood that there is a small amount of rotational deviation from the X-axis to the Z-axis (the camera, pallet, and workpiece can face each other), and that this position is measured to ensure high accuracy in pallet placement.

[0177] By performing such a result check display, it can be easily verified that image measurement results equivalent to the results of image processing of the initial (first) system are obtained in the second or subsequent system, or the second and subsequent camera installation operations in the same system. Here, it is obvious that whether the output value is within the preferred error range depends on the performance required of the system, etc.

[0178] As described above, in this embodiment, by using the camera and the image processing device adjusted in the initial (first) conveying system or the production system, a search pattern (measurement feature information) for measurement can be generated and recorded.

[0179] Furthermore, in order to support the installation of second and subsequent equivalent systems or second and subsequent cameras in the same system, the generation, storage and transmission of installation support feature information can be performed, etc. Thus, the adjustment operations for installing cameras can be significantly reduced.

[0180] Furthermore, by searching for feature information from an image, search parameters such as 505 to 510 can be adjusted and set for position and phase measurement. In other words, when searching based on the search information, the search parameters can be adjusted, allowing robust detection of feature information even when there are variations in the input image, workpiece, and measurement environment. This allows for flexible support of camera installation.

[0181] Furthermore, various types of characteristic information can be analyzed as feature information, such as the size of the characteristic shape, the size of the shadow deviation, and information on the deviation from the search pattern setting area 504 on the image in which the workpiece model is generated. This allows the user's judgment to be supported by evaluating the characteristic information used for camera installation based on the characteristics of the production system's processing (e.g., the background, the conditions for the inflow of workpieces, and the mounting and layout of the equipment) and outputting an evaluation value.

[0182] Then, during second and subsequent camera installation operations in the same system, or in the second and subsequent camera installation operations, the aforementioned installation support feature information and setting data are read or received. Using the thus generated installation support feature information, setting data, captured images acquired by the camera, and image processing, the reproducibility of the camera's installation position and posture is evaluated, and the evaluation results or equivalent image processing results can be output. This allows the user to install the camera while verifying the evaluation results (numerical values, support displays, etc.), i.e., the reproducibility, thereby eliminating the need for repeated trial and error for camera adjustment and system operation verification, and significantly reducing the number of adjustment steps.

[0183] Furthermore, based on the evaluation results of the feature information, the installation support feature information for camera installation support can be selected again, thereby easily selecting the optimal feature information according to the system configuration, the environment and background information of the image processing process, etc. In addition, the user's trial and error can be reduced.

[0184] Furthermore, in the case where a camera or an adjustment stage of a camera includes adjustment mechanisms for pan, tilt, and zoom, the user can try automatic adjustment before manually adjusting the camera position, which can reduce adjustment operations.

[0185] Furthermore, in the case where the camera or the camera adjustment stage includes adjustment mechanisms such as pan, tilt, and zoom, for example, the arrangement of the camera can be adjusted while manually operating such adjustment mechanisms so that the camera is positioned in the correct direction. Figure 10 The various numerical values ​​indicated by 1008 shown in FIG become preferred. This can shorten the manual adjustment process of the user for the position and posture of the camera.

[0186] Furthermore, it can be easily verified that image measurement results equivalent to those of the image processing of the initial (first) production system are acquired in second or subsequent copied production systems or in second and subsequent camera installation operations in the same production system.

[0187] Furthermore, according to the present embodiment, a production system including a gripping unit that grips an object based on a measurement result acquired by the measurement unit of the above-described image processing apparatus can be efficiently replicated.

[0188] Furthermore, a product manufacturing method including a step of gripping an object based on a measurement result obtained by a measurement unit of an image processing device and a step of performing a predetermined process on the object gripped in the gripping step can be efficiently replicated. Consequently, the time required for adjustment operations in second and subsequent production systems obtained by replicating an initial (first) production system, or in second and subsequent camera installation operations in the same production system, can be significantly shortened, and the replication efficiency of the production (product manufacturing) system can be significantly improved.

[0189] Although the present disclosure has been described with reference to typical embodiments, it will be understood that the present disclosure is not limited to the disclosed typical embodiments. The scope of the following claims will be interpreted in the broadest manner to include all of these variations and equivalent structures and functions. In addition, as part or as a whole of the control according to the present embodiment, a computer program that implements the functions of the embodiments described above can be provided to the image processing device via a network or various storage media. The computer (or CPU or MPU, etc.) of the image processing device can then be configured to read and execute the program. In this case, the program and the storage medium storing the program constitute the present disclosure.

[0190] This application claims the benefit of Japanese Patent Application No. 2021-020421, filed February 12, 2021, which is hereby incorporated by reference herein in its entirety.

Claims

1. A system for removing an object from a tray loaded with a plurality of objects, comprising at least one processor configured to: a camera unit configured to acquire an image; an extraction unit configured to extract shape feature information from the entire image including the plurality of objects acquired by the camera unit; a second generating unit configured to generate measurement feature information used to measure the position and posture of the object based on the shape feature information extracted by the extracting unit; a measuring unit configured to measure the position and the posture of the object by searching for the measurement feature information from the image acquired by the camera unit; a control unit configured to perform control so as to perform predetermined processing on the object based on the position and the posture of the object measured by the measurement unit; as well as A first generating unit is configured to generate installation support feature information for supporting installation of a camera unit of another system for taking out objects from a tray loaded with a plurality of objects, based on the shape feature information extracted from the entire image by the extracting unit.

2. The system according to claim 1, wherein the at least one processor is further configured to function as an evaluation unit configured to evaluate the shape feature information, wherein The first generating unit is configured to generate the installation support feature information corresponding to the evaluation performed by the evaluating unit.

3. The system according to claim 2, wherein: The first generating unit is configured to assign a priority to the shape feature information corresponding to the evaluation.

4. The system according to claim 1, wherein: The extraction unit is further configured to extract feature information of the periphery of the object, and the first generation unit is configured to generate the mounting support feature information based on the feature information of the periphery of the object in addition to the shape feature information. 5 . The system according to claim 1 , wherein the predetermined process includes a process of holding the object and transporting the object to a predetermined position.

Images