A robot collaboration device based on multispectral machine vision

Abstract

The application relates to the technical field of automatic assembly, and more specifically to a robot cooperation device based on multispectral machine vision, which comprises an image acquisition module arranged on the upside of a working area; a timing module electrically connected with the image acquisition module and used for sending a first frequency acquisition signal to the image acquisition module; a robot arranged in the working area, the robot being used for driving the end of the robot to move or rotate; an image analysis module, the image acquisition module transmitting the collected image to the image analysis module, the image analysis module positioning the end position of the robot in the image and positioning the position of a staff in the working area in the image; and a distance calculation module electrically connected with the image analysis module and used for calculating the distance between the end of the robot and the staff. The robot cooperation device based on the multispectral machine vision aims to provide a semi-automatic assembly device and solves the problem of low assembly efficiency in the prior art.

Description

Technical Field

[0001] This invention relates to the field of automated assembly technology, and more specifically, to a robot collaboration device based on multispectral machine vision. Background Technology

[0002] The emergence of smart toilets has brought greater convenience to human life and opened up endless possibilities for imagination. The control system of existing smart toilets is often located inside or on the toilet seat, such as the hinge structure that supports the rotation of the toilet, the knob that the user needs to rotate, the microwave sensor that detects the usage status, the flip-top drive plate that drives the toilet seat to rotate, and the waterproof covers on both sides of the toilet seat.

[0003] In existing technologies, toilet seat assembly is typically done manually. For example, a worker first places a toilet seat base to be assembled at their workstation, then places each part to be assembled in its corresponding position, picks up screws and uses a drill to lock them in place, assembles the locked parts, and finally sends the assembled semi-finished toilet seat to the next process. This process is complex and inefficient. Summary of the Invention

[0004] The main objective of this invention is to propose a robot collaborative device based on multispectral machine vision, which aims to provide a semi-automated assembly device and solve the problem of low assembly efficiency in the prior art.

[0005] To address the aforementioned technical issues, a robot collaboration device based on multispectral machine vision is proposed, comprising: an image acquisition module, positioned above the working area, for acquiring images;

[0006] A timing module, electrically connected to the image acquisition module, is used to send a first frequency acquisition signal to the image acquisition module, and the image acquisition module performs image acquisition after receiving the first frequency acquisition signal.

[0007] A robot is set up in a work area, and the robot's activity area is within the work area. The robot is used to move or rotate its own end effector.

[0008] An image analysis module is electrically connected to the image acquisition module. The image acquisition module transmits the acquired images to the image analysis module. The image analysis module is used to locate the end position of the robot in the image, and at the same time, it is used to locate the position of the workers in the working area in the image.

[0009] The distance calculation module, electrically connected to the image analysis module, is used to calculate the distance between the robot's end effector and the worker. When the distance is less than or equal to a first threshold, the distance calculation module sends a warning signal to the robot, and the robot stops moving according to the warning signal. When the distance is greater than the first threshold, the distance calculation module sends a safety signal to the robot, and the robot continues to move.

[0010] In any of the above technical solutions, further comprising:

[0011] A trend prediction module is electrically connected to the image analysis module. The trend prediction module is used to predict the position of the robot's end effector at the next moment based on the position of the robot's end effector as determined by the image analysis module multiple times.

[0012] The trend prediction module is used to predict the location of the staff at the next time point based on the location of the staff located multiple times by the image analysis module.

[0013] The distance calculation module is electrically connected to the trend prediction module. The distance calculation module is used to calculate the distance between the robot end and the worker at the next moment based on the position predicted by the trend prediction module, and to compare the distance at the next moment with the first threshold.

[0014] In any of the above technical solutions, further comprising:

[0015] The path planning module is electrically connected to both the robot and the image analysis module. The collaborative device sends the robot's starting point and ending point to the path planning module. The image analysis module determines the location of obstacles based on the acquired images and sends the results to the path planning module. The path planning module plans the robot's movement path based on the starting point, ending point, and obstacle locations.

[0016] In any of the above technical solutions, further comprising:

[0017] The path planning module is also electrically connected to the distance calculation module and the trend prediction module, respectively. The path planning module is used to adjust the robot's movement path when the distance calculated by the distance calculation module at the next moment is less than or equal to the first threshold.

[0018] In any of the above technical solutions, further, there are two robots, both of which are set in the working area, and the activity areas of the two robots overlap. The image analysis module is used to locate the end positions of the two robots respectively, and the distance calculation module is used to calculate the distance between the end positions of the two robots and the worker respectively. At the same time, the distance calculation module is used to calculate the distance between the end positions of the two robots.

[0019] In any of the above technical solutions, the image acquisition module is further described as a multispectral image acquisition mechanism.

[0020] In any of the above technical solutions, further comprising:

[0021] An angle calculation module is electrically connected to the image analysis module and the trend prediction module, respectively. The angle calculation module obtains the position of the robot end and the position of the worker, and is used to calculate the angle between the movement direction of the robot or worker from time N-2 to time N-1 and the movement direction from time N-1 to time N based on the obtained position. When the angle is greater than a second threshold, the angle calculation module sends a reset signal to the trend prediction module. Time N is the image acquisition time closest to the current time.

[0022] In any of the above technical solutions, the angle calculation module is further electrically connected to the timing module. When the angle calculation module sends a reset signal to the trend prediction module, the timing module sends a second frequency acquisition signal to the image acquisition module. After receiving the second frequency acquisition signal, the image acquisition module performs image acquisition.

[0023] The beneficial effects are:

[0024] 1. The robot collaborative device based on multispectral machine vision of the present invention controls the acquisition frequency of the image acquisition module through the timing module. It can acquire images at a slower frequency when the movement direction changes little and at a faster frequency when the movement direction changes greatly, thereby saving computing power and improving the safety factor of the device.

[0025] 2. The robot collaboration device of the present invention analyzes the images acquired by the image acquisition module through the image analysis module and locates the positions of the robot end effector and the worker. Then, it calculates the distance between the robot end effector and the worker through the distance calculation module, and controls the robot to stop when the distance is less than or equal to a first threshold to ensure the safety of the worker.

[0026] 3. The robot collaboration device of the present invention calculates the distance between the robot end and the worker in advance by setting the trend prediction module, thereby extending the response time of the device and improving its safety. At the same time, in conjunction with the path planning module and the distance calculation module, it can adjust the robot's movement direction in real time during the robot's movement on the optimal path, avoiding the distance between the robot and the worker being too close. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is a flowchart of a robot collaboration device based on multispectral machine vision according to an embodiment of the present invention;

[0029] Figure 2 This is a distribution diagram of the working area and activity area of ​​a robot collaborative device based on multispectral machine vision according to an embodiment of the present invention.

[0030] The annotations in the attached figures are explained as follows:

[0031] 1. Image acquisition module;

[0032] 2. Timer module;

[0033] 3. Robots;

[0034] 4. Image analysis module;

[0035] 5. Distance calculation module;

[0036] 6. Trend prediction module;

[0037] 7. Path planning module;

[0038] 8. Angle Calculation Module. Detailed Implementation

[0039] Hereinafter, exemplary embodiments according to this application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely a part of the embodiments of this application, and not all of the embodiments of this application. It should be understood that this application is not limited to the exemplary embodiments described herein. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without inventive effort are within the scope of protection of this invention.

[0040] It should be noted that, as shown in this application and claims, unless the context clearly indicates otherwise, the words "a," "an," "an," and / or "the" do not specifically refer to the singular and may also include the plural. Generally speaking, the terms "comprising" and "including" only indicate the inclusion of explicitly identified steps and elements, and these steps and elements do not constitute an exclusive list; the method or apparatus may also include other steps or elements.

[0041] If the embodiments of the present invention involve directional indicators (such as up, down, left, right, front, back, etc.), the directional indicators are only used to explain the relative positional relationship and movement of the components in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indicators will also change accordingly.

[0042] In this invention, unless otherwise explicitly specified and limited, the terms "connection," "fixed," etc., should be interpreted broadly. For example, "fixed" can mean a fixed connection, a detachable connection, or an integral part; it can mean a mechanical connection or an electrical connection; it can mean a direct connection or an indirect connection through an intermediate medium; it can mean the internal communication of two components or the interaction between two components, unless otherwise explicitly limited. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0043] Furthermore, if the embodiments of this invention involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the meaning of "and / or" throughout the text includes three parallel solutions; for example, "A and / or B" includes solution A, solution B, or a solution where both A and B are satisfied simultaneously. Furthermore, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by this invention.

[0044] The following embodiments will provide a detailed description of a robot collaboration device based on multispectral machine vision according to this application.

[0045] In this embodiment, as Figure 1 and Figure 2 As shown, the robot collaboration device based on multispectral machine vision includes: an image acquisition module, which is set on the upper side of the working area for acquiring images;

[0046] The timing module is electrically connected to the image acquisition module and is used to send a first frequency acquisition signal to the image acquisition module. After receiving the first frequency acquisition signal, the image acquisition module performs image acquisition.

[0047] The robot is set up within the work area, and its activity area is within the work area. The robot is used to move or rotate its own end effector.

[0048] The image analysis module is electrically connected to the image acquisition module. The image acquisition module transmits the acquired images to the image analysis module. The image analysis module is used to locate the end position of the robot in the image, and at the same time, it is used to locate the position of the workers in the working area in the image.

[0049] The distance calculation module, electrically connected to the image analysis module, is used to calculate the distance between the robot's end effector and the worker. When the distance is less than or equal to a first threshold, the distance calculation module sends a warning signal to the robot, and the robot stops moving based on the warning signal. When the distance is greater than the first threshold, the distance calculation module sends a safety signal to the robot, and the robot continues to move.

[0050] In this technical solution, the image acquisition module is a multispectral camera, fixed to the upper side of the working area. The timing module is a timer that sends acquisition signals to the image acquisition module at a certain frequency. Each time an acquisition signal is sent, the image acquisition module acquires an image, which is then sent to the image analysis module. The robot is a multi-axis robot with a gripper or tooling at its end effector to pick up corresponding targets and move or rotate them. The image analysis module performs image analysis and 3D reconstruction of the items in the working area based on the received images, and performs real-time spatial positioning of the robot's end effector and the worker's position.

[0051] After the image analysis module reconstructs the image, it assigns three-dimensional coordinates to each structure within the work area. It then sends the spatial positions of the robot end effector and the worker to the distance calculation module for spacing calculation. The distance calculation module calculates the spacing between the worker and the robot end effector in real time and sends a warning signal to the robot when the spacing is less than a first threshold, causing the robot to stop. Subsequently, as the worker continues to move, the distance calculation module sends a safety signal to the robot when the spacing exceeds the first threshold, allowing the robot to continue working. This ensures the safety of the worker within the work area.

[0052] Positioning of the worker includes locating their hands, elbows, shoulders, head, and standing position. The distance calculation module needs to calculate the distance between each part of the worker and the robot's end effector. When the distance to any part falls below a first threshold, a warning signal is sent to the robot, and it stops. In some specific technical solutions, the first threshold distance value is set differently for different parts; for example, the threshold for the hand is greater than the threshold for the shoulder joint.

[0053] In this embodiment, it also includes:

[0054] The trend prediction module is electrically connected to the image analysis module. The trend prediction module is used to predict the position of the robot's end effector at the next moment based on the position of the robot's end effector as determined by the multiple positioning by the image analysis module.

[0055] The trend prediction module is used to predict the location of staff at the next time point based on the location of staff identified multiple times by the image analysis module.

[0056] The distance calculation module is electrically connected to the trend prediction module. The distance calculation module is used to calculate the distance between the robot end and the worker at the next moment based on the position predicted by the trend prediction module, and to compare the distance at the next moment with a first threshold.

[0057] In this technical solution, the trend prediction module is electrically connected to the image analysis module and the distance calculation module respectively. The trend prediction module obtains the positions of the robot end effector and the worker at the two closest moments or multiple consecutive moments from the image analysis module. Based on the positions of the robot end effector at the two closest moments or multiple consecutive moments, it predicts the position of the robot end effector at the next moment and provides the corresponding predicted coordinates. At the same time, based on the positions of the worker at the two closest moments or multiple consecutive moments, it predicts the position of the worker at the next moment and provides the corresponding predicted coordinates. The distance calculation module calculates the distance between the robot end effector and the worker at the next moment based on the predicted coordinates, so as to achieve the purpose of calculating the distance one moment in advance and increase the reaction time for controlling the robot to stop when the distance is less than or equal to the first threshold.

[0058] It is worth mentioning that the interval between the two adjacent moments mentioned above can be the time interval between two consecutive image acquisitions by the image acquisition module, or it can be an integer multiple of the time required for two consecutive image acquisitions, such as twice or three times the duration.

[0059] In this embodiment, it also includes:

[0060] The path planning module is electrically connected to both the robot and the image analysis module. The collaborative device sends the robot's starting point and ending point to the path planning module. The image analysis module determines the location of obstacles based on the acquired images and sends the results to the path planning module. The path planning module then plans the robot's movement path based on the starting point, ending point, and obstacle locations.

[0061] In this technical solution, the path planning module is electrically connected to the robot's control system and image analysis module, respectively. This allows the robot's end effector to input the required movement point information, such as the starting point, path point, and ending point, into the collaborative device. Based on this movement point information, the path planning module creates an initial movement path that does not need to avoid external objects. This initial movement path is the fastest and shortest path. Then, the path planning module obtains the spatial range of fixed obstacles from the image analysis module. Based on the initial movement path, the path planning module re-plans the movement path of the robot's end effector at certain intervals to avoid fixed obstacles. After that, the path planning module sends the re-planned movement path to the robot's control system. The robot's control system then controls its various motors to move collaboratively, so that the end effector moves according to the re-planned path.

[0062] The aforementioned obstacles refer to fixed structures within the work area, such as conveyor lines, workbenches, and electrical control boxes.

[0063] In some technical solutions, the robot's end effector is equipped with an industrial camera, which enables the robot's end effector to perform tasks such as picking and assembling at various points in conjunction with its own control system.

[0064] In this embodiment, it also includes:

[0065] The path planning module is also electrically connected to the distance calculation module and the trend prediction module respectively. The path planning module is used to adjust the robot's movement path when the distance calculated by the distance calculation module at the next moment is less than or equal to the first threshold.

[0066] In this technical solution, during the process of the robot control end moving along the replanned path, the distance calculation module synchronously calculates the distance between the robot end and the worker at the next moment based on the position predicted by the trend prediction module, and feeds back to the path planning module when the distance is less than the first threshold at the next moment. The path planning module then adjusts the robot's movement path again based on the predicted position of the worker, thereby reducing the probability of robot stopping and ensuring overall work efficiency.

[0067] In this embodiment, there are two robots, both of which are set up in the work area and their activity areas overlap. The image analysis module is used to locate the end positions of the two robots respectively, and the distance calculation module is used to calculate the distance between the end positions of the two robots and the worker.

[0068] In this technical solution, two robots are positioned side-by-side, with the image acquisition module, i.e., a multispectral camera, fixed above and between the two robots. The activity areas of the two robots overlap in the middle. By using the two robots, different assembly steps can be completed. Simultaneously, the image acquisition module, image analysis module, and distance calculation module monitor the positions of the two robots and their respective positions relative to the worker, thereby improving assembly efficiency.

[0069] In some technical solutions, a worker is positioned to the side of each of the two robots. A distance calculation module calculates the distance between the worker and the robot's end effector, respectively. When the distance between either worker and robot is less than a first threshold, the distance calculation module sends a warning signal to the corresponding robot's control system. Similarly, when the distance between the two robots is less than the first threshold, the distance calculation module sends a warning signal to the control system of the robot on the left.

[0070] In this embodiment, the image acquisition module is a multispectral image acquisition mechanism.

[0071] In this technical solution, by setting the image acquisition module as a multispectral image acquisition mechanism, specifically a multispectral camera, it is convenient for the image analysis module to perform three-dimensional reconstruction of the image. In some technical solutions, the image acquisition module sends light waves of different wavelengths to the part or tooling to be picked up, and the received wavelength response value can be used to calculate whether the surface of the part or tooling is coated with lubricating oil, which is convenient for determining some workpieces that need to be oiled before assembly.

[0072] In this embodiment, it also includes:

[0073] The angle calculation module is electrically connected to the image analysis module and the trend prediction module respectively. The angle calculation module obtains the position of the robot end and the position of the worker, and is used to calculate the angle between the movement direction of the robot or worker from time N-2 to time N-1 and the movement direction from time N-1 to time N based on the obtained position. When the angle is greater than the second threshold, the angle calculation module sends a reset signal to the trend prediction module. Time N is the image acquisition time closest to the current time.

[0074] In this technical solution, the angle calculation module is electrically connected to the image analysis module and the trend prediction module respectively. Taking the robot end position as an example, the angle calculation module obtains the robot end position P1 at the current time (time N), the robot end position P2 at the previous time (time N-1), and the robot end position P3 at the two times before the current time (time N-2) from the image analysis module. It then calculates the motion direction Y1 between positions P1 and P2, and the motion direction Y2 between positions P2 and P3. Finally, it calculates the angle between Y1 and Y2. When the angle is greater than a second threshold, the angle calculation module sends a reset signal to the trend prediction module. After receiving the reset signal, the trend prediction module stops the trend prediction at the current time. This setting can improve the accuracy of the trend prediction module's prediction.

[0075] In some technical solutions, the interval between time N and time N-1 can be the time interval between two consecutive image acquisitions by the image acquisition module, or it can be an integer multiple of the time required for two consecutive image acquisitions, such as twice or three times the duration.

[0076] In this embodiment, the angle calculation module is electrically connected to the timing module. When the angle calculation module sends a reset signal to the trend prediction module, the timing module sends a second frequency acquisition signal to the image acquisition module. After receiving the second frequency acquisition signal, the image acquisition module performs image acquisition.

[0077] In this technical solution, the operating frequency of the second frequency acquisition signal is greater than that of the first frequency acquisition signal. When the angle calculation module sends a reset signal, it is usually when the robot's movement direction or the worker's movement direction changes significantly. At this time, the image acquisition frequency is increased to improve the working efficiency of the image analysis module and the distance calculation module, thereby ensuring the safety of the worker.

[0078] In some technical solutions, the image analysis module performs image analysis and 3D reconstruction using existing technologies.

[0079] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A robot collaborative device based on multispectral machine vision, characterized in that, include: An image acquisition module (1) is set on the upper side of the working area for acquiring images; The timing module (2) is electrically connected to the image acquisition module (1) and is used to send a first frequency acquisition signal to the image acquisition module (1). After receiving the first frequency acquisition signal, the image acquisition module (1) performs image acquisition. A robot (3) is set up in the working area, and the activity area of ​​the robot (3) is within the working area. The robot (3) is used to drive its own end effector to move or rotate. The image analysis module (4) is electrically connected to the image acquisition module (1). The image acquisition module (1) transmits the acquired image to the image analysis module (4). The image analysis module (4) is used to locate the end position of the robot (3) in the image. At the same time, the image analysis module (4) is used to locate the position of the staff in the working area in the image. The distance calculation module (5) is electrically connected to the image analysis module (4) and is used to calculate the distance between the end of the robot (3) and the worker. When the distance is less than or equal to a first threshold, the distance calculation module (5) sends a warning signal to the robot (3), and the robot (3) stops moving according to the warning signal. When the distance is greater than the first threshold, the distance calculation module (5) sends a safety signal to the robot (3), and the robot (3) continues to move. Also includes: The trend prediction module (6) is electrically connected to the image analysis module (4). The trend prediction module (6) is used to predict the position of the end of the robot (3) at the next moment based on the position of the end of the robot (3) located multiple times by the image analysis module (4). The trend prediction module (6) is used to predict the location of the staff at the next time point based on the location of the staff located multiple times by the image analysis module (4). The distance calculation module (5) is electrically connected to the trend prediction module (6). The distance calculation module (5) is used to calculate the distance between the end of the robot (3) and the worker at the next moment based on the position predicted by the trend prediction module (6), and to compare the distance at the next moment with the first threshold. Also includes: Angle calculation module (8) is electrically connected to the image analysis module (4) and the trend prediction module (6) respectively. Angle calculation module (8) obtains the position of the end of the robot (3) and the position of the worker, and is used to calculate the angle between the movement direction of the robot (3) or the worker from time N-2 to time N-1 and the movement direction from time N-1 to time N based on the obtained position. When the angle is greater than the second threshold, angle calculation module (8) sends a reset signal to trend prediction module (6). Time N is the image acquisition time closest to the current time.

2. The robot collaborative device based on multispectral machine vision according to claim 1, characterized in that, Also includes: The path planning module (7) is electrically connected to the robot (3) and the image analysis module (4) respectively. The collaborative device sends the starting point and ending point of the robot (3) to the path planning module (7). The image analysis module (4) determines the location of obstacles based on the collected images and sends it to the path planning module (7). The path planning module (7) plans the movement path of the robot (3) based on the starting point, ending point and obstacle location.

3. The robot collaboration device based on multispectral machine vision according to claim 2, characterized in that, Also includes: The path planning module (7) is also electrically connected to the distance calculation module (5) and the trend prediction module (6), respectively. The path planning module (7) is used to adjust the movement path of the robot (3) when the distance calculated by the distance calculation module (5) at the next moment is less than or equal to the first threshold.

4. The robot collaborative device based on multispectral machine vision according to claim 1, characterized in that, There are two robots (3), both of which are set in the work area and their activity areas overlap. The image analysis module (4) is used to locate the end positions of the two robots (3) respectively. The distance calculation module (5) is used to calculate the distance between the end of the two robots (3) and the worker respectively. At the same time, the distance calculation module (5) is used to calculate the distance between the end of the two robots (3).

5. The robot collaborative device based on multispectral machine vision according to claim 1, characterized in that, The image acquisition module (1) is a multispectral image acquisition mechanism.

6. The robot collaboration device based on multispectral machine vision according to claim 1, characterized in that, The angle calculation module (8) is electrically connected to the timing module (2). When the angle calculation module (8) sends a reset signal to the trend prediction module (6), the timing module (2) sends a second frequency acquisition signal to the image acquisition module (1). After receiving the second frequency acquisition signal, the image acquisition module (1) performs image acquisition.

Images