Multi-sensor fusion robot control method, system, equipment and medium

Abstract

The invention provides a multi-sensor fusion robot control method, system and device and a medium. According to the method, a workpiece image of a target workpiece is obtained through a visual sensor, the workpiece image is sent to an indication controller, position information of a grinding actuator is obtained through a position sensor, and the position information is sent to the indication controller; and the controller determines the material surface attribute in the corresponding polishing range of the polishing actuator according to the workpiece image and the position information, and then determines the polishing mode corresponding to the polishing actuator according to the material surface attribute, so that different polishing strategies are adopted in a targeted manner according to different physical characteristics of different target workpiece material surface areas, and the polishing efficiency is improved. And the grinding quality is ensured.

Description

Technical Field

[0001] The present application relates to data processing technology, and more particularly to a multi-sensor fusion robot control method, system, device and medium. Background Art

[0002] With the rapid development of industrial automation, robotics technology is increasingly being used in various fields. In surface treatment operations such as grinding and polishing, robots have gradually replaced traditional manual operations with their advantages of high precision and high efficiency. However, traditional robotic grinding control systems often rely on information provided by a single sensor, such as relying solely on position sensors to control the motion trajectory of the grinding actuator. This single sensor control method has many limitations and is difficult to adapt to complex and changing workpiece surfaces and material properties.

[0003] Specifically, in actual grinding operations, the target workpiece often contains multiple different material surface areas, which have different physical properties, such as hardness, roughness, etc. The traditional single sensor control method cannot accurately identify the properties of these different material surface areas, so it is difficult to intelligently select the appropriate grinding strategy based on the material properties. This may lead to over-grinding or under-grinding during the grinding process, which not only affects the grinding quality, but may also cause damage to the workpiece. Summary of the invention

[0004] The present application provides a multi-sensor fusion robot control method, system, device and medium, which are used to adopt different polishing strategies according to the different physical properties of the surface areas of different target workpiece materials, thereby ensuring the polishing quality.

[0005] In a first aspect, the present application provides a multi-sensor fusion robot control method, which is applied to a robot control system, wherein the robot control system comprises: a controller, a grinding actuator, a visual sensor, and a position sensor, wherein the controller is communicatively connected with the grinding actuator, the visual sensor, and the position sensor; the method comprises: Acquire a workpiece image of a target workpiece by the visual sensor, and send the workpiece image to instruct the controller, wherein the target workpiece includes at least two material surface areas; Acquire the position information of the grinding actuator through the position sensor, and send the position information to indicate the controller; The controller determines, according to the workpiece image and the position information, a material surface property within a grinding range corresponding to the grinding actuator, wherein the material surface property is associated with a position distribution of different material surface areas on the target workpiece; The controller determines a grinding mode corresponding to the grinding actuator according to the surface properties of the material.

[0006] In the above scheme, the workpiece image of the target workpiece is obtained by the visual sensor, and the image contains the shape information of the surface of the target workpiece. At the same time, the position sensor obtains the position information of the grinding actuator in real time, including its coordinates and posture in space. The controller can identify the material surface properties within the grinding range corresponding to the grinding actuator based on the received workpiece image and position information. This recognition process is based on the material surface area distribution template corresponding to the target workpiece. By extracting the feature area set in the workpiece image and combining the position information to determine the projection area of ​​the grinding range in the workpiece image, the material surface properties within the grinding range can be accurately judged. According to the identified material surface properties, the controller can intelligently select the corresponding grinding mode. Different material surface areas have different physical properties (such as hardness, roughness, etc.), so different grinding strategies are required. Optionally, the grinding mode can be preset, and parameters such as the grinding feed amount can be automatically adjusted according to the material surface properties to ensure the efficiency and accuracy of the grinding process. For example, for the surface area of ​​the material with higher hardness, the controller will select a grinding mode with a smaller feed amount to avoid damage to the workpiece; while for the surface area of ​​the material with greater roughness, the controller will select a grinding mode with a larger feed amount to improve the grinding efficiency.

[0007] Optionally, the controller determines the surface properties of the material within the grinding range corresponding to the grinding actuator according to the workpiece image and the position information, including: The controller extracts a feature region set from the workpiece image according to a material surface region distribution template corresponding to the target workpiece, the feature region set comprising adjacent first feature regions and second feature regions, the first feature regions and the second feature regions respectively corresponding to different material surface regions; The controller determines, according to the position information, a projection area of ​​the grinding range in the workpiece image along a grinding feed direction of the grinding actuator; The controller determines the surface property of the material according to the relative position relationship between the projection area and the first characteristic area and the second characteristic area.

[0008] In the above scheme, the controller first extracts a feature area set from the workpiece image according to the material surface area distribution template corresponding to the target workpiece. This set includes adjacent first feature areas and second feature areas, which correspond to different material surface areas respectively. By using the predefined material surface area distribution template, the controller can quickly and accurately identify key feature areas from complex workpiece images. This process significantly improves the efficiency and accuracy of feature extraction and provides a solid foundation for subsequent material surface property judgment. Then, the controller determines the grinding range of the grinding actuator on the workpiece surface according to the information provided by the position sensor, and calculates the projection area of ​​the range in the workpiece image along the grinding feed direction. This process ensures the accurate definition of the grinding range and avoids grinding deviation caused by inaccurate position information. At the same time, the determination of the projection area also provides a spatial reference frame for subsequent material surface property judgment. The controller judges the material surface properties within the grinding range according to the relative position relationship between the projection area and the first feature area and the second feature area. This judgment process involves the analysis of relative position relationships, such as whether the projection area completely covers a feature area, whether it intersects with multiple feature areas, etc. By analyzing the relative position relationship, the controller can accurately determine the surface properties of the material within the grinding range. This judgment result directly affects the subsequent grinding mode selection, thereby ensuring the efficiency and accuracy of the grinding operation. In practical applications, the target workpiece often contains multiple different material surface areas. By identifying the relative position relationship between the projection area and multiple feature areas, the robot can formulate personalized grinding strategies based on the characteristics of different material surface areas, thereby improving the quality and efficiency of the grinding operation.

[0009] Optionally, determining the surface property of the material according to the relative position relationship between the projection area and the first characteristic area and the second characteristic area includes: If the controller determines that the projection area only includes a portion of the first feature area, the controller determines that the material surface attribute is a first material surface attribute, wherein the first material surface attribute corresponds to a first grinding mode, and the grinding feed amount in the first grinding mode is associated with a first material characteristic parameter corresponding to the first material surface attribute; If the controller determines that the projection area only includes a portion of the second feature area, the controller determines that the material surface attribute is a second material surface attribute, wherein the second material surface attribute corresponds to a second grinding mode, and the grinding feed amount in the second grinding mode is associated with a second material characteristic parameter corresponding to the second material surface attribute; If the controller determines that the projection area includes a partial area in the first feature area and a partial area in the second feature area, the controller determines that the material surface property is a third material surface property, wherein the third material surface property corresponds to a third grinding mode, and the grinding feed amount in the third grinding mode is associated with the first material characteristic parameter and the second material characteristic parameter.

[0010] In the above scheme, the controller accurately divides the material surface properties within the grinding range into three situations by analyzing the relative position relationship between the projection area and the first characteristic area and the second characteristic area: only a part of the first characteristic area, only a part of the second characteristic area, and a part of the first and second characteristic areas at the same time. This division method ensures the accuracy of the material surface property judgment and provides a reliable basis for the subsequent selection of grinding modes. By identifying the material type within the grinding range, the robot can adopt appropriate grinding strategies for different material properties, thereby improving the grinding quality and efficiency. Then, according to the identified material surface properties, the controller can flexibly switch to the corresponding grinding mode, such as the first grinding mode, the second grinding mode and the third grinding mode. Each grinding mode is optimized for specific material properties to ensure the optimization of the grinding effect. Intelligent grinding mode selection enables the robot to automatically adjust the grinding parameters, such as grinding feed amount, grinding speed, etc., according to the processing requirements of different materials. This adaptive ability significantly improves the processing flexibility and efficiency of the robot and reduces the dependence on manual intervention. In addition, the grinding feed amount in each grinding mode is associated with the corresponding material characteristic parameters. For example, for materials with higher hardness, the controller will select a smaller grinding feed to avoid damage to the workpiece; for materials with greater roughness, it will select a larger grinding feed to improve grinding efficiency. By accurately controlling the grinding feed, the robot can improve processing efficiency while ensuring processing quality. This precise grinding parameter control not only extends the service life of the grinding tool, but also reduces production costs.

[0011] Optionally, the controller determines the grinding mode corresponding to the grinding actuator according to the surface properties of the material, including: If the controller determines that the material surface property corresponds to the first grinding mode, the controller determines a first grinding feed amount according to a first material surface hardness value and a first material surface roughness value in the first material characteristic parameter; If the controller determines that the material surface property corresponds to the second grinding mode, the controller determines the second grinding feed amount according to the second material surface elastic modulus and the second material surface roughness value in the second material characteristic parameter, wherein the first material surface hardness value is greater than the second material surface hardness value in the second material characteristic parameter; If the controller determines that the material surface property corresponds to the third grinding mode, the controller determines a third grinding feed amount according to the first grinding feed amount and the second grinding feed amount.

[0012] In the above scheme, when the controller determines that the material surface property corresponds to the first grinding mode, it determines the first grinding feed amount according to the first material surface hardness value and the first material surface roughness value in the first material characteristic parameter. By calculating the first grinding feed amount, the robot can maximize the grinding efficiency while ensuring the processing quality. This precise control strategy based on material characteristic parameters avoids problems such as workpiece damage or low processing efficiency caused by excessive or insufficient feed amount.

[0013] For the second grinding mode, the controller determines the second grinding feed amount according to the second material surface elastic modulus and the second material surface roughness value in the second material characteristic parameters. Since the hardness value of the second material is usually lower than that of the first material, the controller will take this difference into account when calculating the feed amount to ensure the stability and safety of the grinding process. Reasonable setting of the second grinding feed amount not only improves the robot's adaptability to materials of different hardness, but also further improves the overall efficiency and quality of the grinding operation.

[0014] When the controller determines that the material surface properties correspond to the third grinding mode, that is, when the grinding range contains both the first and second material surface areas, it determines the third grinding feed based on the first and second grinding feeds, a process that usually involves a weighted average of the two feeds. By comprehensively considering the characteristic parameters of the first and second materials, the controller can more accurately determine the third grinding feed, thereby ensuring that the grinding operation on the complex material surface can achieve the desired effect. This comprehensive consideration strategy improves the robot's adaptability to complex workpiece surfaces and expands its application range.

[0015] By calculating the grinding feed and selecting the appropriate grinding mode, the above solution can significantly improve grinding efficiency, which can not only shorten the processing cycle, but also reduce energy consumption and equipment wear. Improving grinding efficiency not only helps to improve the production capacity and market competitiveness of enterprises, but also helps to reduce production costs and energy consumption. In addition, by reasonably controlling the grinding feed and avoiding damage to the workpiece, the service life of grinding tools and equipment can be extended. This helps to reduce the frequency and cost of equipment replacement and maintenance.

[0016] Optionally, the robot control system further comprises a force sensor, and the controller is in communication connection with the force sensor; correspondingly, in the first grinding feed amount And the second grinding feed After determining the third grinding feed amount, it also includes: Acquiring feedback force information of the polishing actuator when polishing the projection area through the force sensor, and sending the feedback force information to the controller; The controller adjusts the third grinding feed amount according to the feedback force information and the preset calibrated feedback force to determine the grinding feed amount at the next moment.

[0017] In the above scheme, a force sensor is integrated into the robot control system, which can monitor the force of the grinding actuator during the grinding process in real time. Through the communication connection with the controller, the force sensor sends the acquired feedback force information to the controller accurately and timely. The integration of the force sensor enables the robot to perceive the force changes in the grinding process in real time, providing key data support for the subsequent adjustment of the grinding feed amount. This real-time feedback mechanism significantly improves the accuracy and controllability of the robot's grinding operation. The controller dynamically adjusts the third grinding feed amount according to the received feedback force information and the preset calibrated feedback force. This process can be achieved by using proportional control, integral control or differential control methods to achieve smooth transition and precise control of the grinding feed amount. By dynamically adjusting the grinding feed amount, the robot can better adapt to the changes in different material properties, workpiece surface quality and grinding tool wear state. This adaptive ability significantly improves the flexibility and stability of the robot's grinding operation. During the dynamic adjustment process, the controller calculates the adjustment amount of the grinding feed amount based on the difference between the feedback force information and the calibrated feedback force. The adjustment is then applied to the current grinding feed to determine the grinding feed at the next moment. This process ensures precise control of the grinding feed, avoiding problems such as damage to the workpiece or low processing efficiency caused by excessive or insufficient feed. By precisely controlling the grinding feed, the robot can maximize grinding efficiency while ensuring processing quality. This precise control strategy helps achieve efficient and high-quality robotic grinding operations. In addition, by controlling the grinding feed and avoiding excessive pressure on the workpiece, the robot can reduce wear and damage to the grinding tool. This helps extend the service life of the grinding tool and reduce production and maintenance costs.

[0018] Optionally, the controller extracts a feature area set from the workpiece image according to a material surface area distribution template corresponding to the target workpiece, including: The controller extracts a target workpiece contour and a target workpiece feature point set feature from the workpiece image, wherein the target workpiece feature point set feature is a preset marking reference point on the target workpiece; The controller generates a target workpiece original grid according to the target workpiece contour and the target workpiece feature point set features, wherein each target workpiece feature point in the target workpiece feature point set is a control point of the target workpiece original grid; The controller adjusts the positions of control points corresponding to the characteristic points of each target workpiece on the original grid of the target workpiece according to a set of reference control points on the workpiece reference grid corresponding to the workpiece reference image, so as to determine an updated grid of the target workpiece; The controller determines the material surface properties of each grid unit in the target workpiece update grid according to the material surface area distribution template corresponding to the workpiece reference image, wherein the material surface area distribution template is an image template with material surface properties marked in each grid on the workpiece reference grid; The controller extracts a feature region set from the workpiece image according to a material surface property of each grid unit in the target workpiece update grid.

[0019] In the above scheme, the controller first uses an image processing algorithm to extract the contour of the target workpiece and the preset marked reference points (i.e., the target workpiece feature point set features) from the workpiece image. These feature points are key reference points for subsequent mesh generation and are essential to ensure the accuracy and adaptability of the mesh. Based on the extracted target workpiece contour and feature point set, the controller generates the target workpiece original mesh. These feature points, as control points of the mesh, determine the shape and layout of the mesh. The generation of the original mesh provides a basic framework for subsequent position adjustment and material surface property determination. By constructing a mesh that highly matches the shape of the target workpiece, the controller can analyze the material properties of the workpiece surface more carefully, thereby formulating a more accurate grinding strategy. The controller further uses the reference control point set on the workpiece reference mesh corresponding to the workpiece reference image to adjust the position of the control points corresponding to each target workpiece feature point on the target workpiece original mesh. This process ensures a high degree of consistency between the mesh and the actual workpiece surface. Through the position adjustment, the controller generates an updated target workpiece mesh that more accurately reflects the actual shape and material distribution of the workpiece surface. This provides a more accurate basis for subsequent material surface property determination and feature area extraction. Then, the controller determines the material surface properties of each grid cell in the target workpiece update grid based on the material surface area distribution template corresponding to the workpiece reference image (the template calibrates the material surface properties in each grid on the workpiece reference grid). Through this process, the controller can accurately identify the material properties of different areas on the workpiece surface, such as hardness, roughness, etc. This provides a key basis for formulating grinding strategies for different material properties, which helps to improve the quality and efficiency of grinding. Finally, the controller extracts a set of feature regions from the workpiece image based on the material surface properties of each grid cell in the target workpiece update grid. These feature regions represent areas on the workpiece surface with different material properties and are an important basis for formulating grinding strategies. By accurately extracting the set of feature regions, the controller can provide more accurate grinding instructions to the grinding actuator to ensure the efficiency and accuracy of the grinding process. This helps to improve the quality and efficiency of the entire grinding operation and reduce production and maintenance costs.

[0020] Optionally, before the controller adjusts the positions of the control points corresponding to the target workpiece feature points on the original grid of the target workpiece according to the reference control point set on the workpiece reference grid corresponding to the workpiece reference image, the controller further includes: The controller determines a workpiece reference image from a workpiece calibration image set according to a visual sensing vector, wherein the workpiece reference image is a workpiece calibration image in the workpiece calibration image set with a minimum angle between a calibrated visual sensing vector and the visual sensing vector, and the visual sensing vector is an optical axis direction when the visual sensor acquires the workpiece image.

[0021] In the above scheme, the visual sensing vector is the direction of the optical axis of the visual sensor when acquiring the workpiece image, which represents the spatial direction of the sensor's viewing angle. The controller can accurately obtain the direction of the visual sensing vector through the built-in angle sensor or image processing algorithm of the visual sensor. Then, the workpiece calibration image set in the above scheme includes calibration images of multiple workpieces taken at different viewing angles and different lighting conditions, and each image is calibrated with the corresponding visual sensing vector. The controller compares the angle between the current visual sensing vector and the visual sensing vector in each calibration image in the workpiece calibration image set, and selects the workpiece calibration image with the smallest angle as the reference image. For greater accuracy, the calibration image with the same shooting position and angle can be selected as the workpiece reference image. By constructing and utilizing the workpiece calibration image set, the controller can adapt to the image processing requirements of different workpieces and different viewing angles, which improves the adaptability and flexibility of the robot system and enables it to operate stably in a variety of working environments. Selecting the workpiece calibration image with the smallest angle as the reference image can minimize the impact of the viewing angle difference on the image processing results, thereby improving the accuracy of feature point matching and grid generation. In addition, selecting the workpiece calibration image with the smallest angle as the reference image can ensure the consistency of feature point matching and grid generation in subsequent processing steps, providing a reliable basis for the subsequent grinding strategy formulation. Moreover, based on the reference control point set on the workpiece reference image, the controller adjusts the position of the control points corresponding to each target workpiece feature point on the original grid of the target workpiece. The adjusted control points generate an updated grid for the target workpiece, which more accurately reflects the actual shape and material distribution of the workpiece surface, and can also compensate for the perspective error or the workpiece position error of the target workpiece, thereby further improving the pertinence and reliability of subsequent grinding.

[0022] In a second aspect, the present application provides a robot control system, comprising: a controller, a grinding actuator, a visual sensor, and a position sensor, wherein the controller is communicatively connected with the grinding actuator, the visual sensor, and the position sensor; Acquire a workpiece image of a target workpiece by the visual sensor, and send the workpiece image to instruct the controller, wherein the target workpiece includes at least two material surface areas; Acquire the position information of the grinding actuator through the position sensor, and send the position information to indicate the controller; The controller determines, according to the workpiece image and the position information, a material surface property within a grinding range corresponding to the grinding actuator, wherein the material surface property is associated with a position distribution of different material surface areas on the target workpiece; The controller determines a grinding mode corresponding to the grinding actuator according to the surface properties of the material.

[0023] Optionally, the controller determines the surface properties of the material within the grinding range corresponding to the grinding actuator according to the workpiece image and the position information, including: The controller extracts a feature region set from the workpiece image according to a material surface region distribution template corresponding to the target workpiece, the feature region set comprising adjacent first feature regions and second feature regions, the first feature regions and the second feature regions respectively corresponding to different material surface regions; The controller determines, according to the position information, a projection area of ​​the grinding range in the workpiece image along a grinding feed direction of the grinding actuator; The controller determines the surface property of the material according to the relative position relationship between the projection area and the first characteristic area and the second characteristic area.

[0024] Optionally, determining the surface property of the material according to the relative position relationship between the projection area and the first characteristic area and the second characteristic area includes: If the controller determines that the projection area only includes a portion of the first feature area, the controller determines that the material surface attribute is a first material surface attribute, wherein the first material surface attribute corresponds to a first grinding mode, and the grinding feed amount in the first grinding mode is associated with a first material characteristic parameter corresponding to the first material surface attribute; If the controller determines that the projection area only includes a portion of the second feature area, the controller determines that the material surface attribute is a second material surface attribute, wherein the second material surface attribute corresponds to a second grinding mode, and the grinding feed amount in the second grinding mode is associated with a second material characteristic parameter corresponding to the second material surface attribute; If the controller determines that the projection area includes a partial area in the first feature area and a partial area in the second feature area, the controller determines that the material surface property is a third material surface property, wherein the third material surface property corresponds to a third grinding mode, and the grinding feed amount in the third grinding mode is associated with the first material characteristic parameter and the second material characteristic parameter.

[0025] Optionally, the controller determines the grinding mode corresponding to the grinding actuator according to the surface properties of the material, including: If the controller determines that the material surface property corresponds to the first grinding mode, the controller determines a first grinding feed amount according to a first material surface hardness value and a first material surface roughness value in the first material characteristic parameter; If the controller determines that the material surface property corresponds to the second grinding mode, the controller determines the second grinding feed amount according to the second material surface elastic modulus and the second material surface roughness value in the second material characteristic parameter, wherein the first material surface hardness value is greater than the second material surface hardness value in the second material characteristic parameter; If the controller determines that the material surface property corresponds to the third grinding mode, the controller determines a third grinding feed amount according to the first grinding feed amount and the second grinding feed amount.

[0026] Optionally, the robot control system further includes a force sensor, and the controller is in communication connection with the force sensor; correspondingly, after determining the third grinding feed amount according to the first grinding feed amount and the second grinding feed amount, it further includes: Acquiring feedback force information of the polishing actuator when polishing the projection area through the force sensor, and sending the feedback force information to the controller; The controller adjusts the third grinding feed amount according to the feedback force information and the preset calibrated feedback force to determine the grinding feed amount at the next moment.

[0027] Optionally, the controller extracts a feature area set from the workpiece image according to a material surface area distribution template corresponding to the target workpiece, including: The controller extracts a target workpiece contour and a target workpiece feature point set feature from the workpiece image, wherein the target workpiece feature point set feature is a preset marking reference point on the target workpiece; The controller generates a target workpiece original grid according to the target workpiece contour and the target workpiece feature point set features, wherein each target workpiece feature point in the target workpiece feature point set is a control point of the target workpiece original grid; The controller adjusts the positions of control points corresponding to the characteristic points of each target workpiece on the original grid of the target workpiece according to a set of reference control points on the workpiece reference grid corresponding to the workpiece reference image, so as to determine an updated grid of the target workpiece; The controller determines the material surface properties of each grid unit in the target workpiece update grid according to the material surface area distribution template corresponding to the workpiece reference image, wherein the material surface area distribution template is an image template with material surface properties marked in each grid on the workpiece reference grid; The controller extracts a feature region set from the workpiece image according to a material surface property of each grid unit in the target workpiece update grid.

[0028] Optionally, before the controller adjusts the positions of the control points corresponding to the target workpiece feature points on the original grid of the target workpiece according to the reference control point set on the workpiece reference grid corresponding to the workpiece reference image, the controller further includes: The controller determines a workpiece reference image from a workpiece calibration image set according to a visual sensing vector, wherein the workpiece reference image is a workpiece calibration image in the workpiece calibration image set with a minimum angle between a calibrated visual sensing vector and the visual sensing vector, and the visual sensing vector is an optical axis direction when the visual sensor acquires the workpiece image.

[0029] In a third aspect, the present application provides an electronic device, including: processor; and, A memory, configured to store executable instructions of the processor; The processor is configured to perform any possible method described in the first aspect by executing the executable instructions.

[0030] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement any possible method described in the first aspect.

[0031] The multi-sensor fusion robot control method, system, equipment and medium provided in the present application obtain the workpiece image of the target workpiece through the visual sensor, and send the workpiece image to the instruction controller, obtain the position information of the grinding actuator through the position sensor, and send the position information to the instruction controller, then, the controller determines the material surface properties within the grinding range corresponding to the grinding actuator according to the workpiece image and the position information, and then determines the grinding mode corresponding to the grinding actuator according to the material surface properties, so as to adopt different grinding strategies according to the different physical properties of the surface areas of different target workpiece materials, thereby ensuring the grinding quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0033] Figure 1 It is a flowchart of a robot control method of multi-sensor fusion shown in the present application according to an example embodiment; Figure 2 is a flowchart of a robot control method of multi-sensor fusion according to another exemplary embodiment of the present application; Figure 3 is a schematic diagram of a robot control system according to an exemplary embodiment of the present application; Figure 4 It is a schematic diagram of the structure of an electronic device according to an exemplary embodiment of the present application.

[0034] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0035] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0036] In order to solve the above problems, the embodiments provided in this application achieve comprehensive perception of the surface properties of the target workpiece material by fusing information from visual sensors, position sensors (and optional force sensors). The visual sensor obtains the workpiece image, which contains the shape information of the target workpiece surface; the position sensor obtains the position information of the grinding actuator; and the force sensor (if used) provides feedback force information during the grinding process. Through multi-sensor information fusion, the controller can accurately identify the surface properties of the material within the grinding range corresponding to the grinding actuator, and then intelligently select the appropriate grinding strategy.

[0037] The controller determines the material surface properties within the grinding range, such as hardness, roughness, etc., based on the workpiece image and position information. Based on the material surface properties, the controller selects a suitable grinding mode from the preset grinding mode library and sets the corresponding grinding parameters (such as grinding feed amount, grinding speed, etc.). Through this grinding mode selection method based on material surface properties, the present application implements personalized grinding strategies for different material surface areas and ensures the grinding quality.

[0038] During the grinding process, the controller monitors the feedback force information of the grinding actuator in real time through the force sensor (if used). According to the comparison result between the feedback force information and the preset calibrated feedback force, the controller dynamically adjusts the grinding parameters (such as the grinding feed amount) to ensure the stability and safety of the grinding process.

[0039] The visual sensor acquires the workpiece image of the target workpiece in real time and sends the image data to the controller. The position sensor detects the position information (including three-dimensional coordinates and posture) of the grinding actuator in real time and sends the position information to the controller. The controller extracts a set of feature areas from the workpiece image according to the material surface area distribution template corresponding to the target workpiece. These feature areas correspond to different material surface areas. The controller determines the projection area of ​​the grinding range in the workpiece image along the grinding feed direction of the grinding actuator according to the position information. By analyzing the relative position relationship between the projection area and the feature area set, the controller determines the material surface properties within the grinding range.

[0040] According to the determined material surface properties, the controller selects the appropriate grinding mode from the preset grinding mode library. According to the material surface characteristic parameters (such as hardness, roughness, etc.) and the grinding mode, the controller sets the corresponding grinding parameters (such as grinding feed, grinding speed, etc.). During the grinding process, the force sensor monitors the feedback force information of the grinding actuator in real time and sends the information to the controller. The controller dynamically adjusts the grinding parameters (such as grinding feed) based on the comparison result of the feedback force information and the preset calibrated feedback force to ensure the stability and safety of the grinding process.

[0041] The controller sends the selected grinding mode and grinding parameters to the grinding actuator, controlling it to perform grinding operations according to the predetermined trajectory and parameters. During the grinding process, the controller monitors the motion state of the grinding actuator and the changes on the workpiece surface in real time through visual sensors and position sensors (and optional force sensors), and adjusts the grinding strategy in time to ensure the grinding quality.

[0042] Figure 1 FIG. 1 is a flow chart of a multi-sensor fusion robot control method according to an exemplary embodiment of the present application. Figure 1 As shown, the method provided in this embodiment includes: S101, acquiring a workpiece image of a target workpiece through a visual sensor.

[0043] In this step, a workpiece image of the target workpiece may be acquired through a visual sensor, and the workpiece image may be sent to an instruction controller, wherein the target workpiece includes at least two material surface areas. For example, the target workpiece may be a workpiece that is assembled first and then polished, or a workpiece formed by splicing or welding different materials, or a workpiece generated by the same material with different parts subjected to different heat treatments.

[0044] Specifically, when the robot is working, the visual sensor is activated to shoot the target workpiece and obtain image information of its surface. The visual sensor sends the obtained workpiece image data to the controller, which receives and stores the image data.

[0045] S102, obtaining position information of the grinding actuator through a position sensor.

[0046] In this step, the position information of the grinding actuator is obtained through the position sensor, and the position information is sent to the instruction controller.

[0047] Specifically, the position sensor detects the position information of the grinding actuator in real time, including its coordinates and posture in three-dimensional space. The position sensor sends the acquired position information data to the controller, which receives and processes the information to accurately control the movement of the grinding actuator.

[0048] S103: The controller determines the surface properties of the material within the grinding range corresponding to the grinding actuator according to the workpiece image and position information.

[0049] In this step, the controller determines the material surface properties within the grinding range corresponding to the grinding actuator according to the workpiece image and position information, and the material surface properties are associated with the position distribution of different material surface areas on the target workpiece.

[0050] Specifically, the controller first uses an image processing algorithm to process the received workpiece image and extract key information such as the contour and feature points of the target workpiece. The controller matches the extracted features with the features in the template according to the material surface area distribution template corresponding to the target workpiece to determine the material surface properties within the grinding range corresponding to the grinding actuator. Through feature matching, the controller can identify the position distribution of different material surface areas within the grinding range. Based on this position distribution information, the controller further determines the material surface properties within the grinding range, such as hardness, roughness, etc.

[0051] S104: The controller determines a grinding mode corresponding to the grinding actuator according to the surface properties of the material.

[0052] Specifically, the controller can select a suitable grinding mode from a preset grinding mode library according to the determined material surface properties. Different material surface properties correspond to different grinding modes. For example, for a material surface with higher hardness, the controller may select a grinding mode with a smaller feed amount and a slower grinding speed to avoid damage to the workpiece.

[0053] After determining the grinding mode, the controller also needs to set the corresponding grinding parameters according to the specific material surface properties, such as grinding feed, grinding speed, grinding force, etc. The setting of these parameters is crucial to ensure the grinding quality and efficiency.

[0054] The controller sends the selected grinding mode and grinding parameters to the grinding actuator, and controls it to grind according to the predetermined trajectory and parameters. Optionally, during the grinding process, the controller monitors the motion state of the grinding actuator and the changes on the workpiece surface in real time through visual sensors and position sensors. If deviations or abnormalities are found during the grinding process, the controller will adjust the grinding parameters or motion trajectory in time to ensure the smooth progress of the grinding task.

[0055] In this embodiment, a workpiece image of the target workpiece is acquired through a visual sensor, and the workpiece image is sent to an instruction controller, and the position information of the grinding actuator is acquired through a position sensor, and the position information is sent to an instruction controller. Then, the controller determines the material surface properties within the grinding range corresponding to the grinding actuator based on the workpiece image and the position information, and then determines the grinding mode corresponding to the grinding actuator based on the material surface properties, so as to adopt different grinding strategies according to the different physical properties of the surface areas of different target workpiece materials, thereby ensuring the grinding quality.

[0056] Figure 2 FIG. 1 is a flow chart of a multi-sensor fusion robot control method according to another exemplary embodiment of the present application. Figure 2 As shown, the multi-sensor fusion robot control method provided in this embodiment includes: S201, acquiring a workpiece image of a target workpiece through a visual sensor.

[0057] In this step, a workpiece image of the target workpiece may be acquired through a visual sensor, and the workpiece image may be sent to an instruction controller, wherein the target workpiece includes at least two material surface areas. For example, the target workpiece may be a workpiece that is assembled first and then polished, or a workpiece formed by splicing or welding different materials, or a workpiece generated by the same material with different parts subjected to different heat treatments.

[0058] Specifically, when the robot is working, the visual sensor is activated to shoot the target workpiece and obtain image information of its surface. The visual sensor sends the obtained workpiece image data to the controller, which receives and stores the image data.

[0059] S202, obtaining position information of the grinding actuator through a position sensor.

[0060] In this step, the position information of the grinding actuator is obtained through the position sensor, and the position information is sent to the instruction controller.

[0061] Specifically, the position sensor detects the position information of the grinding actuator in real time, including its coordinates and posture in three-dimensional space. The position sensor sends the acquired position information data to the controller, which receives and processes the information to accurately control the movement of the grinding actuator.

[0062] S203: The controller extracts a feature region set from the workpiece image according to a material surface region distribution template corresponding to the target workpiece.

[0063] In this step, the controller extracts a feature area set from the workpiece image according to the material surface area distribution template corresponding to the target workpiece, and the feature area set includes adjacent first feature areas and second feature areas, and the first feature areas and the second feature areas correspond to different material surface areas respectively.

[0064] Specifically, before the robot polishing operation, the material surface area of ​​the target workpiece is analyzed in advance to construct a material surface area distribution template. The template records in detail the position, shape, size and other information of the different material surfaces on the workpiece. For example, for a workpiece composed of metal and plastic, the template will clearly mark the specific position and boundary of the metal area and the plastic area.

[0065] The image of the workpiece is captured in real time by a high-definition camera mounted on the robot arm. The image undergoes preprocessing steps such as denoising and contrast enhancement to improve the accuracy of subsequent feature extraction. The controller uses an image recognition algorithm to extract a set of feature regions from the preprocessed workpiece image based on the material surface area distribution template. These feature regions correspond to different material surface areas defined in the template. For example, if the template indicates that the upper half of the workpiece is a metal area and the lower half is a plastic area, the controller will extract the feature region sets of these two areas from the image respectively. During the extraction process, the controller can pay special attention to the adjacent first feature area and the second feature area, because they represent the junction of different material surfaces, which is crucial for the adjustment of grinding parameters.

[0066] In a possible implementation, the controller may extract the target workpiece contour and the target workpiece feature point set features from the workpiece image, wherein the target workpiece feature point set features are preset marking reference points on the target workpiece.

[0067] Among them, the image of the target workpiece can be captured in real time by a visual sensor (such as a high-definition camera) installed on the robot arm. The image should be clear, unobstructed, and contain sufficient information for subsequent processing. The captured workpiece image is preprocessed, including denoising, contrast enhancement, grayscale and other operations to improve image quality and feature extraction accuracy. The contour of the target workpiece is extracted from the preprocessed image using an edge detection algorithm (such as Canny edge detection). The contour should accurately reflect the shape and boundary of the workpiece. According to the preset marked reference points (such as specific marks, holes on the workpiece, etc.), a set of feature points of the target workpiece is extracted from the image. These feature points will serve as control points for subsequent mesh generation.

[0068] The controller generates a target workpiece original mesh according to the target workpiece contour and the target workpiece feature point set features, wherein each target workpiece feature point in the target workpiece feature point set is a control point of the target workpiece original mesh.

[0069] Among them, according to the extracted target workpiece contour and feature point set, an original mesh covering the entire workpiece is initialized. The resolution of the mesh (i.e. the size of the mesh unit) should be adjusted according to the complexity of the workpiece and the grinding accuracy requirements. Each target workpiece feature point in the feature point set is used as the control point of the original mesh. These control points will be used for subsequent mesh adjustment and deformation.

[0070] The controller adjusts the positions of the control points corresponding to the characteristic points of each target workpiece on the original grid of the target workpiece according to the reference control point set on the workpiece reference grid corresponding to the workpiece reference image, so as to determine the updated grid of the target workpiece.

[0071] Among them, a reference image similar to the current workpiece image is selected from the workpiece calibration image set. The reference image should contain the corresponding workpiece reference grid and reference control point set. According to the reference control point set on the workpiece reference grid, the control points corresponding to each target workpiece feature point on the original grid of the target workpiece are adjusted in position. This step can use a mesh deformation algorithm (such as affine transformation, non-rigid transformation, etc.) to ensure that the updated mesh can accurately reflect the shape and material surface distribution of the target workpiece. After the position adjustment, an updated mesh of the target workpiece is generated. This mesh will be used for subsequent material surface property determination and feature area extraction.

[0072] The controller determines the material surface properties of each grid unit in the target workpiece update grid according to the material surface area distribution template corresponding to the workpiece reference image, wherein the material surface area distribution template is an image template with material surface properties marked in each grid on the workpiece reference grid.

[0073] The material surface properties of each grid cell in the updated grid are determined using the material surface area distribution template corresponding to the workpiece reference image. The template should have material surface properties (such as metal, plastic, etc.) marked in each grid on the workpiece reference grid. The properties in the material surface area distribution template are mapped to the corresponding grid cells in the updated grid. This step ensures that each grid cell has clear material surface properties.

[0074] The controller extracts a set of feature regions from the workpiece image according to the material surface properties of each grid cell in the target workpiece update grid.

[0075] Among them, the corresponding feature region set can be identified from the workpiece image according to the material surface properties of each grid unit in the updated grid. These feature regions correspond to different material surface regions and have clear boundaries and shapes. Then, the image segmentation algorithm (such as threshold segmentation, region growing, etc.) is used to extract the feature region set from the workpiece image. These feature regions will be used for subsequent grinding parameter adjustment and grinding actuator control.

[0076] In the above scheme, the controller first uses an image processing algorithm to extract the contour of the target workpiece and the preset marked reference points (i.e., the target workpiece feature point set features) from the workpiece image. These feature points are key reference points for subsequent mesh generation and are essential to ensure the accuracy and adaptability of the mesh. Based on the extracted target workpiece contour and feature point set, the controller generates the target workpiece original mesh. These feature points, as control points of the mesh, determine the shape and layout of the mesh. The generation of the original mesh provides a basic framework for subsequent position adjustment and material surface property determination. By constructing a mesh that highly matches the shape of the target workpiece, the controller can analyze the material properties of the workpiece surface more carefully, thereby formulating a more accurate grinding strategy. The controller further uses the reference control point set on the workpiece reference mesh corresponding to the workpiece reference image to adjust the position of the control points corresponding to each target workpiece feature point on the target workpiece original mesh. This process ensures a high degree of consistency between the mesh and the actual workpiece surface. Through the position adjustment, the controller generates an updated target workpiece mesh that more accurately reflects the actual shape and material distribution of the workpiece surface. This provides a more accurate basis for subsequent material surface property determination and feature area extraction. Then, the controller determines the material surface properties of each grid cell in the target workpiece update grid based on the material surface area distribution template corresponding to the workpiece reference image (the template calibrates the material surface properties in each grid on the workpiece reference grid). Through this process, the controller can accurately identify the material properties of different areas on the workpiece surface, such as hardness, roughness, etc. This provides a key basis for formulating grinding strategies for different material properties, which helps to improve the quality and efficiency of grinding. Finally, the controller extracts a set of feature regions from the workpiece image based on the material surface properties of each grid cell in the target workpiece update grid. These feature regions represent areas on the workpiece surface with different material properties and are an important basis for formulating grinding strategies. By accurately extracting the set of feature regions, the controller can provide more accurate grinding instructions to the grinding actuator to ensure the efficiency and accuracy of the grinding process. This helps to improve the quality and efficiency of the entire grinding operation and reduce production and maintenance costs.

[0077] Furthermore, before the controller adjusts the positions of the control points corresponding to the target workpiece feature points on the original grid of the target workpiece according to the reference control point set on the workpiece reference grid corresponding to the workpiece reference image, the controller can also determine the workpiece reference image from the workpiece calibration image set according to the visual sensing vector, wherein the workpiece reference image is the workpiece calibration image with the smallest angle between the calibrated visual sensing vector and the visual sensing vector in the workpiece calibration image set, and the visual sensing vector is the direction of the optical axis when the visual sensor acquires the workpiece image.

[0078] The controller calculates the visual sensing vector according to the installation position and parameters of the visual sensor. The visual sensing vector is the direction of the optical axis when the visual sensor acquires the image of the workpiece, usually expressed in the form of a three-dimensional vector.

[0079] Before the robot performs grinding operations, multiple calibration images of the workpiece under different viewing angles and lighting conditions are acquired in advance. These images should cover all key areas of the workpiece and contain sufficient feature information for subsequent matching and reference. The acquired calibration images of the workpiece are stored in the robot's storage system, and an index is created for each image for fast retrieval and matching.

[0080] The controller traverses each image in the workpiece calibration image set and calculates the angle between the calibrated visual sensor vector and the current visual sensor vector in each image. The smaller the angle, the smaller the difference in viewing angle between the two images and the higher the matching degree.

[0081] The controller selects the workpiece calibration image with the smallest angle as the workpiece reference image. This image will be used in the subsequent mesh adjustment and material surface property determination process. The angle calculation can be achieved through the vector dot product formula.

[0082] The corresponding workpiece reference grid and reference control point set are extracted from the determined workpiece reference image. These control points will be used for subsequent adjustment of the original grid of the target workpiece. According to the reference control point set on the workpiece reference grid, the control points corresponding to each target workpiece feature point on the original grid of the target workpiece are adjusted in position.

[0083] In the above scheme, the visual sensing vector is the direction of the optical axis of the visual sensor when acquiring the workpiece image, which represents the spatial direction of the sensor's viewing angle. The controller can accurately obtain the direction of the visual sensing vector through the built-in angle sensor or image processing algorithm of the visual sensor. Then, the workpiece calibration image set in the above scheme includes calibration images of multiple workpieces taken at different viewing angles and different lighting conditions, and each image is calibrated with the corresponding visual sensing vector. The controller compares the angle between the current visual sensing vector and the visual sensing vector in each calibration image in the workpiece calibration image set, and selects the workpiece calibration image with the smallest angle as the reference image. For greater accuracy, the calibration image with the same shooting position and angle can be selected as the workpiece reference image. By constructing and utilizing the workpiece calibration image set, the controller can adapt to the image processing requirements of different workpieces and different viewing angles, which improves the adaptability and flexibility of the robot system and enables it to operate stably in a variety of working environments. Selecting the workpiece calibration image with the smallest angle as the reference image can minimize the impact of the viewing angle difference on the image processing results, thereby improving the accuracy of feature point matching and grid generation. In addition, selecting the workpiece calibration image with the smallest angle as the reference image can ensure the consistency of feature point matching and grid generation in subsequent processing steps, providing a reliable basis for the subsequent grinding strategy formulation. Moreover, based on the reference control point set on the workpiece reference image, the controller adjusts the position of the control points corresponding to each target workpiece feature point on the original grid of the target workpiece. The adjusted control points generate an updated grid for the target workpiece, which more accurately reflects the actual shape and material distribution of the workpiece surface, and can also compensate for the perspective error or the workpiece position error of the target workpiece, thereby further improving the pertinence and reliability of subsequent grinding.

[0084] S204. The controller determines a projection area of ​​the grinding range in the workpiece image along the grinding feed direction of the grinding actuator according to the position information.

[0085] Specifically, the robot obtains the position information of the grinding actuator (such as the grinding head) in real time through built-in sensors (such as encoders, laser rangefinders, etc.), including its three-dimensional coordinates and posture on the workpiece surface. The controller calculates the projection area of ​​the grinding range in the workpiece image based on the position information of the grinding actuator and the grinding feed direction (i.e., the moving direction of the grinding head relative to the workpiece surface). This can be achieved by mapping the grinding range in three-dimensional space onto a two-dimensional image plane, and taking into account the influence of the grinding feed direction on the shape and position of the projection area.

[0086] S205: The controller determines the surface properties of the material according to the relative position relationship between the projection area and the first characteristic area and the second characteristic area.

[0087] Specifically, the controller conducts an in-depth analysis of the relative position relationship between the projection area and the first feature area and the second feature area in the feature area set. This includes determining whether the projection area is completely located within a feature area, whether it crosses the boundary of two or more feature areas, and the specific location of the crossing boundary. Based on the analysis results of the relative position relationship, the controller determines the surface properties of the materials involved in the grinding range. For example, if the projection area is completely located within the metal feature area, the controller determines that the surface property of the material within the grinding range is metal; if the projection area crosses the boundary of the metal and plastic feature areas, the controller needs to intelligently select or adjust the grinding parameters to adapt to different material surface properties based on the specific location of the crossing (such as close to the metal area or the plastic area) and the accuracy requirements of the grinding operation.

[0088] In one possible implementation, if the controller determines that the projection area only includes a portion of the first feature area, it determines that the material surface attribute is the first material surface attribute, wherein the first material surface attribute corresponds to the first grinding mode, and the grinding feed amount in the first grinding mode is associated with the first material characteristic parameter corresponding to the first material surface attribute.

[0089] Specifically, if the controller determines that the material surface property corresponds to the first grinding mode, the controller uses Formula 1 and determines the first grinding feed amount according to the first material characteristic parameter. , where Formula 1 is:

[0090] in, To preset the first material correlation coefficient, is the surface hardness value of the first material, is the surface roughness value of the first material, is the exponential factor affecting the weight of hardness and roughness, To polish the actuator shape correlation coefficient, To polish the initial contact angle between the actuator and the target workpiece, is the first fluctuation correction term.

[0091] In the above scheme, when the controller determines that the material surface property corresponds to the first grinding mode, the first grinding feed amount is calculated using Formula 1. The formula comprehensively considers the surface hardness value of the first material, the surface roughness value of the first material, and the exponential factor of the influence weight of hardness and roughness, wherein the exponential factor of the influence weight of hardness and roughness can be calibrated through prior experiments. This comprehensive consideration ensures that the grinding feed amount can adapt to the material surfaces of different hardness and roughness, thereby avoiding insufficient or excessive grinding due to differences in material properties. Formula 1 also introduces the grinding actuator shape correlation coefficient and the initial contact angle between the grinding actuator and the target workpiece. The introduction of these parameters makes the calculation of the grinding feed amount more consistent with the physical conditions in the actual grinding process, further improving the grinding accuracy. In addition, a first fluctuation correction term is set to compensate for the grinding feed amount deviation caused by system errors, environmental changes and other factors. By introducing this correction term, the system can adaptively adjust the grinding feed amount to a certain extent, thereby improving the robustness and stability of the system. The first fluctuation correction term may also be obtained by calibration through prior experiments, or by establishing an error correction fluctuation table and then mapping and determining it according to the first material characteristic parameter.

[0092] If the controller determines that the projection area only includes a partial area of ​​the second feature area, it determines that the material surface attribute is the second material surface attribute, wherein the second material surface attribute corresponds to the second grinding mode, and the grinding feed amount in the second grinding mode is associated with the second material characteristic parameter corresponding to the second material surface attribute.

[0093] Specifically, if the controller determines that the material surface property corresponds to the second grinding mode, the controller uses Formula 2 and determines the second grinding feed amount according to the second material characteristic parameter. , where the surface hardness value of the first material is greater than the surface hardness value of the second material in the second material characteristic parameter, and Formula 2 is:

[0094] in, To preset the second material correlation coefficient, is the surface elastic modulus of the second material, is the surface roughness value of the second material, is the exponential factor of the elastic modulus and roughness influence weight, To polish the actuator shape correlation coefficient, To polish the initial contact angle between the actuator and the target workpiece, is the second fluctuation correction term.

[0095] When the controller determines that the material surface property corresponds to the second grinding mode, the second grinding feed amount is calculated using Formula 2. Unlike Formula 1, Formula 2 takes into account the combined influence of the elastic modulus and roughness value of the second material surface, and weights it by the exponential factor m of the elastic modulus and roughness influence weight. This processing method makes the calculation of the grinding feed amount more suitable for material surfaces with a large elastic modulus, thereby ensuring the consistency and stability of the grinding effect. Formula 2 has certain similarities with Formula 1 in structure, such as considering the influence of the grinding actuator shape correlation coefficient and the initial contact angle, and introducing a fluctuation correction term. However, Formula 2 achieves precise control of the grinding feed amount in the second grinding mode by introducing a new material characteristic parameter, the elastic modulus, which reflects the flexibility and adaptability of the technology. It is worth noting that in the above scheme, the first material is a harder metal material, such as iron, aluminum, copper, metal alloys, etc., and the second material is a softer organic material, such as polycarbonate, acrylonitrile-butadiene-styrene, polypropylene, and polyphenylene sulfide. In addition, considering the difference in material properties between the first material and the second material, the hardness factor is mainly considered in Formula 1, while the elastic modulus factor is mainly considered in Formula 2.

[0096] If the controller determines that the projection area includes a partial area in the first feature area and a partial area in the second feature area, it determines that the material surface property is a third material surface property, wherein the third material surface property corresponds to a third grinding mode, and the grinding feed amount in the third grinding mode is associated with the first material characteristic parameter and the second material characteristic parameter.

[0097] Specifically, if the controller determines that the material surface property corresponds to the third grinding mode, it uses Formula 3 and according to the first grinding feed amount And the second grinding feed Determine the third grinding feed , where Formula 3 is:

[0098] in, is the first weight coefficient, is the second weight coefficient.

[0099] In the above scheme, when the controller determines that the material surface property corresponds to the third grinding mode, that is, when the grinding range contains multiple material surface properties, it will use Formula 3 to calculate the third grinding feed, which combines the first grinding feed and the second grinding feed by weighted summation, and adjusts according to the first weight coefficient and the second weight coefficient. This processing method enables the system to accurately control the grinding feed to meet the grinding requirements of mixed material surfaces. Among them, the first weight coefficient and the second weight coefficient can be dynamically adjusted according to factors such as the area ratio and hardness difference of different material surfaces in the grinding range. This dynamic adjustment mechanism enables the system to more flexibly respond to grinding tasks of different complexities, thereby improving the adaptability and flexibility of the system.

[0100] It can be seen that the above scheme not only improves the accuracy and efficiency of robot grinding operations, but also enhances the adaptability and flexibility of the system, enabling it to cope with the grinding needs of workpieces of different shapes and surfaces. At the same time, through dynamic adjustment of weight coefficients and other mechanisms, the system can also more flexibly cope with grinding tasks of different complexities, further improving the practicality and reliability of the system.

[0101] It is further worth noting that the third grinding feed amount is determined according to the first material characteristic parameter and the second material characteristic parameter. Previously, the controller may also use Formula 4 to determine the first weight coefficient and the second weight coefficient according to the relative position relationship between the projection area and the first feature area and the second feature area, wherein Formula 4 is:

[0102] in, is the area of ​​the projection region, is the area of ​​the first characteristic region included in the projection area, is the area of ​​the second characteristic region included in the projection area, is the surface hardness value of the second material, is the surface elastic modulus of the first material.

[0103] In the above scheme, when determining the weight coefficient, the controller first considers the relative position relationship between the projection area and the first feature area and the second feature area. The projection area is the mapping of the grinding actuator in the workpiece image in the grinding feed direction, while the feature area represents different material surface properties. By calculating the area ratio of different feature areas in the projection area, the controller can more accurately evaluate the distribution of different materials in the current grinding area.

[0104] The characteristic parameters such as the surface hardness value of the first material, the surface hardness value of the second material, the surface elastic modulus of the first material and the surface elastic modulus of the second material are introduced into Formula 4. The introduction of these parameters enables the calculation of the weight coefficient to take into account not only the area ratio of the projection area, but also the physical properties of different materials, thereby more comprehensively reflecting the material distribution in the grinding area.

[0105] By dynamically calculating the weight coefficient, the controller can more accurately control the third grinding feed. In the third grinding mode, the grinding area contains a variety of material surface properties, so it is necessary to comprehensively consider the grinding requirements of different materials. The introduction of the weight coefficient enables the system to dynamically adjust the weights of the first grinding feed and the second grinding feed according to the material distribution in the current grinding area, so as to obtain a more appropriate third grinding feed.

[0106] The dynamic adaptive calculation of the weight coefficients mentioned above helps to improve the accuracy and efficiency of the robot grinding operation. By more accurately controlling the grinding feed, the system can avoid insufficient or excessive grinding, thereby reducing damage to the workpiece and improving the grinding quality. At the same time, the dynamic adjustment of the weight coefficients can also make the system more flexible in dealing with grinding tasks of different complexities and improve grinding efficiency.

[0107] The above scheme is not only applicable to specific workpieces and material combinations, but also has good scalability. By adjusting the parameters in Formula 4 and the calculation method of the weight coefficient, the system can adapt to the grinding needs of workpieces with different material surfaces. In addition, it can also be combined with other robot control technologies, such as force control grinding technology, visual tracking technology, etc. By comprehensively using a variety of technical means, the system can further improve the accuracy and efficiency of grinding operations and realize more intelligent and automated robot grinding operations.

[0108] Furthermore, the robot control system further includes a force sensor, and the controller is in communication connection with the force sensor. Correspondingly, after determining the first weight coefficient and the second weight coefficient according to the relative position relationship between the projection area and the first feature area and the second feature area, it may also include: Acquire feedback force information of the grinding actuator when grinding the projection area through the force sensor, and send the feedback force information to the controller; The controller uses Formula 5 and corrects the first weight coefficient and the second weight coefficient according to the feedback force information and the preset calibration feedback force. Formula 5 is:

[0109] in, is the first weight coefficient after update, is the updated second weight coefficient, is the feedback force value in the feedback force information, It is the feedback force value in the preset calibrated feedback force; The controller uses Formula 6 and adjusts the first grinding feed amount according to And the second grinding feed Determine the third grinding feed amount at the next moment, where Formula 6 is:

[0110] in, This is the third grinding feed for the next moment.

[0111] In the above scheme, by introducing force sensors into the robot control system, the system can perceive the feedback force information of the grinding actuator when grinding the projection area in real time. The introduction of this information provides the controller with direct data support for force changes during the grinding process, allowing the system to more accurately evaluate the grinding effect and adjust the grinding strategy. The addition of force sensors enhances the perception ability of the robot control system. Compared with visual sensors and position sensors, force sensors can provide more direct and accurate grinding force information, thereby helping the controller to more accurately judge the material removal and grinding effect during the grinding process. Weight coefficient adjustment based on feedback force: The controller uses Formula 5 to make real-time corrections to the first weight coefficient and the second weight coefficient according to the feedback force information and the preset calibration feedback force. This correction mechanism enables the system to dynamically adjust the weight coefficient according to the current grinding force, so as to more accurately reflect the material distribution and grinding requirements in the current grinding area. By correcting the weight coefficient in real time, the system can more accurately calculate the third grinding feed. In particular, when the hardness or roughness of the material changes during the grinding process, the dynamic adjustment of the weight coefficient can ensure that the system adjusts the grinding strategy in time to avoid insufficient or excessive grinding. After the weight coefficient is corrected in real time, the controller uses Formula 6 to calculate the third grinding feed at the next moment. This calculation process comprehensively considers the corrected weight coefficient and the first and second grinding feeds to ensure the accuracy and adaptability of the grinding feeds. By dynamically adjusting the grinding feed, the system can more accurately control the amount of material removed during the grinding process, thereby improving the grinding accuracy and stability. In particular, when processing workpieces with complex shapes or a combination of multiple materials, this dynamic adjustment mechanism can ensure that the system always maintains the best grinding state.

[0112] Through the above scheme, real-time correction and dynamic adjustment of weight coefficient and grinding feed amount are achieved. This mechanism significantly improves the accuracy, stability and adaptability of robot grinding operations, enabling it to more accurately respond to the grinding needs of workpieces with different shapes and different material surfaces. At the same time, the technical effect also has good scalability and practicality, and can meet more diverse application scenarios and robot grinding needs.

[0113] S206: The controller determines the grinding mode corresponding to the grinding actuator according to the surface properties of the material.

[0114] Specifically, the controller can select a suitable grinding mode from a preset grinding mode library according to the determined material surface properties. Different material surface properties correspond to different grinding modes. For example, for a material surface with higher hardness, the controller may select a grinding mode with a smaller feed amount and a slower grinding speed to avoid damage to the workpiece.

[0115] Figure 3 FIG. 1 is a schematic diagram of a robot control system according to an exemplary embodiment of the present application. Figure 3 As shown, the robot control system 300 provided in this embodiment includes: a controller 310, a grinding actuator 320, a visual sensor 330 and a position sensor 340, and the controller 310 is communicatively connected with the grinding actuator 320, the visual sensor 330 and the position sensor 340; Acquire a workpiece image of a target workpiece through the visual sensor 330, and send the workpiece image to instruct the controller 310, wherein the target workpiece includes at least two material surface areas; Acquire the position information of the grinding actuator 320 through the position sensor 340, and send the position information to indicate the controller 310; The controller 310 determines the material surface properties within the corresponding grinding range of the grinding actuator 320 according to the workpiece image and the position information, wherein the material surface properties are associated with the position distribution of different material surface areas on the target workpiece; The controller 310 determines the grinding mode corresponding to the grinding actuator 320 according to the surface properties of the material.

[0116] Optionally, the controller 310 determines the surface properties of the material within the grinding range corresponding to the grinding actuator 320 according to the workpiece image and the position information, including: The controller 310 extracts a feature region set from the workpiece image according to a material surface region distribution template corresponding to the target workpiece, wherein the feature region set includes adjacent first feature regions and second feature regions, and the first feature regions and the second feature regions correspond to different material surface regions respectively; The controller 310 determines a projection area of ​​the grinding range in the workpiece image along the grinding feed direction of the grinding actuator 320 according to the position information; The controller 310 determines the surface property of the material according to the relative position relationship between the projection area and the first characteristic area and the second characteristic area.

[0117] Optionally, determining the surface property of the material according to the relative position relationship between the projection area and the first characteristic area and the second characteristic area includes: If the controller 310 determines that the projection area only includes a portion of the first feature area, then the controller 310 determines that the material surface attribute is a first material surface attribute, wherein the first material surface attribute corresponds to a first grinding mode, and the grinding feed amount in the first grinding mode is associated with a first material characteristic parameter corresponding to the first material surface attribute; If the controller 310 determines that the projection area only includes a portion of the second feature area, the controller 310 determines that the material surface attribute is a second material surface attribute, wherein the second material surface attribute corresponds to a second grinding mode, and the grinding feed amount in the second grinding mode is associated with a second material characteristic parameter corresponding to the second material surface attribute; If the controller 310 determines that the projection area includes a partial area in the first feature area and a partial area in the second feature area, it determines that the material surface attribute is a third material surface attribute, wherein the third material surface attribute corresponds to a third grinding mode, and the grinding feed amount in the third grinding mode is associated with the first material characteristic parameter and the second material characteristic parameter.

[0118] Optionally, the controller 310 determines the grinding mode corresponding to the grinding actuator 320 according to the surface properties of the material, including: If the controller 310 determines that the material surface property corresponds to the first grinding mode, the controller 310 determines a first grinding feed amount according to a first material surface hardness value and a first material surface roughness value in the first material characteristic parameter; If the controller 310 determines that the material surface property corresponds to the second grinding mode, the controller 310 determines the second grinding feed amount according to the second material surface elastic modulus and the second material surface roughness value in the second material characteristic parameter, wherein the first material surface hardness value is greater than the second material surface hardness value in the second material characteristic parameter; If the controller 310 determines that the material surface property corresponds to the third grinding mode, the controller 310 determines a third grinding feed amount according to the first grinding feed amount and the second grinding feed amount.

[0119] Optionally, the robot control system further includes a force sensor 350, and the controller 310 is in communication connection with the force sensor 350; correspondingly, after determining the third grinding feed amount according to the first grinding feed amount and the second grinding feed amount, it further includes: Acquiring feedback force information of the grinding actuator 320 when grinding the projection area through the force sensor 350, and sending the feedback force information to the controller 310; The controller 310 adjusts the third grinding feed amount according to the feedback force information and the preset calibrated feedback force to determine the grinding feed amount at the next moment.

[0120] Optionally, the controller 310 extracts a feature area set from the workpiece image according to a material surface area distribution template corresponding to the target workpiece, including: The controller 310 extracts a target workpiece contour and a target workpiece feature point set feature from the workpiece image, wherein the target workpiece feature point set feature is a preset marking reference point on the target workpiece; The controller 310 generates a target workpiece original grid according to the target workpiece contour and the target workpiece feature point set features, wherein each target workpiece feature point in the target workpiece feature point set is a control point of the target workpiece original grid; The controller 310 adjusts the positions of the control points corresponding to the feature points of each target workpiece on the original grid of the target workpiece according to the reference control point set on the workpiece reference grid corresponding to the workpiece reference image, so as to determine the updated grid of the target workpiece; The controller 310 determines the material surface properties of each grid unit in the target workpiece update grid according to the material surface area distribution template corresponding to the workpiece reference image, wherein the material surface area distribution template is an image template with material surface properties marked in each grid on the workpiece reference grid; The controller 310 extracts a feature region set from the workpiece image according to the material surface properties of each grid unit in the target workpiece update grid.

[0121] Optionally, before the controller 310 adjusts the position of the control points corresponding to each target workpiece feature point on the target workpiece original grid according to the reference control point set on the workpiece reference grid corresponding to the workpiece reference image, the process further includes: The controller 310 determines a workpiece reference image from a workpiece calibration image set based on a visual sensing vector, wherein the workpiece reference image is a workpiece calibration image in the workpiece calibration image set with a minimum angle between a calibrated visual sensing vector and the visual sensing vector, and the visual sensing vector is the direction of an optical axis when the visual sensor 330 acquires the workpiece image.

[0122] Figure 4 is a schematic diagram of the structure of an electronic device according to an exemplary embodiment of the present application. Figure 4 As shown, an electronic device 400 provided in this embodiment includes: a processor 401 and a memory 402; wherein: The memory 402 is used to store computer programs, and the memory may also be a flash memory.

[0123] The processor 401 is used to execute the execution instructions stored in the memory to implement each step in the above method. For details, please refer to the relevant description in the above method embodiment.

[0124] Optionally, the memory 402 may be independent or integrated with the processor 401 .

[0125] When the memory 402 is a device independent of the processor 401, the electronic device 400 may further include: The bus 403 is used to connect the memory 402 and the processor 401 .

[0126] This embodiment further provides a readable storage medium, in which a computer program is stored. When at least one processor of an electronic device executes the computer program, the electronic device executes the methods provided in the above-mentioned various implementation modes.

[0127] This embodiment also provides a program product, which includes a computer program stored in a readable storage medium. At least one processor of the electronic device can read the computer program from the readable storage medium, and at least one processor executes the computer program so that the electronic device implements the methods provided in the above various embodiments.

[0128] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the claims.

[0129] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A multi-sensor fusion robot control method, characterized in that: Applied to a robot control system, the robot control system includes: a controller, a grinding actuator, a visual sensor and a position sensor, the controller is communicatively connected with the grinding actuator, the visual sensor and the position sensor; the method includes: Acquire a workpiece image of a target workpiece by the visual sensor, and send the workpiece image to instruct the controller, wherein the target workpiece includes at least two material surface areas; Acquire the position information of the grinding actuator through the position sensor, and send the position information to indicate the controller; The controller determines, according to the workpiece image and the position information, a material surface property within a grinding range corresponding to the grinding actuator, wherein the material surface property is associated with a position distribution of different material surface areas on the target workpiece; The controller determines a grinding mode corresponding to the grinding actuator according to the surface properties of the material.

2. The multi-sensor fusion robot control method according to claim 1, characterized in that: The controller determines the surface properties of the material within the grinding range corresponding to the grinding actuator according to the workpiece image and the position information, including: The controller extracts a feature region set from the workpiece image according to a material surface region distribution template corresponding to the target workpiece, the feature region set comprising adjacent first feature regions and second feature regions, the first feature regions and the second feature regions respectively corresponding to different material surface regions; The controller determines, according to the position information, a projection area of ​​the grinding range in the workpiece image along a grinding feed direction of the grinding actuator; The controller determines the surface property of the material according to the relative position relationship between the projection area and the first characteristic area and the second characteristic area.

3. The multi-sensor fusion robot control method according to claim 2, characterized in that: The determining the surface property of the material according to the relative position relationship between the projection area and the first characteristic area and the second characteristic area includes: If the controller determines that the projection area only includes a portion of the first feature area, the controller determines that the material surface attribute is a first material surface attribute, wherein the first material surface attribute corresponds to a first grinding mode, and the grinding feed amount in the first grinding mode is associated with a first material characteristic parameter corresponding to the first material surface attribute; If the controller determines that the projection area only includes a portion of the second feature area, the controller determines that the material surface attribute is a second material surface attribute, wherein the second material surface attribute corresponds to a second grinding mode, and the grinding feed amount in the second grinding mode is associated with a second material characteristic parameter corresponding to the second material surface attribute; If the controller determines that the projection area includes a partial area in the first feature area and a partial area in the second feature area, the controller determines that the material surface property is a third material surface property, wherein the third material surface property corresponds to a third grinding mode, and the grinding feed amount in the third grinding mode is associated with the first material characteristic parameter and the second material characteristic parameter.

4. The multi-sensor fusion robot control method according to claim 3, characterized in that: The controller determines the grinding mode corresponding to the grinding actuator according to the surface properties of the material, including: If the controller determines that the material surface property corresponds to the first grinding mode, the controller determines a first grinding feed amount according to a first material surface hardness value and a first material surface roughness value in the first material characteristic parameter; If the controller determines that the material surface property corresponds to the second grinding mode, the controller determines the second grinding feed amount according to the second material surface elastic modulus and the second material surface roughness value in the second material characteristic parameter, wherein the first material surface hardness value is greater than the second material surface hardness value in the second material characteristic parameter; If the controller determines that the material surface property corresponds to the third grinding mode, the controller determines a third grinding feed amount according to the first grinding feed amount and the second grinding feed amount.

5. The multi-sensor fusion robot control method according to claim 4, characterized in that: The robot control system further includes a force sensor, and the controller is in communication with the force sensor; correspondingly, after determining the third grinding feed amount according to the first grinding feed amount and the second grinding feed amount, it further includes: Acquiring feedback force information of the polishing actuator when polishing the projection area through the force sensor, and sending the feedback force information to the controller; The controller adjusts the third grinding feed amount according to the feedback force information and the preset calibrated feedback force to determine the grinding feed amount at the next moment.

6. The multi-sensor fusion robot control method according to any one of claims 2 to 5, characterized in that: The controller extracts a feature area set from the workpiece image according to a material surface area distribution template corresponding to the target workpiece, including: The controller extracts a target workpiece contour and a target workpiece feature point set feature from the workpiece image, wherein the target workpiece feature point set feature is a preset marking reference point on the target workpiece; The controller generates a target workpiece original grid according to the target workpiece contour and the target workpiece feature point set features, wherein each target workpiece feature point in the target workpiece feature point set is a control point of the target workpiece original grid; The controller adjusts the positions of control points corresponding to the characteristic points of each target workpiece on the original grid of the target workpiece according to a set of reference control points on the workpiece reference grid corresponding to the workpiece reference image, so as to determine an updated grid of the target workpiece; The controller determines the material surface properties of each grid unit in the target workpiece update grid according to the material surface area distribution template corresponding to the workpiece reference image, wherein the material surface area distribution template is an image template with material surface properties marked in each grid on the workpiece reference grid; The controller extracts a feature region set from the workpiece image according to a material surface property of each grid unit in the target workpiece update grid.

7. The multi-sensor fusion robot control method according to claim 5, characterized in that: Before the controller adjusts the positions of the control points corresponding to the target workpiece feature points on the original grid of the target workpiece according to the reference control point set on the workpiece reference grid corresponding to the workpiece reference image, the method further includes: The controller determines a workpiece reference image from a workpiece calibration image set according to a visual sensing vector, wherein the workpiece reference image is a workpiece calibration image in the workpiece calibration image set with a minimum angle between a calibrated visual sensing vector and the visual sensing vector, and the visual sensing vector is an optical axis direction when the visual sensor acquires the workpiece image.

8. A robot control system, characterized in that: include: A controller, a grinding actuator, a visual sensor and a position sensor, wherein the controller is communicatively connected with the grinding actuator, the visual sensor and the position sensor; Acquire a workpiece image of a target workpiece by the visual sensor, and send the workpiece image to instruct the controller, wherein the target workpiece includes at least two material surface areas; Acquire the position information of the grinding actuator through the position sensor, and send the position information to indicate the controller; The controller determines, according to the workpiece image and the position information, a material surface property within a grinding range corresponding to the grinding actuator, wherein the material surface property is associated with a position distribution of different material surface areas on the target workpiece; The controller determines a grinding mode corresponding to the grinding actuator according to the surface properties of the material.

9. An electronic device, characterized in that: include: processor; as well as, A memory, configured to store executable instructions of the processor; The processor is configured to perform the method of any one of claims 1 to 7 by executing the executable instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 7 when executed by a processor.

Images