Robot control method, apparatus, device, and storage medium

Abstract

The present disclosure provides a robot control method, device, equipment and storage medium, and belongs to the technical field of robots. The method comprises: acquiring a plurality of reference image data of a target object, each reference image data corresponding to different pose information; acquiring target image data of the target object through a camera; determining reference image data matched with the target image data based on the matching degree between the target image data and each reference image data; determining pose information corresponding to the target image data based on the pose information corresponding to the reference image data matched with the target image data; and controlling the robot to work based on the pose information corresponding to the target image data. According to the present disclosure, the robot can directly determine the current spatial position and attitude of the target object according to the pose information, and can more accurately perform assembly or detection operations on the target object, thereby improving the accuracy of robot operation.

Description

Technical Field

[0001] This disclosure relates to the field of robotics, and in particular to a robot control method, apparatus, device, and storage medium. Background Technology

[0002] The use of conveyor belts on industrial production lines has greatly accelerated the assembly speed of products. However, with the increase in labor costs and the requirements for product assembly precision, the original manual assembly work can no longer meet the current needs. Therefore, industrial robots have emerged.

[0003] The typical control method for robots is as follows: an encoder is fixed on the shaft of the conveyor belt, and the conveying distance of the conveyor belt is determined in real time by the encoder. When a product is placed on the conveyor belt, an initial position of the product is determined. Based on the initial position of the product and the real-time conveying distance sent by the encoder, the robot can determine the position of the product so that the robot can perform intelligent operations such as assembly or inspection of the product.

[0004] However, the above control methods are only applicable when the product does not change position or posture on the conveyor belt. If the conveyor belt is unstable or the staff accidentally touches the product on the conveyor belt, the position and angle of the product will change, resulting in a large error when the robot operates, which reduces the accuracy of the robot operation. Summary of the Invention

[0005] This disclosure provides a robot control method that can improve the accuracy of robot operation. The technical solution is as follows:

[0006] Firstly, a robot control method is provided, the method comprising:

[0007] Acquire multiple reference image data of the target object, each reference image data corresponding to different pose information;

[0008] The target image data of the target object is acquired through a camera;

[0009] Based on the matching degree between the target image data and each reference image data, reference image data that matches the target image data is determined;

[0010] Based on the pose information corresponding to the reference image data that matches the target image data, the pose information corresponding to the target image data is determined;

[0011] The robot is controlled to work based on the pose information corresponding to the target image data.

[0012] In a possible implementation, the determining the reference image data matched with the target image data based on the matching degree between the target image data and each reference image data comprises:

[0013] The reference image data is binarized to obtain reference binarized image data;

[0014] The target image data is binarized to obtain target binarized image data;

[0015] For each reference binarized image data, the matching degree between the reference image data corresponding to the reference binarized image data and the target image data is determined based on the gray values of the plurality of reference pixel points in the reference binarized image data and the gray values of the plurality of target pixel points in the target binarized image data;

[0016] The reference image data with the largest matching degree between the target image data is determined as the reference image data matched with the target image data.

[0017] In a possible implementation, the method further comprises:

[0018] The target region corresponding to the target object in the target binarized image data is determined based on the target binarized image data and a visual tracking algorithm;

[0019] The determining the matching degree between the reference image data corresponding to the reference binarized image data and the target image data based on the gray values of the plurality of reference pixel points in the reference binarized image data and the gray values of the plurality of target pixel points in the target binarized image data comprises:

[0020] The matching degree between the reference image data corresponding to the reference binarized image data and the target image data is determined based on the gray values of the plurality of reference pixel points in the reference binarized image data and the gray values of the plurality of target pixel points in the target region.

[0021] In a possible implementation, the determining the matching degree between the reference image data corresponding to the reference binarized image data and the target image data based on the gray values of the plurality of reference pixel points in the reference binarized image data and the gray values of the plurality of target pixel points in the target region comprises:

[0022] A plurality of frame selection regions are framed in the reference binarized image data a plurality of times based on a preset window, wherein the size of the preset window is the same as the size of the target region;

[0023] For each of the multiple selected regions, determine a region matching degree between the selected region and the target region based on gray values of multiple reference pixels in the selected region and gray values of multiple target pixels in the target region.

[0024] Determine a matching degree between the reference image data corresponding to the reference binary image data and the target image data based on the region matching degrees between the multiple selected regions and the target region.

[0025] In a possible implementation, the determining of the matching degree between the reference image data corresponding to the reference binary image data and the target image data based on the region matching degrees between the multiple selected regions and the target region comprises:

[0026] Determine the region matching degree with the maximum value among the region matching degrees corresponding to the multiple selected regions as the matching degree between the reference image data corresponding to the reference binary image data and the target image data.

[0027] In a possible implementation, the determining of the matching degree between the reference image data corresponding to the reference binary image data and the target image data based on the region matching degrees between the multiple selected regions and the target region comprises:

[0028] Determine the selected region with the maximum value of the corresponding region matching degree as the selected region to be adjusted.

[0029] Determine a matching region in the reference binary image data that matches the target region based on an ICP (Iterative Closest Point) algorithm, the selected region to be adjusted, the reference binary image data, and the target region, wherein a region matching degree between the matching region and the target region is greater than or equal to a region matching degree between the selected region to be adjusted and the target region.

[0030] Determine the region matching degree between the matching region and the target region as the matching degree between the reference image data corresponding to the reference binary image data and the target image data.

[0031] In a possible implementation, the method further comprises:

[0032] Determine multiple feature points and multiple feature lines corresponding to the target object.

[0033] For each of the reference image data, determine position information corresponding to each feature point and position information corresponding to each feature line in the reference image data.

[0034] The matching degree between the reference image data corresponding to the reference binarization image data and the target image data is determined based on the gray values of the plurality of reference pixels in the reference binarization image data and the gray values of the plurality of target pixels in the target binarization image data, and the matching degree between the reference image data corresponding to the reference binarization image data and the target image data is determined based on the gray values of the plurality of reference pixels in the reference binarization image data and the gray values of the plurality of target pixels in the target binarization image data.

[0035] For each reference binarization image data, the gray values of the reference pixels outside the feature points and the feature lines are adjusted to 0 to obtain reference adjustment image data corresponding to the reference binarization image data.

[0036] The target binarization image data is subjected to edge detection processing to obtain a plurality of edge positions in the target binarization image data.

[0037] In the target binarization image data, the gray values of the target pixels outside the edge positions are adjusted to 0 to obtain target adjustment image data corresponding to the target binarization image data.

[0038] The matching degree between the reference image data corresponding to the reference adjustment image data and the target image data is determined based on the gray values of the plurality of reference pixels in the reference adjustment image data and the gray values of the plurality of target pixels in the target adjustment image data.

[0039] In a possible implementation, the method further includes:

[0040] The encoder information corresponding to the conveyor belt is obtained by an encoder.

[0041] The pose information corresponding to the target image data is determined based on the pose information corresponding to the reference image data matched with the target image data, and the pose information corresponding to the target image data is determined based on the pose information corresponding to the reference image data matched with the target image data.

[0042] The pose information corresponding to the reference image data matched with the target image data and the encoder information are subjected to data fusion processing based on a Kalman filter to obtain the pose information corresponding to the target image data.

[0043] In a second aspect, a robot control device is provided, and the device includes:

[0044] A first obtaining module is configured to obtain a plurality of reference image data of a target object, each reference image data corresponding to different pose information.

[0045] A second obtaining module is configured to obtain target image data of the target object by a camera.

[0046] A first determining module is configured to determine reference image data matched with the target image data based on the matching degree between the target image data and each reference image data.

[0047] a second determining module, configured to determine pose information corresponding to the target image data based on pose information corresponding to reference image data matched with the target image data;

[0048] a control module, configured to control the robot work based on the pose information corresponding to the target image data.

[0049] In a possible implementation, the first determining module is configured to:

[0050] perform binarization processing on the reference image data to obtain reference binarization image data;

[0051] perform binarization processing on the target image data to obtain target binarization image data;

[0052] for each reference binarization image data, determine a matching degree between reference image data corresponding to the reference binarization image data and the target image data based on gray values of a plurality of reference pixel points in the reference binarization image data and gray values of a plurality of target pixel points in the target region;

[0053] determine, as the reference image data matched with the target image data, reference image data with the largest matching degree between the reference image data and the target image data.

[0054] In a possible implementation, the apparatus further includes a third determining module, configured to:

[0055] determine a target region corresponding to the target object in the target binarization image data based on the target binarization image data and a visual tracking algorithm;

[0056] the first determining module is configured to:

[0057] determine a matching degree between reference image data corresponding to the reference binarization image data and the target image data based on gray values of a plurality of reference pixel points in the reference binarization image data and gray values of a plurality of target pixel points in the target region.

[0058] In a possible implementation, the first determining module is configured to:

[0059] based on a preset window, frame a plurality of framed regions in the reference binarization image data a plurality of times, wherein a size of the preset window is the same as a size of the target region;

[0060] for each framed region, determine a region matching degree between the framed region and the target region based on gray values of a plurality of reference pixel points in the framed region and gray values of a plurality of target pixel points in the target region.

[0061] determine a matching degree between the reference image data corresponding to the reference binarization image data and the target image data based on the region matching degrees between the multiple frame-enclosed regions and the target region.

[0062] In a possible implementation, the first determining module is configured to:

[0063] determine the matching degree between the reference image data corresponding to the reference binarization image data and the target image data as the region matching degree with the maximum value among the region matching degrees corresponding to the multiple frame-enclosed regions.

[0064] In a possible implementation, the first determining module is configured to:

[0065] determine the frame-enclosed region with the maximum value of the corresponding region matching degrees as the frame-enclosed region to be adjusted;

[0066] determine a matching region in the reference binarization image data that matches the target region based on the ICP algorithm, the frame-enclosed region to be adjusted, the reference binarization image data, and the target region, wherein a region matching degree between the matching region and the target region is greater than or equal to a region matching degree between the frame-enclosed region to be adjusted and the target region;

[0067] determine the matching degree between the reference image data corresponding to the reference binarization image data and the target image data as the region matching degree between the matching region and the target region.

[0068] In a possible implementation, the apparatus further includes a fourth determining module configured to:

[0069] determine multiple feature points and multiple feature lines corresponding to the target object;

[0070] for each reference image data, determine position information corresponding to each feature point and position information corresponding to each feature line in the reference image data;

[0071] the first determining module is configured to:

[0072] for each reference binarization image data, adjust a gray value of a reference pixel point other than the feature points and the feature lines to 0 to obtain reference adjustment image data corresponding to the reference binarization image data;

[0073] perform edge detection processing on the target binarization image data to obtain multiple edge positions in the target binarization image data;

[0074] In the target binarization image data, the gray value of the target pixel point outside the edge position is adjusted to 0, and the target adjustment image data corresponding to the target binarization image data is obtained.

[0075] Based on the gray values of the plurality of reference pixel points in the reference adjustment image data and the gray values of the plurality of target pixel points in the target adjustment image data, a matching degree between the reference image data corresponding to the reference adjustment image data and the target image data is determined.

[0076] In a possible implementation, the apparatus further includes a third acquisition module configured to:

[0077] The encoder information corresponding to the conveyor belt is acquired by an encoder.

[0078] The second determination module is configured to:

[0079] Based on the Kalman filter, the pose information corresponding to the reference image data matched with the target image data and the encoder information are subjected to data fusion processing, and the pose information corresponding to the target image data is obtained.

[0080] In a third aspect, a computer device is provided, which includes a processor and a memory, and the memory stores at least one instruction, which is loaded and executed by the processor to implement the operations performed by the robot control method.

[0081] In a fourth aspect, a computer readable storage medium is provided, which stores at least one instruction, which is loaded and executed by the processor to implement the operations performed by the robot control method.

[0082] In a fifth aspect, a computer program product is provided, which includes at least one instruction, which is loaded and executed by the processor to implement the operations performed by the robot control method.

[0083] The technical scheme provided by the embodiments of the present disclosure has the beneficial effects that: the scheme mentioned in the embodiments of the present disclosure determines the pose information corresponding to the target image data, so that the robot can directly determine the current spatial position and attitude of the target object according to the pose information, and can more accurately perform assembly or detection operations on the target object, thereby improving the accuracy of robot operation. BRIEF DESCRIPTION OF DRAWINGS

[0084] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings needed to be used in the embodiments description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present disclosure, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0085] Figure 1 is a flowchart of a robot control method provided by an embodiment of the present disclosure;

[0086] Figure 2 is a flowchart of a robot control method provided by an embodiment of the present disclosure;

[0087] Figure 3 is a flowchart of a robot control method provided by an embodiment of the present disclosure;

[0088] Figure 4 is a schematic diagram of a target area provided by an embodiment of the present disclosure;

[0089] Figure 5 is a flowchart of a robot control method provided by an embodiment of the present disclosure;

[0090] Figure 6 is a flowchart of a method for determining a matching degree provided by an embodiment of the present disclosure;

[0091] Figure 7 is a flowchart of a method for determining a matching degree provided by an embodiment of the present disclosure;

[0092] Figure 8 is a flowchart of a robot control method provided by an embodiment of the present disclosure;

[0093] Figure 9 is a structural schematic diagram of a robot control device provided by an embodiment of the present disclosure;

[0094] Figure 10 is a structural block diagram of a terminal provided by an embodiment of the present disclosure;

[0095] Figure 11 is a structural block diagram of a server provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0096] In order to make the objects, technical solutions and advantages of the present disclosure clearer, the embodiments of the present disclosure will be further described in detail below with reference to the drawings.

[0097] The method can be implemented by a computer device. The computer device can be a terminal and a server, etc. The terminal can be a desktop computer, a notebook computer, a tablet computer, a mobile phone, etc. The server can be a single server or a server cluster, etc.

[0098] The computer device can include a processor, a memory, a communication component, etc.

[0099] The processor can be a central processing unit (CPU). The processor can be configured to read instructions and process data, for example, obtaining a plurality of reference image data of a target object, obtaining target image data of the target object, determining reference image data matched with the target image data, determining pose information corresponding to the target image data, controlling the robot to work, etc.

[0100] The memory can be various volatile memories or non-volatile memories, such as a solid state disk (SSD), a dynamic random access memory (DRAM), etc. The memory can be configured to store data, for example, storing the obtained plurality of reference image data of the target object, storing the obtained target image data of the target object, storing the determined reference image data matched with the target image data, storing the determined pose information corresponding to the target image data, storing intermediate data generated in the process of controlling the robot to work, etc.

[0101] The communication component can be a wired network connector, a wireless fidelity (WiFi) module, a Bluetooth module, a cellular communication module, etc. The communication component can be configured to transmit data with other devices, for example, receiving target image data of the target object sent by a camera, etc.

[0102] The robot control method provided by the present disclosure can be applied to some industrial production lines equipped with robots, for controlling the robots to assemble, detect, etc. products, wherein the robot can be various types of robots in the industrial production line, for example, a mechanical arm, etc.

[0103] Figure 1 And Figure 2 is a flowchart of a robot control method provided by an embodiment of the present disclosure. Referring to Figure 1 And Figure 2 The embodiment includes:

[0104] 101. Obtain a plurality of reference image data of a target object, each reference image data corresponding to different pose information.

[0105] In implementation, the target object can be photographed by the camera at each angle and each position of 6D freedom, so as to obtain a plurality of reference image data of the target object, each of which corresponds to different pose information. It can be understood that the pose information corresponding to the reference image data refers to the pose information of the target object in the reference image data.

[0106] Alternatively, a 3D model of the target object can be established, and the reference image data of the target object at different pose information can be obtained through computer vision algorithm.

[0107] In a possible implementation, the number of reference image data of the target object can be set according to requirements, for example, 10,000, 50,000 or the like, and the embodiments of the present disclosure do not make specific limitation thereon.

[0108] 102. Obtain target image data of the target object through the camera.

[0109] In implementation, when the robot needs to operate the target object on the industrial production line, the target object can be photographed by the camera, so as to obtain the target image data of the target object.

[0110] The camera can be fixed at any reasonable position on the industrial production line, for example, can be fixed at a fixed position, or can be fixed at the end of the mechanical arm of the robot, and the like, and the embodiments of the present disclosure do not make specific limitation thereon.

[0111] 103. Determine the reference image data matched with the target image data based on the matching degree between the target image data and each reference image data.

[0112] In implementation, after obtaining the target image data of the target object, the matching degree between the target image data and each reference image data can be calculated respectively.

[0113] Then, the reference image data with the maximum value of the matching degree is determined. Since the matching degree between the reference image data and the target image data in the plurality of reference image data is the largest, it is indicated that the pose information of the target object in the target image data is also the most similar to the pose information of the target object in the reference image data with the largest matching degree. Therefore, the reference image data with the largest value of the matching degree between the target image data can be determined as the reference image data matched with the target image data.

[0114] In a possible implementation, the pose information can include the three-dimensional coordinates of the centroid of the target object, and the roll angle, the pitch angle and the yaw angle of the centroid.

[0115] 104. determining the pose information corresponding to the target image data based on the pose information corresponding to the reference image data matched with the target image data.

[0116] In implementation, since the matching degree of the reference image data matched with the target image data is the largest among the plurality of reference image data, it is indicated that the reference image data is the most similar to the target image data, and the pose information of the target object in the reference image data is also the most similar to the pose information of the target object in the target image data. Therefore, the pose information corresponding to the target image data can be determined based on the pose information corresponding to the reference image data with the largest value of the matching degree. In this way, the pose information corresponding to the target image data can be obtained more accurately.

[0117] It can be understood that the pose information corresponding to the target image data refers to the pose information of the target object in the target image data.

[0118] 105. controlling the robot based on the pose information corresponding to the target image data.

[0119] In implementation, after the pose information corresponding to the target image data is determined, the pose information corresponding to the target image data is sent to the robot controller, and the robot control is based on the obtained pose information corresponding to the target image data to generate a control signal, and the control signal is sent to the motion control operator module of the robot. The final control instruction is generated through the PID algorithm combined with the robot model, and the robot moves based on the control instruction, thereby realizing the operation on the target object.

[0120] In the process of controlling the robot operation, the above steps 101-105 are periodically processed, thereby realizing the visual closed-loop control of the robot. Based on the operation of the robot, new target image data is obtained, and the pose information corresponding to the new target image data is determined, thereby realizing the closed-loop adjustment of the robot operation and improving the accuracy of the operation.

[0121] In summary, the scheme mentioned in the embodiments of the present disclosure can determine the pose information corresponding to the target image data. In this way, the robot can directly determine the current spatial position and attitude of the target object based on the pose information, and can more accurately perform assembly or detection operations on the target object, thereby improving the accuracy of the robot operation.

[0122] And, the scheme mentioned in the embodiments of the present disclosure does not need to be limited for the type of target object, and can be an object of any structure or shape, so as to adapt to most industrial follow-up assembly and detection lines, thereby liberating manual work, saving labor, and having higher consistency in operation pace, intensity, and work quality, improving the efficiency of industrial production lines and the qualified rate of finished products.

[0123] In step 103, there are various methods for determining the reference image data matched with the target image data based on the matching degree between the target image data and each reference image data, which will be introduced as follows.

[0124] Referring to Figure 3 The method can be: performing binaryzation processing on the reference image data to obtain reference binaryzation image data; performing binaryzation processing on the target image data to obtain target binaryzation image data; for each reference binaryzation image data, determining the matching degree between the reference image data corresponding to the reference binaryzation image data and the target image data based on the gray values of the plurality of reference pixel points in the reference binaryzation image data and the gray values of the plurality of target pixel points in the target binaryzation image data; and determining the reference image data with the largest matching degree with the target image data as the reference image data matched with the target image data.

[0125] In implementation, the reference image data can be first converted into a gray scale image, and then based on a binaryzation threshold, the gray values of the reference pixel points in the gray scale image corresponding to the reference image data that are greater than or equal to the binaryzation threshold are all adjusted to 1, and the gray values of the reference pixel points in the gray scale image corresponding to the reference image data that are less than the binaryzation threshold are all adjusted to 0, so as to obtain the reference binaryzation image data composed of 1 and 0.

[0126] Similarly, for the target image data, the target image data can be first converted into a gray scale image, and then based on a binaryzation threshold, the gray values of the target pixel points in the gray scale image corresponding to the target image data that are greater than or equal to the binaryzation threshold are all adjusted to 1, and the gray values of the target pixel points in the gray scale image corresponding to the target image data that are less than the binaryzation threshold are all adjusted to 0, so as to obtain the target binaryzation image data composed of 1 and 0.

[0127] Then, for each reference binaryzation image data, the matching degree between the reference image data corresponding to the reference binaryzation image data and the target image data can be determined based on the gray values of the plurality of reference pixel points in the reference binaryzation image data and the gray values of the plurality of target pixel points in the target binaryzation image data.

[0128] The matching degree between each reference binarization image data and the target binarization image data is determined by the above method, that is, the matching degree between each reference image data and the target image data.

[0129] The reference image data with the maximum value of the matching degree between the target image data is determined as the reference image data matched with the target image data.

[0130] In the above method, the reference image data and the target image data are binarized to obtain reference binarization image data and target binarization image data. The outline of the target object is clear in the reference binarization image data and the target binarization image data. The matching degree is determined based on the reference binarization image data and the target binarization image data, which can ensure the accuracy of the matching degree while improving the calculation speed, thereby realizing more accurate and real-time control of the robot.

[0131] In a possible implementation, there are multiple methods for determining the matching degree between the reference image data and the target image data based on the gray values of the multiple reference pixel points in the reference binarization image data and the gray values of the multiple target pixel points in the target binarization image data. Two of them are introduced as follows:

[0132] The first method for determining the matching degree is to input the reference binarization image data and the target binarization image data into a matching degree prediction model to obtain a predicted matching degree as output. The predicted matching degree is the matching degree between the reference image data corresponding to the reference binarization image data and the target image data corresponding to the target binarization image data.

[0133] The matching degree prediction model described above can be any reasonable structure of a machine learning model, which is not specifically limited in the embodiments of the present disclosure.

[0134] The second method for determining the matching degree is to perform multiple frame selection in the reference binarization image data based on a first preset window. The size of the first preset window can be the same as the size of the target binarization image data.

[0135] Then, for each frame selection region, the matching degree between the frame selection region and the target image data can be determined based on the gray values of the multiple reference pixel points in the frame selection region and the gray values of the multiple target pixel points in the target image data.

[0136] The matching degree between the multiple frame selection regions and the target image data is obtained by the above method. The matching degree with the maximum value in the multiple matching degrees is determined as the matching degree between the reference image data and the target image data.

[0137] The two methods for determining the matching degree are merely illustrative. The method for determining the matching degree between the reference image data and the target image data in the embodiments of the present disclosure can also be any other reasonable method, which is not limited in the embodiments of the present disclosure.

[0138] In a possible implementation, the robot control method provided by the embodiments of the present disclosure can further include the following step: determining a target region corresponding to the target object in the target binary image data based on the target binary image data and the visual tracking algorithm.

[0139] In implementation, the visual tracking algorithm needs to be initialized for the target object, that is, the grayscale image data of the target object and the region where the target object is located in the grayscale image data are acquired first, and then the grayscale image data of the target object is input into the visual tracking algorithm, and each parameter in the visual tracking algorithm is initialized and adjusted based on the region where the target object is located.

[0140] Then, the initialized visual tracking algorithm can be used to process the target binary image data, so as to obtain the target region corresponding to the target object in the target binary image data, as shown in Figure 4 .

[0141] In this way, the part of other objects or background in the target binary image data except the target object can be removed, so as to avoid subsequent redundant calculation.

[0142] After the target region corresponding to the target object is determined, when the above step (determining the matching degree between the reference image data and the target image data based on the grayscale values of the plurality of reference pixel points in the reference binary image data and the grayscale values of the plurality of target pixel points in the target binary image data) is performed, as shown in Figure 5 , the following processing can be performed: determining the matching degree between the reference image data and the target image data based on the grayscale values of the plurality of reference pixel points in the reference binary image data and the grayscale values of the plurality of target pixel points in the target region.

[0143] In implementation, after the target region where the target object is located in the target binary image data is determined, the reference binary image data and the target region can be matched directly, so as to obtain the matching degree between each reference image data and the target region, that is, the matching degree between each reference image data and the target image data.

[0144] In this way, the matching calculation of other regions irrelevant to the target object in the target binary image data is avoided, the calculation amount is greatly reduced, the speed of determining the matching degree is improved, and more accurate and real-time control of the robot is realized.

[0145] In this case, the method for determining the matching degree between the reference image data and the target image data can also be various, for example:

[0146] The first method for determining the matching degree is to input the reference binarization image data and the image data corresponding to the target region into a matching degree prediction model to obtain an output predicted matching degree, which is the matching degree between the reference image data and the target image data.

[0147] The matching degree prediction model described above can be any reasonable structure machine learning model, and the embodiments of the present disclosure do not make specific limitations thereto.

[0148] The second method for determining the matching degree is to frame multiple frame regions in the reference binarization image data based on a preset window, wherein the size of the preset window is the same as that of the target region; for each frame region, the region matching degree between the frame region and the target region is determined based on the gray values of the multiple reference pixel points in the frame region and the gray values of the multiple target pixel points in the target region; and the matching degree between the reference image data corresponding to the reference binarization image data and the target image data is determined based on the region matching degrees between the multiple frame regions and the target region.

[0149] In implementation, a preset window with the same size as the target region can be first set, and then the preset window is framed in the reference binarization image data, to obtain a first frame region; after one framing, the preset window is moved in the reference binarization image data according to a preset moving track, and after the movement, the preset window frames a second frame region; then the preset window is moved in the reference binarization image data according to the preset moving track again, to frame a third frame region, and so on. The preset window is moved in the reference binarization image data in a traversal manner, to obtain multiple frame regions.

[0150] For example, the preset moving track is: starting from the top left corner of the reference binarization image data, first moving to the right, moving to the rightmost side of the reference binarization image data, then moving down once, then moving to the left until moving to the leftmost side, then moving down once, then moving to the right, and so on, to form an S-shaped preset moving track.

[0151] The moving distance of each movement can be any reasonable distance set in advance, for example, each movement can move a distance of three pixels, or a distance of two pixels, etc., and the embodiments of the present disclosure do not make specific limitations thereto.

[0152] After the multiple frame regions are obtained based on the method, each frame region is matched with the target region respectively to obtain a region matching degree between each frame region and the target region, and then the matching degree between the reference image data and the target image data is determined based on the multiple region matching degrees.

[0153] In the embodiments of the present disclosure, there are multiple methods for determining the matching degree between the reference image data and the target image data based on the multiple region matching degrees, and the following are two kinds of methods:

[0154] The first method for determining the matching degree is as follows: Figure 6 The region matching degree with the maximum value in the multiple region matching degrees corresponding to the multiple frame regions is determined as the matching degree between the reference image data corresponding to the reference binary image data and the target image data.

[0155] The second method for determining the matching degree is as follows: Figure 7 The frame region with the maximum value in the corresponding region matching degrees is determined as the frame region to be adjusted; the matching region in the reference binary image data that matches the target region is determined based on the ICP algorithm, the frame region to be adjusted, the reference binary image data and the target region, wherein the region matching degree between the matching region and the target region is greater than or equal to the region matching degree between the frame region to be adjusted and the target region; and the region matching degree between the matching region and the target region is determined as the matching degree between the reference image data corresponding to the reference binary image data and the target image data.

[0156] In the implementation, when the moving distance of the preset window is relatively large, the frame region with the maximum value in the region matching degrees may not be directly corresponding to the target region, and there may be a misalignment between the two. Therefore, after the frame region to be adjusted is determined, the reference binary image data, the position of the frame region to be adjusted in the reference binary image data and the target region can be input into the ICP algorithm for iterative matching, so that the image data around the frame region to be adjusted in the binary image data is matched by moving in a smaller and more fine manner, so that the matching region directly corresponding to the image data of the target region is obtained. The region matching degree between the matching region obtained in this way and the target region is greater than or equal to the region matching degree between the frame region to be adjusted and the target region.

[0157] For example, when the moving distance between the above-mentioned preset window and the target region is 3 pixels, the ICP algorithm can be set to move one pixel at a time. When the reference binary image data, the position of the to-be-adjusted bounding region in the reference binary image data, and the target region are input into the ICP algorithm, the to-be-adjusted bounding region can be moved in the reference binary image data based on the preset window, and one pixel is moved at a time. After moving one pixel to the right for the first time, the first bounding region is obtained, and then the area matching degree between the first bounding region and the target region is calculated. If the calculated area matching degree is less than the area matching degree between the to-be-adjusted bounding region and the target region, it indicates that the movement to the right is incorrect. In the next movement, the to-be-adjusted bounding region can be moved one pixel to the left to obtain the second bounding region, and then the area matching degree between the second bounding region and the target region is calculated. If the calculated area matching degree is greater than the area matching degree between the to-be-adjusted bounding region and the target region, it indicates that the movement to the left is correct, and the third movement can continue to move one pixel to the left. The area matching degree between the bounding region obtained after each movement and the target region is calculated each time. In this way, the movement to the left is continued until the calculated area matching degree is less than the area matching degree calculated after the last movement, which indicates that the bounding region obtained after the last movement is more matched with the target region. Then, the bounding region obtained after the last movement can be moved downward, and the movement to the left is similar. If the area matching degree obtained after the movement downward is improved, it indicates that the movement downward can be continued until the position with the highest area matching degree is reached. If the area matching degree obtained after the movement downward is reduced, it indicates that the movement direction is incorrect. At this time, the movement is upward until the area matching degree is improved to the highest position, so that the matching region corresponding to the target region is obtained.

[0158] Based on this method, more accurate positioning can be performed, and a matching region with a higher matching degree between the target region can be obtained.

[0159] After the matching region is determined, the area matching degree between the matching region and the target region can be determined as the matching degree between the reference image data corresponding to the reference binary image data and the target image data.

[0160] The above-mentioned method for determining the matching degree is only a few methods. The method for determining the matching degree between the reference image data and the target image data in the embodiments of the present disclosure can be any one or a combination of several methods, or can be any other reasonable method. The embodiments of the present disclosure do not make specific limitations in this regard.

[0161] In a possible implementation, the robot control method provided by the embodiments of the present disclosure can further include the following steps: determining a plurality of feature points and feature lines corresponding to the target object; and determining, for each reference image data, position information corresponding to each feature point and position information corresponding to each feature line in the reference image data.

[0162] In implementation, the staff can pre-set the plurality of feature points and feature lines corresponding to the target object, where the feature points can include a center point of a groove bottom of a groove on the target object, a center point of a region on the surface of the target object with a through hole, a tip point on the outer contour of the target object, and the like.

[0163] The feature lines can include an axis of a groove on the target object, an axis of a through hole, an outer contour line of the target object, and the like.

[0164] The feature points and feature lines can be selected according to the actual shape features of the target object, and the embodiments of the present disclosure do not make specific limitations in this regard.

[0165] The position information of the feature points is the three-dimensional coordinates of the feature points, and the position information of the feature lines can be the three-dimensional coordinates of a plurality of points on the feature lines.

[0166] After the feature points and feature lines of the target object are determined, when the step of determining the matching degree between the reference image data corresponding to the reference binary image data and the target image data based on the gray values of the plurality of reference pixel points in the reference binary image data and the gray values of the plurality of target pixel points in the target binary image data is performed, the following processing can be performed: for each reference binary image data, adjusting the gray values of the reference pixel points other than the feature points and the feature lines to 0 to obtain reference adjustment image data corresponding to the reference binary image data; performing edge detection processing on the target binary image data to obtain the edge positions in the target binary image data; in the target binary image data, adjusting the gray values of the target pixel points other than the edge positions to 0 to obtain target adjustment image data corresponding to the target binary image data; and determining the matching degree between the reference adjustment image data corresponding to the reference adjustment image data and the target image data based on the gray values of the plurality of reference pixel points in the reference adjustment image data and the gray values of the plurality of target pixel points in the target adjustment image data.

[0167] In implementation, based on a computer vision algorithm, before the reference image data is obtained by photographing the 3D model of the target object, the feature points and feature lines in the 3D model can be labeled first, and then the 3D model is photographed using a camera to obtain each reference image data and the positions of the feature points and feature lines in the reference image data.

[0168] Then, the reference image data is binarized to obtain reference binarization image data corresponding to each reference image data, the gray value of the reference pixel point in each reference binarization image data except the feature point and the feature line is adjusted to 0 to obtain reference adjustment image data corresponding to each reference binarization image data, and the reference adjustment image data is stored.

[0169] When the matching degree needs to be calculated, the target binarization image data can be subjected to edge detection processing to obtain an edge position in the target binarization image data. The edge position is a relatively obvious position in the target binarization image, for example, the contour of the target object or the like. The edge position can include the position of a point and the position of a line composed of multiple points, or other reasonable settings, which are not limited in the embodiments of the present disclosure.

[0170] In the target binarization image data, the gray value of the target pixel point except the edge position is directly adjusted to 0 to obtain target adjustment image data. In the target adjustment image data, only the gray value of the target pixel point at the relatively obvious edge position is 1, and the gray value of the other target pixel points is 0.

[0171] Then, the matching degree between the reference image data corresponding to the reference binarization image data and the target image data can be determined based on the reference adjustment image data and the target adjustment image data.

[0172] In this way, the calculation speed can be improved when the matching degree is determined subsequently, and the interference of other backgrounds is avoided, and the accuracy of the matching is improved.

[0173] In the embodiments of the present disclosure, the method for determining the matching degree between the reference image data and the target image data based on the reference adjustment image data and the target adjustment image data can be any one or a combination of several of the methods for determining the matching degree based on the reference binarization image data and the target binarization image data as described above, which will not be described herein.

[0174] In the embodiments of the present disclosure, there are various methods for determining the pose information corresponding to the target image data based on the pose information corresponding to the reference image data matched with the target image data, which will be introduced as follows:

[0175] In one possible implementation, after the reference image data matched with the target image data is determined, the pose information corresponding to the reference image data matched with the target image data can be directly determined as the pose information corresponding to the target image data.

[0176] In another possible implementation, the robot control method provided by the embodiments of the present disclosure can further include the following step: obtaining the encoder information corresponding to the conveying belt through an encoder.

[0177] In implementation, since a conveyor belt is usually used to transport the target object in an industrial production line, the encoder can be installed on the roller corresponding to the conveyor belt. When the roller drives the conveyor belt to move, the encoder generates encoder information in real time, which can reflect the moving distance of the conveyor belt.

[0178] In this way, when the above step (determining the pose information corresponding to the target image data based on the pose information corresponding to the reference image data matched with the target image data) is performed, the following processing can be performed: performing data fusion processing on the pose information corresponding to the reference image data matched with the target image data and the encoder information based on the Kalman filter to obtain the pose information corresponding to the target image data.

[0179] In implementation, the conveying direction of the conveyor belt can be determined first, and then the component of the pose information corresponding to the reference image data matched with the target image data in the conveying direction is calculated. The obtained component and the encoder information are input into the Kalman filter for Kalman data fusion processing, so as to obtain the position information of the centroid of the target object in the conveying direction.

[0180] Based on the position information of the centroid of the target object in the conveying direction, the information related to the component in the conveying direction in the pose information is adjusted to obtain adjusted pose information, and then the adjusted pose information can be determined as the pose information corresponding to the target image data.

[0181] Since the frequency of the encoder is usually high, the encoder information with high real-time performance is used to adjust the pose information, so as to improve the accuracy of determining the pose information corresponding to the target image data.

[0182] All the optional technical solutions described above can be combined to form optional embodiments of the present disclosure, which will not be described one by one here.

[0183] The scheme mentioned in the embodiments of the present disclosure determines the pose information corresponding to the target image data. In this way, the robot can directly determine the current spatial position and attitude of the target object according to the pose information, and can more accurately perform assembly or detection operations on the target object, thereby improving the accuracy of robot operation.

[0184] The embodiments of the present disclosure provide a robot control device. The device can be the computer device in the above embodiments, as shown in the following Figure 9 The device includes:

[0185] The first acquisition module 910 is configured to acquire a plurality of reference image data of a target object, each reference image data corresponding to different pose information.

[0186] The second acquisition module 920 is configured to acquire target image data of the target object by using a camera.

[0187] The first determination module 930 is configured to determine reference image data matched with the target image data based on matching degrees between the target image data and each reference image data.

[0188] The second determination module 940 is configured to determine pose information corresponding to the target image data based on pose information corresponding to the reference image data matched with the target image data.

[0189] The control module 950 is configured to control the robot work based on the pose information corresponding to the target image data.

[0190] In a possible implementation, the first determination module 930 is configured to:

[0191] perform binaryzation processing on the reference image data to obtain reference binaryzation image data;

[0192] perform binaryzation processing on the target image data to obtain target binaryzation image data;

[0193] For each reference binaryzation image data, determine a matching degree between reference image data corresponding to the reference binaryzation image data and the target image data based on gray values of a plurality of reference pixel points in the reference binaryzation image data and gray values of a plurality of target pixel points in the target binaryzation image data.

[0194] determine, as the reference image data matched with the target image data, reference image data with the largest matching degree between the reference image data and the target image data.

[0195] In a possible implementation, the apparatus further includes a third determination module configured to:

[0196] determine a target region corresponding to the target object in the target binaryzation image data based on the target binaryzation image data and a visual tracking algorithm.

[0197] The first determination module 930 is configured to:

[0198] determine a matching degree between reference image data corresponding to the reference binaryzation image data and the target image data based on gray values of a plurality of reference pixel points in the reference binaryzation image data and gray values of a plurality of target pixel points in the target region.

[0199] In a possible implementation, the first determination module 930 is configured to:

[0200] based on a preset window, multiple frame regions are framed in the reference binarization image data, wherein the size of the preset window is the same as the size of the target region;

[0201] for each frame region, based on the gray values of the multiple reference pixels in the frame region and the gray values of the multiple target pixels in the target region, a region matching degree between the frame region and the target region is determined;

[0202] based on the region matching degrees between the multiple frame regions and the target region, a matching degree between the reference image data corresponding to the reference binarization image data and the target image data is determined.

[0203] In a possible implementation, the first determining module 930 is configured to:

[0204] the region matching degree with the maximum value in the region matching degrees corresponding to the multiple frame regions is determined as the matching degree between the reference image data corresponding to the reference binarization image data and the target image data.

[0205] In a possible implementation, the first determining module 930 is configured to:

[0206] the frame region with the maximum value of the corresponding region matching degrees is determined as the frame region to be adjusted;

[0207] based on the ICP algorithm, the frame region to be adjusted, the reference binarization image data and the target region, a matching region in the reference binarization image data that matches the target region is determined, wherein the region matching degree between the matching region and the target region is greater than or equal to the region matching degree between the frame region to be adjusted and the target region;

[0208] the region matching degree between the matching region and the target region is determined as the matching degree between the reference image data corresponding to the reference binarization image data and the target image data.

[0209] In a possible implementation, the apparatus further includes a fourth determining module configured to:

[0210] determine multiple feature points and multiple feature lines corresponding to the target object;

[0211] for each reference image data, determine the position information of each feature point and the position information of each feature line in the reference image data;

[0212] the first determining module 930 is configured to:

[0213] For each reference binarization image data, the gray value of the reference pixel point outside the feature point and the feature line is adjusted to 0, to obtain the reference adjustment image data corresponding to the reference binarization image data;

[0214] The target binarization image data is subjected to edge detection processing, to obtain a plurality of edge positions in the target binarization image data;

[0215] In the target binarization image data, the gray value of the target pixel point outside the edge position is adjusted to 0, to obtain the target adjustment image data corresponding to the target binarization image data;

[0216] Based on the gray values of the plurality of reference pixel points in the reference adjustment image data and the gray values of the plurality of target pixel points in the target adjustment image data, the matching degree between the reference image data corresponding to the reference adjustment image data and the target image data is determined.

[0217] In a possible implementation, the apparatus further includes a third acquisition module configured to:

[0218] acquire, by an encoder, the encoder information corresponding to the conveyor belt;

[0219] The second determination module 940 is configured to:

[0220] Based on the Kalman filter, the pose information corresponding to the reference image data matched with the target image data and the encoder information are subjected to data fusion processing, to obtain the pose information corresponding to the target image data.

[0221] It should be noted that the robot control apparatus provided in the above embodiments is only used for example to illustrate the division of the above functional modules in controlling the robot, and in actual application, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the apparatus is divided into different functional modules to complete all or part of the above described functions. In addition, the robot control apparatus and the robot control method provided in the above embodiments belong to the same concept, and the specific implementation process is described in detail in the method embodiments, which will not be repeated here.

[0222] Figure 10A structural block diagram of a terminal 1000 provided by one exemplary embodiment of the present disclosure is shown. The terminal can be the computer device in the above-described embodiments. The terminal 1000 can be a smartphone, a tablet computer, an MP3 player, an MP4 player, a notebook computer, or a desktop computer. The terminal 1000 can also be referred to as a user equipment, a portable terminal, a laptop terminal, a desktop terminal, or other names.

[0223] Generally, the terminal 1000 includes a processor 1001 and a memory 1002.

[0224] The processor 1001 can include one or more processing cores, such as a 4-core processor, an 8-core processor, or the like. The processor 1001 can be implemented in at least one of a hardware form of a DSP (digital signal processing), an FPGA (field-programmable gate array), a PLA (programmable logic array). The processor 1001 can also include a main processor and a coprocessor, the main processor being a processor for processing data in an awake state, also referred to as a CPU (central processing unit), and the coprocessor being a low-power processor for processing data in a standby state. In some embodiments, the processor 1001 can be integrated with a GPU (graphics processing unit) for rendering and drawing content required to be displayed by a display screen. In some embodiments, the processor 1001 can further include an AI (artificial intelligence) processor for processing computing operations related to machine learning.

[0225] The memory 1002 can include one or more computer-readable storage media, which can be non-transitory. The memory 1002 can also include a high-speed random access memory, and a non-volatile memory such as one or more disk storage devices, flash storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 1002 is used to store at least one instruction for being executed by the processor 1001 to implement the robot control method provided by the method embodiment of the present disclosure.

[0226] In some embodiments, terminal 1000 can further include a peripheral device interface 1003 and at least one peripheral device. The processor 1001, the memory 1002 and the peripheral device interface 1003 can be connected through a bus or a signal line. Each peripheral device can be connected with the peripheral device interface 1003 through a bus, a signal line or a circuit board. Specifically, the peripheral device includes at least one of a radio frequency circuit 1004, a display screen 1005, a camera 1006, an audio circuit 1007, a positioning component 1008 and a power supply 1009.

[0227] The peripheral device interface 1003 can be used to connect at least one peripheral device related to input / output to the processor 1001 and the memory 1002. In some embodiments, the processor 1001, the memory 1002 and the peripheral device interface 1003 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 1001, the memory 1002 and the peripheral device interface 1003 can be implemented on a separate chip or circuit board, and the present embodiment is not limited in this regard.

[0228] The radio frequency circuit 1004 is used to receive and transmit RF (radio frequency) signals, also known as electromagnetic signals. The radio frequency circuit 1004 communicates with a communication network and other communication devices through electromagnetic signals. The radio frequency circuit 1004 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the radio frequency circuit 1004 includes an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a subscriber identity module card, and the like. The radio frequency circuit 1004 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes but is not limited to metropolitan area networks, various generations of mobile communication networks (2G, 3G, 4G and 5G), wireless local area networks and / or WiFi (wireless fidelity) networks. In some embodiments, the radio frequency circuit 1004 can also include NFC (near field communication) related circuit, and the present disclosure is not limited in this regard.

[0229] The display screen 1005 is configured to display a UI (user interface). The UI can include graphics, text, icons, video, and any combination thereof. When the display screen 1005 is a touch display screen, the display screen 1005 is further configured to capture touch signals on or above the surface of the display screen 1005. The touch signals can be input to the processor 1001 as control signals for processing. In this case, the display screen 1005 can also be configured to provide virtual buttons and / or virtual keyboard, also known as soft buttons and / or soft keyboard. In some embodiments, the display screen 1005 can be one, arranged on the front panel of the terminal 1000; in other embodiments, the display screen 1005 can be at least two, arranged on different surfaces of the terminal 1000 or in a folding design; in still other embodiments, the display screen 1005 can be a flexible display screen, arranged on a curved surface or a folding surface of the terminal 1000. Even, the display screen 1005 can also be arranged in an irregular shape other than a rectangle, i.e., a special-shaped screen. The display screen 1005 can be made of LCD (liquid crystal display), OLED (organic light-emitting diode), etc.

[0230] The camera assembly 1006 is configured to capture images or videos. Optionally, the camera assembly 1006 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is arranged on the front panel of the terminal, and the rear-facing camera is arranged on the back of the terminal. In some embodiments, the rear-facing camera is at least two, which are any one of a main camera, a depth-of-field camera, a wide-angle camera, and a telephoto camera, to realize the background blur function by fusing the main camera and the depth-of-field camera, the panoramic shooting and VR (virtual reality) shooting function by fusing the main camera and the wide-angle camera, or other fusion shooting functions. In some embodiments, the camera assembly 1006 can further include a flash. The flash can be a single-color-temperature flash or a dual-color-temperature flash. The dual-color-temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation under different color temperatures.

[0231] The audio circuit 1007 can include a microphone and a speaker. The microphone is used to collect sound waves of a user and an environment, and convert the sound waves into an electrical signal input to the processor 1001 for processing, or input to the radio frequency circuit 1004 to realize voice communication. For the purpose of stereo sound collection or noise reduction, the microphone can be multiple, which are respectively arranged at different parts of the terminal 1000. The microphone can also be an array microphone or an omnidirectional collection type microphone. The speaker is used to convert an electrical signal from the processor 1001 or the radio frequency circuit 1004 into sound waves. The speaker can be a traditional diaphragm speaker, or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, not only can the electrical signal be converted into a sound wave audible to humans, but also can be converted into a sound wave inaudible to humans for ranging purposes. In some embodiments, the audio circuit 1007 can also include a headphone jack.

[0232] The positioning component 1008 is used to position the current geographic location of the terminal 1000 to realize navigation or LBS (location based service). The positioning component 1008 can be a positioning component based on a GPS (global positioning system), a Beidou system, a Glonass system or a Galileo system.

[0233] The power supply 1009 is used to supply power to each component in the terminal 1000. The power supply 1009 can be an alternating current, a direct current, a disposable battery or a rechargeable battery. When the power supply 1009 includes a rechargeable battery, the rechargeable battery can support wired charging or wireless charging. The rechargeable battery can also be used to support fast charging technology.

[0234] In some embodiments, the terminal 1000 further includes one or more sensors 1010. The one or more sensors 1010 include but are not limited to an acceleration sensor 1011, a gyroscope sensor 1012, a pressure sensor 1013, a fingerprint sensor 1014, an optical sensor 1015 and a proximity sensor 1016.

[0235] The acceleration sensor 1011 can detect the acceleration magnitude in three coordinate axes of the coordinate system established by the terminal 1000. For example, the acceleration sensor 1011 can be used to detect the components of gravitational acceleration in three coordinate axes. The processor 1001 can control the display screen 1005 to display the user interface in a landscape view or a portrait view according to the gravitational acceleration signal collected by the acceleration sensor 1011. The acceleration sensor 1011 can also be used for game or user motion data collection.

[0236] The gyroscope sensor 1012 can detect the body direction and rotation angle of the terminal 1000, and can collect 3D motions of the user on the terminal 1000 in cooperation with the acceleration sensor 1011. The processor 1001 can implement the following functions according to the data collected by the gyroscope sensor 1012: motion sensing (such as changing the UI according to the tilt operation of the user), image stabilization when shooting, game control, and inertial navigation.

[0237] The pressure sensor 1013 can be arranged on the side frame of the terminal 1000 and / or the lower layer of the display screen 1005. When the pressure sensor 1013 is arranged on the side frame of the terminal 1000, the holding signal of the user on the terminal 1000 can be detected, and the left-hand or right-hand recognition or shortcut operation can be performed by the processor 1001 according to the holding signal collected by the pressure sensor 1013. When the pressure sensor 1013 is arranged on the lower layer of the display screen 1005, the controllable control on the UI interface can be controlled by the processor 1001 according to the pressure operation of the user on the display screen 1005. The controllable control includes at least one of a button control, a scroll bar control, an icon control, and a menu control.

[0238] The fingerprint sensor 1014 is used to collect the fingerprint of the user, and the identity of the user can be recognized by the processor 1001 according to the fingerprint collected by the fingerprint sensor 1014, or by the fingerprint sensor 1014 according to the collected fingerprint. When the identity of the user is recognized as a trusted identity, the processor 1001 authorizes the user to perform related sensitive operations, including unlocking the screen, viewing encrypted information, downloading software, payment, and changing settings. The fingerprint sensor 1014 can be arranged on the front, back or side of the terminal 1000. When the terminal 1000 is provided with a physical button or a manufacturer's logo, the fingerprint sensor 1014 can be integrated with the physical button or the manufacturer's logo.

[0239] The optical sensor 1015 is used to collect the ambient light intensity. In one embodiment, the processor 1001 can control the display brightness of the display screen 1005 according to the ambient light intensity collected by the optical sensor 1015. Specifically, when the ambient light intensity is high, the display brightness of the display screen 1005 is increased; when the ambient light intensity is low, the display brightness of the display screen 1005 is decreased. In another embodiment, the processor 1001 can also dynamically adjust the shooting parameters of the camera assembly 1006 according to the ambient light intensity collected by the optical sensor 1015.

[0240] The proximity sensor 1016, also referred to as a distance sensor, is usually arranged on the front panel of the terminal 1000. The proximity sensor 1016 is used to collect the distance between the user and the front of the terminal 1000. In an embodiment, when the proximity sensor 1016 detects that the distance between the user and the front of the terminal 1000 gradually decreases, the display screen 1005 is switched from the bright screen state to the screen-off state under the control of the processor 1001; when the proximity sensor 1016 detects that the distance between the user and the front of the terminal 1000 gradually increases, the display screen 1005 is switched from the screen-off state to the bright screen state under the control of the processor 1001.

[0241] Those skilled in the art can understand that the structure shown in the foregoing embodiments is not a limitation on the terminal 1000, and the terminal 1000 can include more or fewer components than those shown in the drawings, or combine certain components, or adopt a different arrangement of components. Figure 10 Those skilled in the art can understand that the structure shown in the foregoing embodiments is not a limitation on the terminal 1000, and the terminal 1000 can include more or fewer components than those shown in the drawings, or combine certain components, or adopt a different arrangement of components.

[0242] Figure 11 FIG. 11 is a structural schematic diagram of a server according to an embodiment of the present disclosure. The server 1100 can have a large difference due to different configurations or performances, and can include one or more central processing units (CPUs) 1101 and one or more memories 1102. The memory 1102 stores at least one instruction, which is loaded and executed by the processor 1101 to implement the method provided by the above-mentioned various method embodiments. Of course, the server can also have a wired or wireless network interface, a keyboard, and an input and output interface, and other components for implementing device functions, which are not described here.

[0243] In an exemplary embodiment, a computer readable storage medium, such as a memory including instructions, is also provided, which can be executed by a processor in a terminal to complete the robot control method in the above-mentioned embodiments. The computer readable storage medium can be non-transitory. For example, the computer readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like.

[0244] Those of ordinary skill in the art can understand that all or part of the steps of the above-mentioned embodiments can be completed by hardware, or by a program instructing related hardware, and the program can be stored in a computer readable storage medium, such as a read-only memory, a magnetic disk or an optical disk.

[0245] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data for analysis, stored data, displayed data, etc.) and signals (including but not limited to signals transmitted between user terminals and other devices, etc.) involved in the present disclosure are all authorized by the user or fully authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions. For example, the "reference image data" and "target image data" involved in the present disclosure are obtained under full authorization.

[0246] The above only describes optional embodiments of the present disclosure and is not intended to limit the present disclosure. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A robot control method, characterized in that, The method includes: Acquire multiple reference image data of the target object, each reference image data corresponding to different pose information; The target image data of the target object is acquired through a camera; The reference image data is binarized to obtain reference binarized image data; The target image data is binarized to obtain target binarized image data; Based on the target binarized image data and the visual tracking algorithm, the target region corresponding to the target object in the target binarized image data is determined; For each reference binarized image data, a preset window is traversed and moved according to a preset movement trajectory within the reference binarized image data to select multiple bounding regions. The size of the preset window is the same as the size of the target region. For each bounding region, based on the grayscale values ​​of multiple reference pixels in the bounding region and the grayscale values ​​of multiple target pixels in the target region, a region matching degree between the bounding region and the target region is determined. The bounding region with the largest corresponding region matching degree is determined as the bounding region to be adjusted. Based on the nearest point search (ICP) algorithm, the bounding region to be adjusted, the reference binarized image data, and the target region, a matching region in the reference binarized image data that matches the target region is determined. The region matching degree between the matching region and the target region is greater than or equal to the region matching degree between the bounding region to be adjusted and the target region. The region matching degree between the matching region and the target region is determined as the matching degree between the reference image data corresponding to the reference binarized image data and the target image data. The reference image data with the highest matching degree with the target image data is determined as the reference image data that matches the target image data; Obtain the encoder information corresponding to the conveyor belt through the encoder; Based on the Kalman filter, the pose information corresponding to the reference image data that matches the target image data and the encoder information are fused to obtain the pose information corresponding to the target image data. The robot is controlled to work based on the pose information corresponding to the target image data.

2. A robot control device, characterized in that, The device includes: The first acquisition module is used to acquire multiple reference image data of the target object, each reference image data corresponding to different pose information; The second acquisition module is used to acquire target image data of the target object through a camera; The first determining module is configured to: binarize the reference image data to obtain reference binarized image data; binarize the target image data to obtain target binarized image data; determine the target region corresponding to the target object in the target binarized image data based on the target binarized image data and a visual tracking algorithm; for each reference binarized image data, move a preset window through the reference binarized image data according to a preset movement trajectory to select multiple bounding regions, wherein the size of the preset window is the same as the size of the target region; for each bounding region, determine the region between the bounding region and the target region based on the gray values ​​of multiple reference pixels in the bounding region and the gray values ​​of multiple target pixels in the target region. Matching degree; the region with the largest corresponding region matching degree value is determined as the region to be adjusted; based on the nearest point search (ICP) algorithm, the region to be adjusted, the reference binarized image data, and the target region, a matching region in the reference binarized image data that matches the target region is determined, wherein the region matching degree between the matching region and the target region is greater than or equal to the region matching degree between the region to be adjusted and the target region; the region matching degree between the matching region and the target region is determined as the matching degree between the reference image data corresponding to the reference binarized image data and the target image data; the reference image data with the largest matching degree with the target image data is determined as the reference image data that matches the target image data; The second determining module is used to obtain encoder information corresponding to the conveyor belt through the encoder; and to perform data fusion processing on the pose information corresponding to the reference image data that matches the target image data and the encoder information based on the Kalman filter to obtain the pose information corresponding to the target image data. The control module is used to control the robot to work based on the pose information corresponding to the target image data.

3. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing at least one instruction, which is loaded and executed by the processor to perform the operation performed by the robot control method as described in claim 1.

4. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction, which is loaded and executed by a processor to perform the operation of the robot control method as described in claim 1.

Images