Robot control method, device and equipment and storage medium

Abstract

The invention provides a robot control method and device, equipment and a storage medium, and belongs to the technical field of robots. The method comprises the following steps: acquiring multiple pieces of reference image data of a target object, wherein each piece of reference image data corresponds to different pose information; obtaining target image data of the target object through a camera; determining reference image data matched with the target image data based on the matching degree between the target image data and each reference image data; determining pose information corresponding to the target image data based on pose information corresponding to reference image data matched with the target image data; and controlling the robot to work based on the pose information corresponding to the target image data. By the adoption of the method and device, the robot can directly judge the current spatial position and posture of the target object according to the posture information, operation such as assembly or detection can be conducted on the target object more accurately, and therefore the operation accuracy of the robot is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of robotics, and particularly to a robot control method, apparatus, device, and storage medium. Background Art

[0002] The use of conveyor belts on industrial production lines has greatly accelerated the product assembly speed. However, with the increase in labor costs and the requirements for product assembly accuracy, the original manual assembly work can no longer meet the current needs. Therefore, industrial robots have emerged.

[0003] The control method of robots is usually as follows: an encoder is fixed on the rotating shaft of the conveyor belt, and the conveyor distance of the conveyor belt is determined in real time through the encoder. When a product is placed on the conveyor belt, the initial position of a product is determined. The robot can determine the position of the product based on the initial position of the product and the real-time conveyor distance sent by the encoder, so that the robot can perform intelligent operations such as assembling or detecting the product.

[0004] However, the above control method is only applicable to the situation where the position or attitude of the product on the conveyor belt does not change. Once the conveyor belt is unstable or the product on the conveyor belt is accidentally touched by the staff, both will cause the position and angle of the product to change, resulting in a large error when the robot performs operations and reducing the accuracy of the robot operations. Summary of the Invention

[0005] Embodiments of the present disclosure provide a robot control method, which can improve the accuracy of robot operations. The technical solution is as follows:

[0006] In a first aspect, a robot control method is provided, and the method includes:

[0007] Obtain a plurality of reference image data of a target object, and each reference image data corresponds to different pose information;

[0008] Obtain target image data of the target object through a camera;

[0009] Based on the matching degree between the target image data and each reference image data, determine the reference image data that matches the target image data;

[0010] Based on the pose information corresponding to the reference image data that matches the target image data, determine the pose information corresponding to the target image data;

[0011] Control the robot to work based on the pose information corresponding to the target image data.

[0012] In a possible implementation, determining the reference image data that matches the target image data based on the matching degree between the target image data and each reference image data includes:

[0013] Perform binarization processing on the reference image data to obtain reference binarized image data;

[0014] Perform binarization processing on the target image data to obtain target binarized image data;

[0015] For each reference binarized image data, based on the gray values of multiple reference pixel points in the reference binarized image data and the gray values of multiple target pixel points in the target binarized image data, determine the matching degree between the reference image data corresponding to the reference binarized image data and the target image data;

[0016] Determine the reference image data with the largest matching degree with the target image data as the reference image data that matches the target image data.

[0017] In a possible implementation, the method further includes:

[0018] Based on the target binarized image data and a visual tracking algorithm, determine the target area corresponding to the target object in the target binarized image data;

[0019] The determining the matching degree between the reference image data corresponding to the reference binarized image data and the target image data based on the gray values of multiple reference pixel points in the reference binarized image data and the gray values of multiple target pixel points in the target binarized image data includes:

[0020] Based on the gray values of multiple reference pixel points in the reference binarized image data and the gray values of multiple target pixel points in the target area, determine the matching degree between the reference image data corresponding to the reference binarized image data and the target image data.

[0021] In a possible implementation, the determining the matching degree between the reference image data corresponding to the reference binarized image data and the target image data based on the gray values of multiple reference pixel points in the reference binarized image data and the gray values of multiple target pixel points in the target area includes:

[0022] Based on a preset window, repeatedly frame out multiple framed areas in the reference binarized image data, where the size of the preset window is the same as the size of the target area;

[0023] For each selected area, based on the gray values of multiple reference pixel points in the selected area and the gray values of multiple target pixel points in the target area, determine the area matching degree between the selected area and the target area;

[0024] Based on the area matching degrees between the multiple selected areas and the target area, determine the matching degree between the reference image data corresponding to the reference binary image data and the target image data.

[0025] In a possible implementation manner, the determining the matching degree between the reference image data corresponding to the reference binary image data and the target image data based on the area matching degrees between the multiple selected areas and the target area includes:

[0026] Determine the area matching degree with the largest value among the area matching degrees corresponding to the multiple selected areas as the matching degree between the reference image data corresponding to the reference binary image data and the target image data.

[0027] In a possible implementation manner, the determining the matching degree between the reference image data corresponding to the reference binary image data and the target image data based on the area matching degrees between the multiple selected areas and the target area includes:

[0028] Determine the selected area with the largest value of the corresponding area matching degree as the selected area to be adjusted;

[0029] Based on the ICP (Iterative Closest Point) algorithm, the selected area to be adjusted, the reference binary image data, and the target area, determine the matching area in the reference binary image data that matches the target area, where the area matching degree between the matching area and the target area is greater than or equal to the area matching degree between the selected area to be adjusted and the target area;

[0030] Determine the area matching degree between the matching area and the target area as the matching degree between the reference image data corresponding to the reference binary image data and the target image data.

[0031] In a possible implementation manner, the method further includes:

[0032] Determine multiple feature points and multiple feature lines corresponding to the target object;

[0033] For each reference image data, determine the position information corresponding to each feature point and the position information corresponding to each feature line in the reference image data;

[0034] Determining a matching degree between the reference image data corresponding to the reference binary image data and the target image data based on the gray values of a plurality of reference pixel points in the reference binary image data and the gray values of a plurality of target pixel points in the target binary image data includes:

[0035] For each piece of reference binary image data, adjust the gray values of the reference pixel points other than the feature points and the feature lines to 0 to obtain the reference adjusted image data corresponding to the reference binary image data;

[0036] Perform edge detection processing on the target binary image data to obtain a plurality of edge positions in the target binary image data;

[0037] In the target binary image data, adjust the gray values of the target pixel points other than the edge positions to 0 to obtain the target adjusted image data corresponding to the target binary image data;

[0038] Determine a matching degree between the reference image data corresponding to the reference adjusted image data and the target image data based on the gray values of a plurality of reference pixel points in the reference adjusted image data and the gray values of a plurality of target pixel points in the target adjusted image data.

[0039] In a possible implementation manner, the method further includes:

[0040] Obtain encoder information corresponding to the conveyor belt through an encoder;

[0041] The determining the pose information corresponding to the target image data based on the pose information corresponding to the reference image data matching the target image data includes:

[0042] Based on a Kalman filter, perform data fusion processing on the pose information corresponding to the reference image data matching the target image data and the encoder information to obtain the pose information corresponding to the target image data.

[0043] In a second aspect, a robot control device is provided, and the device includes:

[0044] A first acquisition module, configured to acquire a plurality of reference image data of a target object, and each piece of reference image data corresponds to different pose information;

[0045] A second acquisition module, configured to acquire target image data of the target object through a camera;

[0046] A first determination module, configured to determine the reference image data matching the target image data based on the matching degree between the target image data and each piece of reference image data;

[0047] A second determination module, configured to determine the pose information corresponding to the target image data based on the pose information corresponding to the reference image data that matches the target image data;

[0048] A control module, configured to control the robot to work based on the pose information corresponding to the target image data.

[0049] In a possible implementation, the first determination module is configured to:

[0050] Perform binarization processing on the reference image data to obtain reference binarized image data;

[0051] Perform binarization processing on the target image data to obtain target binarized image data;

[0052] For each reference binarized image data, determine the matching degree between the reference image data corresponding to the reference binarized image data and the target image data based on the gray values of multiple reference pixel points in the reference binarized image data and the gray values of multiple target pixel points in the target binarized image data;

[0053] Determine the reference image data with the largest matching degree with the target image data as the reference image data that matches the target image data.

[0054] In a possible implementation, the apparatus further includes a third determination module, configured to:

[0055] Based on the target binarized image data and a visual tracking algorithm, determine the target area corresponding to the target object in the target binarized image data;

[0056] The first determination module is configured to:

[0057] Determine the matching degree between the reference image data corresponding to the reference binarized image data and the target image data based on the gray values of multiple reference pixel points in the reference binarized image data and the gray values of multiple target pixel points in the target area.

[0058] In a possible implementation, the first determination module is configured to:

[0059] Based on a preset window, repeatedly frame out multiple framed areas in the reference binarized image data, where the size of the preset window is the same as the size of the target area;

[0060] For each framed area, determine the area matching degree between the framed area and the target area based on the gray values of multiple reference pixel points in the framed area and the gray values of multiple target pixel points in the target area;

[0061] Based on the region matching degree between the multiple selected regions and the target region, determine the matching degree between the reference image data corresponding to the reference binary image data and the target image data.

[0062] In a possible implementation manner, the first determination module is configured to:

[0063] Determine the region matching degree with the largest value among the region matching degrees corresponding to the multiple selected regions as the matching degree between the reference image data corresponding to the reference binary image data and the target image data.

[0064] In a possible implementation manner, the first determination module is configured to:

[0065] Determine the selected region with the largest value of the corresponding region matching degree as the selected region to be adjusted;

[0066] Based on the ICP algorithm, the selected region to be adjusted, the reference binary image data, and the target region, determine the matching region in the reference binary image data that matches the target region, where the region matching degree between the matching region and the target region is greater than or equal to the region matching degree between the selected region to be adjusted and the target region;

[0067] Determine the region matching degree between the matching region and the target region as the matching degree between the reference image data corresponding to the reference binary image data and the target image data.

[0068] In a possible implementation manner, the device further includes a fourth determination module, configured to:

[0069] Determine a plurality of feature points and a plurality of feature lines corresponding to the target object;

[0070] For each reference image data, determine the position information corresponding to each feature point and the position information corresponding to each feature line in the reference image data;

[0071] The first determination module is configured to:

[0072] For each reference binary image data, adjust the gray values of the reference pixel points other than the feature points and the feature lines to 0 to obtain the reference adjusted image data corresponding to the reference binary image data;

[0073] Perform edge detection processing on the target binary image data to obtain a plurality of edge positions in the target binary image data;

[0074] In the target binary image data, adjust the gray values of the target pixel points outside the edge positions to 0 to obtain target adjusted image data corresponding to the target binary image data;

[0075] Based on the gray values of multiple reference pixel points in the reference adjusted image data and the gray values of multiple target pixel points in the target adjusted image data, determine the matching degree between the reference image data corresponding to the reference adjusted image data and the target image data.

[0076] In a possible implementation, the apparatus further includes a third acquisition module, configured to:

[0077] Obtain encoder information corresponding to the conveyor belt through an encoder;

[0078] The second determination module is configured to:

[0079] Based on a Kalman filter, perform data fusion processing on the pose information corresponding to the reference image data that matches the target image data and the encoder information to obtain the pose information corresponding to the target image data.

[0080] In a third aspect, a computer device is provided. The computer device includes a processor and a memory. At least one instruction is stored in the memory, and the instruction is loaded and executed by the processor to implement the operations performed by the robot control method.

[0081] In a fourth aspect, a computer-readable storage medium is provided. At least one instruction is stored in the storage medium, and the instruction is loaded and executed by the processor to implement the operations performed by the robot control method.

[0082] In a fifth aspect, a computer program product is provided. The computer program product includes at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the operations performed by the robot control method.

[0083] The beneficial effects brought by the technical solutions provided in the embodiments of the present disclosure are as follows: In the solutions mentioned in the embodiments of the present disclosure, the pose information corresponding to the target image data is determined. In this way, the robot can directly judge the current spatial position and attitude of the target object based on the pose information, and can perform operations such as assembling or detecting the target object more accurately, thereby improving the accuracy of the robot operation. Description of the Drawings

[0084] To more clearly illustrate the technical solutions in the embodiments of the present disclosure, the following briefly introduces the accompanying drawings required for the description of the embodiments. Obviously, the accompanying drawings in the following description are only some embodiments of the present disclosure. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can be obtained based on these drawings.

[0085] Figure 1 is a flowchart of a robot control method provided by an embodiment of the present disclosure;

[0086] Figure 2 is a flowchart of a robot control method provided by an embodiment of the present disclosure;

[0087] Figure 3 is a flowchart of a robot control method provided by an embodiment of the present disclosure;

[0088] Figure 4 is a schematic diagram of a target area provided by an embodiment of the present disclosure;

[0089] Figure 5 is a flowchart of a robot control method provided by an embodiment of the present disclosure;

[0090] Figure 6 is a flowchart of a method for determining a matching degree provided by an embodiment of the present disclosure;

[0091] Figure 7 is a flowchart of a method for determining a matching degree provided by an embodiment of the present disclosure;

[0092] Figure 8 is a flowchart of a robot control method provided by an embodiment of the present disclosure;

[0093] Figure 9 is a schematic structural diagram of a robot control device provided by an embodiment of the present disclosure;

[0094] Figure 10 is a block diagram of the structure of a terminal provided by an embodiment of the present disclosure;

[0095] Figure 11 is a block diagram of the structure of a server provided by an embodiment of the present disclosure. Specific Embodiments

[0096] To make the objectives, technical solutions, and advantages of the present disclosure clearer, the following further describes the embodiments of the present disclosure in detail with reference to the accompanying drawings.

[0097] Embodiments of the present disclosure provide a robot control method, which can be implemented by a computer device. The computer device can be a terminal, a server, etc. The terminal can be a desktop computer, a notebook computer, a tablet computer, a mobile phone, etc. The server can be a single server or a server cluster, etc.

[0098] The computer device may include a processor, a memory, a communication component, etc.

[0099] The processor may be a central processing unit (CPU). The processor can be used to read instructions and process data. For example, obtaining multiple reference image data of a target object, obtaining target image data of the target object, determining reference image data that matches the target image data item, determining the pose information corresponding to the target image data, controlling the robot to work, and so on.

[0100] The memory can be various volatile memories or non-volatile memories, such as a solid state disk (SSD), a dynamic random access memory (DRAM), etc. The memory can be used for data storage. For example, storing multiple reference image data of the obtained target object, storing the target image data of the obtained target object, storing the determined reference image data that matches the target image data, storing the pose information corresponding to the determined target image data, storing the intermediate data generated during the process of controlling the robot to work, and so on.

[0101] The communication component can be a wired network connector, a wireless fidelity (WiFi) module, a Bluetooth module, a cellular network communication module, etc. The communication component can be used for data transmission with other devices. For example, receiving the target image data of the target object sent by a camera, and so on.

[0102] The robot control method provided by the present disclosure can be applied to some industrial production lines equipped with robots, and is used to control the robot to perform operations such as assembling and detecting products. Among them, the robot can be various types of robots on the industrial production line, such as a robotic arm, and so on.

[0103] Figure 1 and Figure 2 are the flowcharts of a robot control method provided by embodiments of the present disclosure. Refer to Figure 1 and Figure 2 , this embodiment includes:

[0104] 101. Obtain multiple reference image data of a target object, and each reference image data corresponds to different pose information.

[0105] In implementation, at various angles and positions of 6D degrees of freedom, the target object can be photographed by a camera to obtain multiple reference image data of the target object, and each reference image data corresponds to different pose information. It can be understood that the pose information corresponding to the reference image data refers to the pose information of the target object in the reference image data.

[0106] Alternatively, a 3D model of the target object can also be established, and through computer vision algorithms, reference image data of the target object at different pose information can be obtained.

[0107] In a possible implementation manner, the number of reference image data of the target object can be set according to requirements. For example, it can be 10,000, 50,000, etc. The embodiments of the present disclosure do not make specific limitations on this.

[0108] 102. Obtain the target image data of the target object through a camera.

[0109] In implementation, on an industrial production line, when a robot needs to operate on a target object, the target object can be photographed by a camera to obtain the target image data of the target object.

[0110] Among them, the camera can be fixed at any reasonable position on the industrial production line. For example, it can be fixed at a fixed position, or it can be fixed at the end of the robotic arm of the robot, etc. The embodiments of the present disclosure do not make specific limitations on this.

[0111] 103. Based on the matching degree between the target image data and each reference image data, determine the reference image data that matches the target image data.

[0112] In implementation, after obtaining the target image data of the target object, the matching degree between the target image data and each reference image data can be calculated respectively.

[0113] Then, determine the reference image data with the largest matching degree value. Since among these multiple reference image data, the matching degree between this reference image data and the target image data is the largest, it indicates that the pose information of the target object in the target image data is also the most similar to the pose information of the target object in the reference image data with the largest matching degree. Therefore, the reference image data with the largest matching degree value between the target image data can be determined as the reference image data that matches the target image data.

[0114] In a possible implementation manner, the pose information may include obtaining the three-dimensional coordinates of the centroid of the target object, as well as the roll angle, pitch angle, and yaw angle of the centroid.

[0115] 104. Determine the pose information corresponding to the target image data based on the pose information corresponding to the reference image data that matches the target image data.

[0116] In implementation, among multiple reference image data, the matching degree of the reference image data that matches the target image data is the largest, indicating that this reference image data is the most similar to the target image data. The pose information of the target object in this reference image data is also relatively similar to the pose information of the target object in the target image data. Therefore, the pose information corresponding to the target image data can be determined based on the pose information corresponding to this reference image data with the largest matching degree value. In this way, the pose information corresponding to the target image data can be obtained more accurately.

[0117] It can be understood that the pose information corresponding to the target image data refers to the pose information of the target object in the target image data.

[0118] 105. Control the robot to work based on the pose information corresponding to the target image data.

[0119] In implementation, after determining the pose information corresponding to the target image data, that is, obtaining the pose information of the target object in the target image data, send the pose information corresponding to the target image data to the robot controller. The robot control generates a control signal based on the obtained pose information corresponding to the target image data, sends the control signal to the motion control operator module of the robot, generates a final control instruction through the PID algorithm combined with the robot model, and the robot moves based on this control instruction, thereby realizing the operation on the target object.

[0120] During the process of controlling the robot operation, the above steps 101 - 105 are processed periodically, thereby realizing the visual closed-loop control of the robot. Based on the operation of the robot, new target image data is obtained, and then the pose information corresponding to the new target image data is determined, thereby realizing the closed-loop adjustment of the robot operation and improving the accuracy of its operation.

[0121] In summary, the solution mentioned in the embodiments of the present disclosure can determine the pose information corresponding to the target image data. In this way, the robot can directly judge the current spatial position and posture of the target object according to the pose information, and can perform operations such as assembling or detecting the target object more accurately, thereby improving the accuracy of the robot operation.

[0122] Moreover, for the solutions mentioned in the embodiments of the present disclosure, there is no need to limit the type of the target object, which can be an object with any structure or shape. Thus, it can be adapted to most industrial follow-up assembly and inspection production lines, liberating manual labor, saving manpower, and having high consistency in the operation rhythm, intensity, and work quality, improving the efficiency of the industrial production line and the qualified rate of finished products.

[0123] In the above step 103, there are various methods for determining the reference image data that matches the target image data based on the matching degree between the target image data and each reference image data. Below, an introduction is given to them:

[0124] See Figure 3 , and the method can be: performing binarization processing on the reference image data to obtain reference binarized image data; performing binarization processing on the target image data to obtain target binarized image data; for each reference binarized image data, determining the matching degree between the reference image data corresponding to the reference binarized image data and the target image data based on the gray values of multiple reference pixel points in the reference binarized image data and the gray values of multiple target pixel points in the target binarized image data; determining the reference image data with the largest matching degree with the target image data as the reference image data that matches the target image data.

[0125] In implementation, the reference image data can be first converted into a grayscale image, and then based on the binarization threshold, the gray values of the reference pixel points greater than or equal to the binarization threshold in the grayscale image corresponding to the reference image data are all adjusted to 1, and the gray values of the reference pixel points less than the binarization threshold in the grayscale image corresponding to the reference image data are all adjusted to 0, thereby obtaining the reference binarized image data composed of 1 and 0 gray values.

[0126] Similarly, for the target image data, the target image data can be first converted into a grayscale image, and then based on the binarization threshold, the gray values of the target pixel points greater than or equal to the binarization threshold in the grayscale image corresponding to the target image data are all adjusted to 1, and the gray values of the target pixel points less than the binarization threshold in the grayscale image corresponding to the target image data are all adjusted to 0, thereby obtaining the target binarized image data composed of 1 and 0 gray values.

[0127] Then, for each reference binarized image data, the matching degree between the reference image data corresponding to the reference binarized image data and the target image data can be determined based on the gray values of multiple reference pixel points in the reference binarized image data and the gray values of multiple target pixel points in the target binarized image data.

[0128] The matching degree between each reference binary image data and the target binary image data determined by the above method is the matching degree between each reference image data and the target image data.

[0129] The reference image data with the largest numerical value of the matching degree with the target image data is determined as the reference image data that matches the target image data.

[0130] In the above method, the reference image data and the target image data are binarized to obtain the reference binary image data and the target binary image data. In the reference binary image data and the target binary image data, the contour of the target object is relatively distinct. Determining the matching degree based on the reference binary image data and the target binary image data can ensure the accuracy of the matching degree while improving the calculation speed, thereby achieving more accurate and real-time control of the robot.

[0131] In a possible implementation manner, there are various methods for determining the matching degree between the reference image data and the target image data based on the gray values of multiple reference pixel points in the reference binary image data and the gray values of multiple target pixel points in the target binary image data. Two of them are introduced below:

[0132] The first method for determining the matching degree: Input the reference binary image data and the target binary image data into the matching degree prediction model to obtain the predicted matching degree output. This predicted matching degree is the matching degree between the reference image data corresponding to the reference binary image data and the target image data corresponding to the target binary image data.

[0133] The above matching degree prediction model can be a machine learning model with any reasonable structure, and the embodiments of the present disclosure do not make specific limitations on this.

[0134] The second method for determining the matching degree: Based on the first preset window, perform multiple selections in the reference binary image data to obtain multiple selected areas. The size of the first preset window can be the same as the size of the target binary image data.

[0135] Then, for each selected area, the matching degree between the selected area and the target image data can be determined based on the gray values of multiple reference pixel points in the selected area and the gray values of multiple target pixel points in the target image data.

[0136] The matching degrees between multiple selected areas and the target image data are obtained through the above method, and the matching degree with the largest numerical value among these multiple matching degrees is determined as the matching degree between the reference image data and the target image data.

[0137] The above two methods for determining the matching degree are only examples. The method for determining the matching degree between the reference image data and the target image data in the embodiments of the present disclosure may also be any other reasonable method, and the embodiments of the present disclosure do not make specific limitations thereto.

[0138] In a possible implementation manner, the robot control method provided by the embodiments of the present disclosure may further include the following steps: determining a target area corresponding to a target object in the target binary image data based on the target binary image data and a visual tracking algorithm.

[0139] In implementation, it is necessary to first initialize the target object for the visual tracking algorithm, that is, first obtain the grayscale image data of the target object and the area where the target object is located in the grayscale image data, then input the grayscale image data of the target object into the visual tracking algorithm, and initialize and adjust each parameter in the visual tracking algorithm based on the area where the target object is located.

[0140] Then, the initialized visual tracking algorithm can be used to process the target binary image data, so as to obtain a target area corresponding to the target object in the target binary image data. See Figure 4 .

[0141] In this way, other objects or background parts in the target binary image data except the target object can be removed, avoiding subsequent redundant calculations.

[0142] After determining the target area corresponding to the target object, when performing the above step (determining the matching degree between the reference image data corresponding to the reference binary image data and the target image data based on the grayscale values of multiple reference pixel points in the reference binary image data and the grayscale values of multiple target pixel points in the target binary image data), see Figure 5 , the following processing can be performed: determining the matching degree between the reference image data corresponding to the reference binary image data and the target image data based on the grayscale values of multiple reference pixel points in the reference binary image data and the grayscale values of multiple target pixel points in the target area.

[0143] In implementation, after determining the target area where the target object is located in the target binary image data, the reference binary image data and the target area can be directly matched, so as to obtain the matching degree between each reference image data and the target area, that is, the matching degree between each reference image data and the target image data.

[0144] In this way, the matching calculation of other areas irrelevant to the target object in the target binary image data is avoided, greatly reducing the calculation amount and improving the speed of determining the matching degree, so as to achieve more accurate and real-time control of the robot.

[0145] In this case, there can be various methods for determining the degree of match between the reference image data and the target image data. For example:

[0146] The first method for determining the degree of match: Input the reference binary image data and the image data corresponding to the target area into the degree-of-match prediction model to obtain the predicted degree of match as the output, and this predicted degree of match is the degree of match between the reference image data and the target image data.

[0147] The above-mentioned degree-of-match prediction model can be a machine learning model with any reasonable structure, and the embodiments of the present disclosure do not make specific limitations on this.

[0148] The second method for determining the degree of match: Based on a preset window, multiple framed areas are repeatedly framed out in the reference binary image data, where the size of the preset window is the same as the size of the target area; for each framed area, based on the gray values of multiple reference pixel points in the framed area and the gray values of multiple target pixel points in the target area, determine the regional degree of match between the framed area and the target area; based on the regional degrees of match between multiple framed areas and the target area, determine the degree of match between the reference image data corresponding to the reference binary image data and the target image data.

[0149] In implementation, a preset window with the same size as the target area can be set first, and then based on this preset window, frame selection is performed in the reference binary image data to obtain the first framed area. After one frame selection, move the preset window in the reference binary image data according to the preset movement trajectory. After moving, the preset window frames out the second framed area, and then the preset window moves in the reference binary image data according to the preset movement trajectory again to frame out the third framed area, and so on. The preset window is traversed and moved in the reference binary image data to obtain multiple framed areas.

[0150] For example, the preset movement trajectory is: starting from the upper left corner of the reference binary image data, first move to the right. After moving to the rightmost side of the reference binary image data, move down once, then start moving to the left until moving to the leftmost side, then move down once again, and then start moving to the right, and so on, forming an S-shaped preset movement trajectory.

[0151] The moving distance for each movement can be any reasonable distance set in advance. For example, each movement can move a distance of three pixels, or a distance of two pixels, etc. The embodiments of the present disclosure do not make specific limitations on this.

[0152] After obtaining multiple boxed areas based on the above method, each boxed area is respectively matched with the target area to obtain the area matching degree between each boxed area and the target area. Then, based on these multiple area matching degrees, the matching degree between the reference image data and the target image data is determined.

[0153] In the embodiments of the present disclosure, there are various methods for determining the matching degree between the reference image data and the target image data based on multiple area matching degrees. The following are two examples:

[0154] The first method for determining the matching degree: Refer to Figure 6 , and determine the area matching degree with the largest value among the area matching degrees corresponding to the multiple boxed areas as the matching degree between the reference image data corresponding to the reference binary image data and the target image data.

[0155] The second method for determining the matching degree: Refer to Figure 7 , determine the boxed area with the largest value of the corresponding area matching degree as the boxed area to be adjusted; based on the ICP algorithm, the boxed area to be adjusted, the reference binary image data, and the target area, determine the matching area in the reference binary image data that matches the target area, where the area matching degree between the matching area and the target area is greater than or equal to the area matching degree between the boxed area to be adjusted and the target area; determine the area matching degree between the matching area and the target area as the matching degree between the reference image data corresponding to the reference binary image data and the target image data.

[0156] In implementation, when the moving distance of the above preset window is relatively large, the boxed area with the largest value of the area matching degree may not be exactly corresponding to the target area, and there may be a dislocation between the two. Therefore, after determining the boxed area to be adjusted, the reference binary image data, the position of the boxed area to be adjusted in the reference binary image data, and the target area can be input into the ICP algorithm for iterative matching, so as to perform a finer moving matching with a smaller moving distance on the image data around the boxed area to be adjusted in the binary image data, thereby obtaining a matching area that is exactly corresponding to the image data of the target area. In this way, the area matching degree between the obtained matching area and the target area will be greater than or equal to the area matching degree between the boxed area to be adjusted and the target area.

[0157] For example, when the moving distance is 3 pixel points when calculating the matching degree between each boxed area and the target area based on the preset window, in the ICP algorithm, it can be set to move one pixel point each time. After inputting the reference binary image data, the position of the boxed area to be adjusted in the reference binary image data, and the target area into the ICP algorithm, based on the preset window, start moving in the reference binary image data from the boxed area to be adjusted, moving only one pixel point each time. After moving one pixel point to the right for the first time, the first boxed area is obtained, and then calculate the area matching degree between the first boxed area and the target area. If the calculated area matching degree is less than the area matching degree between the boxed area to be adjusted and the target area, it means that moving to the right is incorrect. For the next move, start moving one pixel point to the left from the boxed area to be adjusted to obtain the second boxed area, and then calculate the area matching degree between the second boxed area and the target area. If the calculated area matching degree is greater than the matching degree between the boxed area to be adjusted and the target area, it means that moving to the left is correct. Then, for the third move, continue to move one pixel point to the left. Calculate the area matching degree between the boxed area obtained after each move and the target area. Move to the left until the calculated area matching degree is less than the area matching degree calculated after the previous move, which means that the boxed area obtained after the previous move is relatively more matched with the target area. Then, start moving downward from the boxed area obtained after the previous move. Similar to moving left and right, if the area matching degree obtained after moving downward increases, it means that it can continue to move downward until the position with the highest area matching degree. If the area matching degree obtained after moving downward decreases, it means that the moving direction is incorrect. At this time, move upward until the area matching degree increases to the highest position, thereby obtaining the matching area corresponding to the target area.

[0158] Based on this method, more precise positioning can be performed to obtain a matching area with a higher matching degree with the target area.

[0159] After determining the matching area, the area matching degree between the matching area and the target area can be determined as the matching degree between the reference image data corresponding to the reference binary image data and the target image data.

[0160] The above methods for determining the matching degree are only several listed methods. The method for determining the matching degree between the reference image data and the target image data in the embodiments of the present disclosure can be any one or a combination of several of the above, or can also be any other reasonable method. The embodiments of the present disclosure do not make specific limitations in this regard.

[0161] In a possible implementation manner, the robot control method provided by the embodiments of the present disclosure may further include the following steps: determining a plurality of feature points and feature lines corresponding to the target object; for each reference image data, determining the position information corresponding to each feature point and the position information corresponding to each feature line in the reference image data.

[0162] In implementation, the staff can preset a plurality of feature points and a plurality of feature lines corresponding to the target object. Among them, the feature points may include the center points of the bottoms of the grooves on the target object, the center points of the areas of the through holes on the surface of the target object, the tip points on the outer contour of the target object, and so on.

[0163] The feature lines may include the axes of the grooves on the target object, the axes of the through holes, the outer contour lines of the target object, and so on.

[0164] The feature points and feature lines can be selected according to the actual shape features of the target object, and the embodiments of the present disclosure do not make specific limitations thereto.

[0165] The position information of the feature points is the three-dimensional coordinates of the feature points, and the position information of the feature lines can be the three-dimensional coordinates of multiple points on the feature lines.

[0166] After determining the feature points and feature lines of the target object, when performing the above steps (determining the matching degree between the reference image data corresponding to the reference binary image data and the target image data based on the gray values of multiple reference pixel points in the reference binary image data and the gray values of multiple target pixel points in the target binary image data), the following processing can be performed: for each reference binary image data, adjusting the gray values of the reference pixel points other than the feature points and feature lines to 0 to obtain the reference adjusted image data corresponding to the reference binary image data; performing edge detection processing on the target binary image data to obtain the edge positions in the target binary image data; in the target binary image data, adjusting the gray values of the target pixel points other than the edge positions to 0 to obtain the target adjusted image data corresponding to the target binary image data; determining the matching degree between the reference image data corresponding to the reference adjusted image data and the target image data based on the gray values of multiple reference pixel points in the reference adjusted image data and the gray values of multiple target pixel points in the target adjusted image data.

[0167] In implementation, based on computer vision algorithms, before taking a photo of the 3D model of the target object to obtain the reference image data, the feature points and feature lines in the 3D model can be marked first, and then the camera is used to take a photo of the 3D model to obtain each reference image data and the positions of the feature points and feature lines in the reference image data.

[0168] Then, perform binarization processing on each reference image data to obtain the reference binarized image data corresponding to each reference image data. Adjust the gray values of the reference pixel points except for the feature points and feature lines in each reference binarized image data to 0 to obtain the reference adjusted image data corresponding to each reference binarized image data, and store it.

[0169] When calculating the matching degree, edge detection processing can be performed on the target binarized image data first to obtain the edge positions in the target binarized image data. The edge positions are the more obvious positions in the target binarized image, such as the more obvious positions like the contour of the target object. The edge positions can include the positions of points, the positions of lines composed of multiple points, or other reasonable settings, which are not limited in the embodiments of the present disclosure.

[0170] In the target binarized image data, directly adjust the gray values of the target pixel points except for the edge positions to 0 to obtain the target adjusted image data. In this target adjusted image data, only the gray values of the target pixel points at the more obvious edge positions are 1, and the gray values of other target pixel points are 0.

[0171] Then, based on the reference adjusted image data and the target adjusted image data, the matching degree between the reference image data corresponding to the reference binarized image data and the target image data can be determined.

[0172] With such a setting, when determining the matching degree subsequently, the calculation speed can be increased, and the interference of other backgrounds can be avoided, improving the accuracy of matching.

[0173] In the embodiments of the present disclosure, the method for determining the matching degree between the reference image data and the target image data based on the reference adjusted image data and the target adjusted image data can be any one or a combination of several of the methods for determining the matching degree based on the reference binarized image data and the target binarized image data introduced above, which will not be elaborated here.

[0174] In the embodiments of the present disclosure, there are various methods for determining the pose information corresponding to the target image data based on the pose information corresponding to the reference image data that matches the target image data. The following is an introduction to them:

[0175] In a possible implementation manner, after determining the reference image data that matches the target image data, the pose information corresponding to the reference image data that matches the target image data can be directly determined as the pose information corresponding to the target image data.

[0176] In another possible implementation manner, the robot control method provided by the embodiments of the present disclosure may further include the following steps: obtaining the encoder information corresponding to the conveyor belt through an encoder.

[0177] In implementation, since conveyors are usually used to transport target objects in industrial production lines, an encoder can be installed on the roller corresponding to the conveyor. When the roller drives the conveyor to move, the encoder will generate encoder information in real time, and this encoder information can reflect the moving distance of the conveyor.

[0178] In this way, when performing the above steps (determining the pose information corresponding to the target image data based on the pose information corresponding to the reference image data matching the target image data), the following processing can be performed: Based on the Kalman filter, perform data fusion processing on the pose information corresponding to the reference image data matching the target image data and the encoder information to obtain the pose information corresponding to the target image data.

[0179] In implementation, the conveying direction of the conveyor can be determined first, and then after determining the pose information corresponding to the reference image data matching the target image data, calculate the component of this pose information in the conveying direction, and input the obtained component and the encoder information into the Kalman filter for Kalman data fusion processing, so as to obtain the position information of the centroid of the target object in the conveying direction of the output.

[0180] Adjust the information related to the component in the conveying direction in the pose information based on the position information of the centroid of the target object in the conveying direction to obtain the adjusted pose information. Then, the adjusted pose information can be determined as the pose information corresponding to the target image data.

[0181] Since the frequency of the encoder is usually high, the encoder information with strong real-time performance is used to adjust the pose information, thereby improving the accuracy of determining the pose information corresponding to the target image data.

[0182] All the above optional technical solutions can be combined arbitrarily to form optional embodiments of the present disclosure, which will not be elaborated one by one here.

[0183] In the embodiments of the present disclosure, the pose information corresponding to the target image data is determined. In this way, the robot can directly judge the current spatial position and posture of the target object according to the pose information, and can perform operations such as assembling or detecting the target object more accurately, thereby improving the accuracy of the robot operation.

[0184] The embodiments of the present disclosure provide a robot control device, and this device can be the computer device in the above embodiments, such as Figure 9 shown, the device includes:

[0185] A first acquisition module 910, configured to acquire multiple reference image data of a target object, and each reference image data corresponds to different pose information;

[0186] A second acquisition module 920, configured to acquire target image data of the target object through a camera;

[0187] A first determination module 930, configured to determine reference image data that matches the target image data based on the degree of match between the target image data and each reference image data;

[0188] A second determination module 940, configured to determine the pose information corresponding to the target image data based on the pose information corresponding to the reference image data that matches the target image data;

[0189] A control module 950, configured to control the robot to work based on the pose information corresponding to the target image data.

[0190] In a possible implementation manner, the first determination module 930 is configured to:

[0191] Perform binarization processing on the reference image data to obtain reference binarized image data;

[0192] Perform binarization processing on the target image data to obtain target binarized image data;

[0193] For each reference binarized image data, determine the degree of match between the reference image data corresponding to the reference binarized image data and the target image data based on the gray values of multiple reference pixel points in the reference binarized image data and the gray values of multiple target pixel points in the target binarized image data;

[0194] Determine the reference image data with the largest degree of match with the target image data as the reference image data that matches the target image data.

[0195] In a possible implementation manner, the apparatus further includes a third determination module, configured to:

[0196] Determine a target area corresponding to the target object in the target binarized image data based on the target binarized image data and a visual tracking algorithm;

[0197] The first determination module 930 is configured to:

[0198] Determine the degree of match between the reference image data corresponding to the reference binarized image data and the target image data based on the gray values of multiple reference pixel points in the reference binarized image data and the gray values of multiple target pixel points in the target area.

[0199] In a possible implementation manner, the first determination module 930 is configured to:

[0200] Based on a preset window, multiple selected regions are repeatedly selected from the reference binary image data, wherein the size of the preset window is the same as the size of the target region;

[0201] For each selected region, based on the gray values of multiple reference pixel points in the selected region and the gray values of multiple target pixel points in the target region, determine the region matching degree between the selected region and the target region;

[0202] Based on the region matching degrees between the multiple selected regions and the target region, determine the matching degree between the reference image data corresponding to the reference binary image data and the target image data.

[0203] In a possible implementation manner, the first determination module 930 is configured to:

[0204] Determine the region matching degree with the largest value among the region matching degrees corresponding to the multiple selected regions as the matching degree between the reference image data corresponding to the reference binary image data and the target image data.

[0205] In a possible implementation manner, the first determination module 930 is configured to:

[0206] Determine the selected region with the largest value of the corresponding region matching degree as the selected region to be adjusted;

[0207] Based on the ICP algorithm, the selected region to be adjusted, the reference binary image data, and the target region, determine the matching region in the reference binary image data that matches the target region, wherein the region matching degree between the matching region and the target region is greater than or equal to the region matching degree between the selected region to be adjusted and the target region;

[0208] Determine the region matching degree between the matching region and the target region as the matching degree between the reference image data corresponding to the reference binary image data and the target image data.

[0209] In a possible implementation manner, the device further includes a fourth determination module, configured to:

[0210] Determine multiple feature points and multiple feature lines corresponding to the target object;

[0211] For each reference image data, determine the position information corresponding to each feature point and the position information corresponding to each feature line in the reference image data;

[0212] The first determination module 930 is configured to:

[0213] For each of the reference binary image data, adjust the gray values of the reference pixel points outside the feature points and the feature lines to 0 to obtain the reference adjusted image data corresponding to the reference binary image data;

[0214] Perform edge detection processing on the target binary image data to obtain multiple edge positions in the target binary image data;

[0215] In the target binary image data, adjust the gray values of the target pixel points outside the edge positions to 0 to obtain the target adjusted image data corresponding to the target binary image data;

[0216] Based on the gray values of multiple reference pixel points in the reference adjusted image data and the gray values of multiple target pixel points in the target adjusted image data, determine the matching degree between the reference image data corresponding to the reference adjusted image data and the target image data.

[0217] In a possible implementation manner, the device further includes a third acquisition module, configured to:

[0218] Acquire encoder information corresponding to the conveyor belt through an encoder;

[0219] The second determination module 940 is configured to:

[0220] Based on a Kalman filter, perform data fusion processing on the pose information corresponding to the reference image data that matches the target image data and the encoder information to obtain the pose information corresponding to the target image data.

[0221] It should be noted that when the robot control device provided in the above embodiment controls the robot, only the division of the above functional modules is used for illustration. In practical applications, the above functions can be assigned to different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the robot control device provided in the above embodiment and the embodiment of the robot control method belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0222] Figure 10The block diagram of the terminal 1000 provided by an exemplary embodiment of the present disclosure is shown. The terminal may be the computer device in the above embodiment. The terminal 1000 may be: a smart phone, a tablet computer, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV) player, a laptop computer or a desktop computer. The terminal 1000 may also be referred to by other names such as a user equipment, a portable terminal, a laptop terminal, a desktop terminal, etc.

[0223] Generally, the terminal 1000 includes: a processor 1001 and a memory 1002.

[0224] The processor 1001 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. The processor 1001 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), PLA (Programmable Logic Array). The processor 1001 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the wake state, also known as the CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 1001 may be integrated with a GPU (Graphics Processing Unit), and the GPU is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 1001 may further include an AI (Artificial Intelligence) processor, and the AI processor is used to process computational operations related to machine learning.

[0225] The memory 1002 may include one or more computer-readable storage media, and the computer-readable storage media may be non-transitory. The memory 1002 may also include high-speed random access memory, as well as non-volatile memory, such as one or more disk storage devices, flash storage devices. In some embodiments, the non-transitory computer-readable storage media in the memory 1002 is used to store at least one instruction, and the at least one instruction is used to be executed by the processor 1001 to implement the robot control method provided in the method embodiments of the present disclosure.

[0226] In some embodiments, the terminal 1000 may further optionally include: a peripheral device interface 1003 and at least one peripheral device. The processor 1001, the memory 1002, and the peripheral device interface 1003 may be connected through a bus or signal lines. Each peripheral device may be connected to the peripheral device interface 1003 through a bus, signal lines, or a circuit board. Specifically, the peripheral device includes at least one of: a radio frequency circuit 1004, a display screen 1005, a camera 1006, an audio circuit 1007, a positioning component 1008, and a power supply 1009.

[0227] The peripheral device interface 1003 can be used to connect at least one peripheral device related to I / O (input / output) to the processor 1001 and the memory 1002. In some embodiments, the processor 1001, the memory 1002, and the peripheral device interface 1003 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 1001, the memory 1002, and the peripheral device interface 1003 can be implemented on a separate chip or circuit board, and this embodiment does not limit this.

[0228] The radio frequency circuit 1004 is used to receive and transmit RF (radio frequency) signals, also known as electromagnetic signals. The radio frequency circuit 1004 communicates with a communication network and other communication devices through electromagnetic signals. The radio frequency circuit 1004 converts an electrical signal into an electromagnetic signal for transmission, or converts a received electromagnetic signal into an electrical signal. Optionally, the radio frequency circuit 1004 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and so on. The radio frequency circuit 1004 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes but is not limited to: a metropolitan area network, each generation of mobile communication networks (2G, 3G, 4G, and 5G), a wireless local area network, and / or a WiFi (wireless fidelity) network. In some embodiments, the radio frequency circuit 1004 may further include a circuit related to NFC (near field communication), and the present disclosure does not limit this.

[0229] The display screen 1005 is used to display the UI (user interface). The UI may include graphics, text, icons, videos, and any combination thereof. When the display screen 1005 is a touch display screen, the display screen 1005 also has the ability to collect touch signals on or above the surface of the display screen 1005. The touch signals can be input as control signals to the processor 1001 for processing. At this time, the display screen 1005 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there may be one display screen 1005, which is provided on the front panel of the terminal 1000; in other embodiments, there may be at least two display screens 1005, which are respectively provided on different surfaces of the terminal 1000 or are in a foldable design; in still other embodiments, the display screen 1005 may be a flexible display screen, which is provided on a curved surface or a folding surface of the terminal 1000. Even further, the display screen 1005 can also be set to an irregular non-rectangular shape, that is, a special-shaped screen. The display screen 1005 can be prepared using materials such as LCD (liquid crystal display) and OLED (organic light-emitting diode).

[0230] The camera module 1006 is used to capture images or videos. Optionally, the camera module 1006 includes a front camera and a rear camera. Generally, the front camera is provided on the front panel of the terminal, and the rear camera is provided on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth-of-field camera, a wide-angle camera, and a telephoto camera, to implement functions such as background blurring by fusing the main camera and the depth-of-field camera, panoramic shooting by fusing the main camera and the wide-angle camera, and VR (virtual reality) shooting functions or other fused shooting functions. In some embodiments, the camera module 1006 may also include a flash. The flash can be a single-color-temperature flash or a two-color-temperature flash. A two-color-temperature flash refers to a combination of a warm-light flash and a cold-light flash, which can be used for light compensation under different color temperatures.

[0231] The audio circuit 1007 may include a microphone and a speaker. The microphone is used to collect sound waves of the user and the environment, and convert the sound waves into electrical signals for input to the processor 1001 for processing, or input to the radio frequency circuit 1004 to achieve voice communication. For the purpose of stereo collection or noise reduction, there may be multiple microphones, which are respectively arranged at different parts of the terminal 1000. The microphone may also be an array microphone or an omnidirectional collection microphone. The speaker is used to convert the electrical signal from the processor 1001 or the radio frequency circuit 1004 into sound waves. The speaker may be a traditional thin film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signal into sound waves audible to humans, but also convert the electrical signal into sound waves inaudible to humans for uses such as ranging. In some embodiments, the audio circuit 1007 may further include a headphone jack.

[0232] The positioning component 1008 is used to locate the current geographical location of the terminal 1000 to achieve navigation or LBS (location based service). The positioning component 1008 may be a positioning component based on GPS (global positioning system), Beidou system, GLONASS system or Galileo system.

[0233] The power supply 1009 is used to supply power to each component in the terminal 1000. The power supply 1009 may be alternating current, direct current, a disposable battery or a rechargeable battery. When the power supply 1009 includes a rechargeable battery, the rechargeable battery may support wired charging or wireless charging. The rechargeable battery may also be used to support fast charging technology.

[0234] In some embodiments, the terminal 1000 further includes one or more sensors 1010. The one or more sensors 1010 include but are not limited to: an acceleration sensor 1011, a gyroscope sensor 1012, a pressure sensor 1013, a fingerprint sensor 1014, an optical sensor 1015 and a proximity sensor 1016.

[0235] The acceleration sensor 1011 can detect the magnitude of acceleration on the three coordinate axes of the coordinate system established with the terminal 1000. For example, the acceleration sensor 1011 can be used to detect the components of the gravitational acceleration on the three coordinate axes. The processor 1001 can control the display screen 1005 to display the user interface in a landscape view or a portrait view according to the gravitational acceleration signal collected by the acceleration sensor 1011. The acceleration sensor 1011 can also be used for game or collection of the user's motion data.

[0236] The gyroscope sensor 1012 can detect the body direction and rotation angle of the terminal 1000. The gyroscope sensor 1012 can cooperate with the acceleration sensor 1011 to collect the 3D actions of the user on the terminal 1000. Based on the data collected by the gyroscope sensor 1012, the processor 1001 can implement the following functions: motion sensing (such as changing the UI according to the user's tilting operation), image stabilization during shooting, game control, and inertial navigation.

[0237] The pressure sensor 1013 can be disposed on the side frame of the terminal 1000 and / or the lower layer of the display screen 1005. When the pressure sensor 1013 is disposed on the side frame of the terminal 1000, it can detect the holding signal of the user on the terminal 1000, and the processor 1001 can perform left / right hand recognition or quick operation based on the holding signal collected by the pressure sensor 1013. When the pressure sensor 1013 is disposed on the lower layer of the display screen 1005, the processor 1001 can control the operable controls on the UI interface according to the pressure operation of the user on the display screen 1005. The operable controls include at least one of button controls, scroll bar controls, icon controls, and menu controls.

[0238] The fingerprint sensor 1014 is used to collect the fingerprint of the user. The processor 1001 can identify the user's identity based on the fingerprint collected by the fingerprint sensor 1014, or the fingerprint sensor 1014 can identify the user's identity based on the collected fingerprint. When the identified user identity is a trusted identity, the processor 1001 authorizes the user to perform relevant sensitive operations, and the sensitive operations include unlocking the screen, viewing encrypted information, downloading software, making payments, and changing settings, etc. The fingerprint sensor 1014 can be disposed on the front, back, or side of the terminal 1000. When there are physical buttons or manufacturer logos on the terminal 1000, the fingerprint sensor 1014 can be integrated with the physical buttons or manufacturer logos.

[0239] The optical sensor 1015 is used to collect the ambient light intensity. In one embodiment, the processor 1001 can control the display brightness of the display screen 1005 according to the ambient light intensity collected by the optical sensor 1015. Specifically, when the ambient light intensity is high, the display brightness of the display screen 1005 is increased; when the ambient light intensity is low, the display brightness of the display screen 1005 is decreased. In another embodiment, the processor 1001 can also dynamically adjust the shooting parameters of the camera module 1006 according to the ambient light intensity collected by the optical sensor 1015.

[0240] The proximity sensor 1016, also known as the distance sensor, is usually disposed on the front panel of the terminal 1000. The proximity sensor 1016 is used to collect the distance between the user and the front of the terminal 1000. In one embodiment, when the proximity sensor 1016 detects that the distance between the user and the front of the terminal 1000 is gradually decreasing, the processor 1001 controls the display screen 1005 to switch from the lit state to the off state; when the proximity sensor 1016 detects that the distance between the user and the front of the terminal 1000 is gradually increasing, the processor 1001 controls the display screen 1005 to switch from the off state to the lit state.

[0241] Those skilled in the art can understand that Figure 10 the structure shown in does not constitute a limitation on the terminal 1000, and may include more or fewer components than shown in the figure, or combine some components, or adopt a different component layout.

[0242] Figure 11 is a schematic structural diagram of a server provided by an embodiment of the present disclosure. The server 1100 may vary greatly due to different configurations or performances, and may include one or more processors (central processing units, CPUs) 1101 and one or more memories 1102. Among them, at least one instruction is stored in the memory 1102, and the at least one instruction is loaded and executed by the processor 1101 to implement the methods provided by the above-mentioned various method embodiments. Of course, the server may also have components such as a wired or wireless network interface, a keyboard, and an input / output interface for input and output. The server may also include other components for implementing the functions of the device, which will not be elaborated here.

[0243] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including instructions, and the above instructions can be executed by a processor in the terminal to complete the robot control method in the above embodiment. The computer-readable storage medium may be non-transitory. For example, the computer-readable storage medium may be a ROM (read-only memory), a RAM (random access memory), a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0244] Those of ordinary skill in the art can understand that all or part of the steps of implementing the above embodiments can be completed by hardware, or can be completed by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and the above-mentioned storage medium can be a read-only memory, a magnetic disk, or an optical disc, etc.

[0245] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data for analysis, stored data, displayed data, etc.), and signals (including but not limited to signals transmitted between user terminals and other devices, etc.) involved in this disclosure are all authorized by users or fully authorized by all parties, and the collection, use, and processing of relevant data need to comply with relevant laws, regulations, and standards in relevant countries and regions. For example, the "reference image data" and "target image data" involved in this disclosure are obtained under full authorization.

[0246] The foregoing are only optional embodiments of this disclosure and are not intended to limit this disclosure. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of this disclosure shall be included within the protection scope of this disclosure.

Claims

1. A robot control method, characterized in that, The method includes: Obtaining a plurality of reference image data of a target object, where each reference image data corresponds to different pose information; Obtaining target image data of the target object through a camera; Determining the reference image data that matches the target image data based on the degree of matching between the target image data and each reference image data; Determining the pose information corresponding to the target image data based on the pose information corresponding to the reference image data that matches the target image data; Controlling the robot to work based on the pose information corresponding to the target image data.

2. The method according to claim 1, characterized in that, The determining the reference image data that matches the target image data based on the degree of matching between the target image data and each reference image data includes: Performing binarization processing on the reference image data to obtain reference binarized image data; Performing binarization processing on the target image data to obtain target binarized image data; For each reference binarized image data, determining the degree of matching between the reference image data corresponding to the reference binarized image data and the target image data based on the gray values of a plurality of reference pixel points in the reference binarized image data and the gray values of a plurality of target pixel points in the target binarized image data; Determining the reference image data with the largest degree of matching with the target image data as the reference image data that matches the target image data.

3. The method according to claim 2, wherein The method further includes: Determining a target area corresponding to the target object in the target binarized image data based on the target binarized image data and a visual tracking algorithm; The determining the degree of matching between the reference image data corresponding to the reference binarized image data and the target image data based on the gray values of a plurality of reference pixel points in the reference binarized image data and the gray values of a plurality of target pixel points in the target binarized image data includes: Determining the degree of matching between the reference image data corresponding to the reference binarized image data and the target image data based on the gray values of a plurality of reference pixel points in the reference binarized image data and the gray values of a plurality of target pixel points in the target area.

4. The method according to claim 3, wherein The determining the degree of matching between the reference image data corresponding to the reference binarized image data and the target image data based on the gray values of a plurality of reference pixel points in the reference binarized image data and the gray values of a plurality of target pixel points in the target area includes: Based on a preset window, repeatedly selecting a plurality of selected areas in the reference binarized image data, where the size of the preset window is the same as the size of the target area; For each selected area, determining the area matching degree between the selected area and the target area based on the gray values of a plurality of reference pixel points in the selected area and the gray values of a plurality of target pixel points in the target area; Determining the degree of matching between the reference image data corresponding to the reference binarized image data and the target image data based on the area matching degrees between the plurality of selected areas and the target area.

5. The method according to claim 4, wherein Determining the matching degree between the reference image data corresponding to the reference binary image data and the target image data based on the region matching degree between the multiple selected regions and the target region includes: Determining the region matching degree with the largest value among the region matching degrees corresponding to the multiple selected regions as the matching degree between the reference image data corresponding to the reference binary image data and the target image data.

6. The method according to claim 4, characterized in that, Determining the matching degree between the reference image data corresponding to the reference binary image data and the target image data based on the region matching degree between the multiple selected regions and the target region includes: Determining the selected region with the largest value of the corresponding region matching degree as the selected region to be adjusted; Based on the closest point search ICP algorithm, the selected region to be adjusted, the reference binary image data, and the target region, determining the matching region in the reference binary image data that matches the target region, where the region matching degree between the matching region and the target region is greater than or equal to the region matching degree between the selected region to be adjusted and the target region; Determining the region matching degree between the matching region and the target region as the matching degree between the reference image data corresponding to the reference binary image data and the target image data.

7. The method according to claim 2, wherein The method further includes: Determining multiple feature points and multiple feature lines corresponding to the target object; For each reference image data, determining the position information corresponding to each feature point and the position information corresponding to each feature line in the reference image data; Determining the matching degree between the reference image data corresponding to the reference binary image data and the target image data based on the gray values of multiple reference pixel points in the reference binary image data and the gray values of multiple target pixel points in the target binary image data includes: For each reference binary image data, adjusting the gray values of the reference pixel points other than the feature points and the feature lines to 0 to obtain the reference adjusted image data corresponding to the reference binary image data; Performing edge detection processing on the target binary image data to obtain multiple edge positions in the target binary image data; In the target binary image data, adjusting the gray values of the target pixel points other than the edge positions to 0 to obtain the target adjusted image data corresponding to the target binary image data; Based on the gray values of multiple reference pixel points in the reference adjusted image data and the gray values of multiple target pixel points in the target adjusted image data, determining the matching degree between the reference image data corresponding to the reference adjusted image data and the target image data.

8. The method according to claim 1, wherein The method further includes: Obtaining encoder information corresponding to the conveyor belt through an encoder; Determining the pose information corresponding to the target image data based on the pose information corresponding to the reference image data that matches the target image data includes: Based on a Kalman filter, perform data fusion processing on the pose information corresponding to the reference image data that matches the target image data and the encoder information to obtain the pose information corresponding to the target image data.

9. A robot control device, characterized in that, The device includes: A first acquisition module, configured to acquire a plurality of reference image data of a target object, each reference image data corresponding to different pose information; A second acquisition module, configured to acquire the target image data of the target object through a camera; A first determination module, configured to determine the reference image data that matches the target image data based on the degree of matching between the target image data and each reference image data; A second determination module, configured to determine the pose information corresponding to the target image data based on the pose information corresponding to the reference image data that matches the target image data; A control module, configured to control the operation of the robot based on the pose information corresponding to the target image data.

10. A computer device, characterized in that, The computer device includes a processor and a memory, and at least one instruction is stored in the memory, and the at least one instruction is loaded and executed by the processor to implement the operations performed by the robot control method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, At least one instruction is stored in the storage medium, and the at least one instruction is loaded and executed by a processor to implement the operations performed by the robot control method according to any one of claims 1 to 8.

Images