Robot control method, apparatus, and electronic device
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- FIBOCOM WIRELESS
- Filing Date
- 2026-03-25
- Publication Date
- 2026-07-14
Smart Images
Figure CN122391298A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of robot control, and more particularly to a robot control method, device, and electronic device. Background Technology
[0002] In existing technologies, robot control typically involves capturing images through the robot's camera, which are then recognized by the robot. Based on the recognition results, the robot can generate control commands to execute related actions, such as sorting goods or controlling a robotic arm to pick up goods.
[0003] However, in the existing technology, RGB images captured by low-cost cameras do not include depth information, resulting in inaccurate recognition results based on these images, which further leads to inaccurate control of the robot. Summary of the Invention
[0004] This application provides a robot control method, apparatus, storage medium, and electronic device to solve the technical problem of inaccurate robot control.
[0005] In a first aspect, this application provides a robot control method, comprising: acquiring an RGB image collected by a robot; filtering the luminance channel of the RGB image to obtain a filtered image; calculating the gradient information of the filtered image; suppressing non-edge points of the filtered image according to the gradient information to obtain an edge map of the RGB image; inputting the RGB image and the edge map into a visual model, and controlling the robot according to the generated control instructions.
[0006] Secondly, this application provides a robot control device, comprising: an acquisition module for acquiring an RGB image collected by the robot; a filtering module for filtering the luminance channel of the RGB image to obtain a filtered image; a calculation module for calculating the gradient information of the filtered image; a suppression module for suppressing non-edge points of the filtered image according to the gradient information to obtain an edge map of the RGB image; and an input module for inputting the RGB image and the edge map into a visual model and controlling the robot according to the generated control instructions.
[0007] Thirdly, this application provides an electronic device, comprising: at least one communication interface; at least one bus connected to the at least one communication interface; at least one processor connected to the at least one bus; and at least one memory connected to the at least one bus, wherein the memory stores a computer program, and the processor is configured to implement the robot control method described above when executing the computer program.
[0008] Compared with the prior art, the above-mentioned technical solution provided in this application has the following advantages: The solution provided in this application acquires an RGB image collected by a robot; filters the brightness channel of the RGB image to obtain a filtered image; calculates the gradient information of the filtered image; suppresses non-edge points of the filtered image according to the gradient information to obtain an edge map of the RGB image; inputs the RGB image and the edge map into a visual model, and controls the robot according to the generated control instructions, thereby generating an edge map based on the RGB image, providing depth information not found in the RGB image, and inputting the RGB image and the edge map together into the visual model for recognition, thereby obtaining accurate recognition results and further improving the control accuracy of the robot. Attached Figure Description
[0009] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.
[0010] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0011] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.
[0012] Figure 1 A flowchart illustrating a robot control method provided in this application embodiment; Figure 2 A filtering schematic diagram of a robot control method provided in an embodiment of this application; Figure 3 This is a schematic diagram of a converted grayscale image provided in an embodiment of the present application for a robot control method; Figure 4 This is a schematic diagram of the structure of a robot control device provided in an embodiment of this application; Figure 5 This is a schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation
[0013] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0014] The following disclosure provides numerous different embodiments or examples for implementing various structures of the invention. To simplify the disclosure, specific examples of components and arrangements are described below. These are merely examples and are not intended to limit the scope of the invention. Furthermore, reference numerals and / or letters may be repeated in different examples. Such repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed.
[0015] To address the technical problem of inaccurate robot control in existing technologies, this application provides a robot control method that can improve the accuracy of robot control.
[0016] Figure 1 This is a flowchart illustrating a robot control method provided in an embodiment of this application. Figure 1 As shown, the above robot control method includes: S102, acquire the RGB image collected by the robot; S104, filter the luminance channel of the RGB image to obtain the filtered image; S106, Calculate the gradient information of the filtered image; S108: Based on the gradient information, suppress non-edge points in the filtered image to obtain the edge map of the RGB image; S110 inputs RGB images and edge maps into the visual model and controls the robot according to the generated control instructions.
[0017] Optionally, this application can be applied to scenarios involving robot control. The robot can be used for carrying people, carrying goods, transporting, manipulating cargo, interacting, learning, educating, and recognizing, but is not limited to these functions. The robot can be equipped with a camera to capture RGB images. The camera used in this application is a low-cost camera; low-cost cameras capture RGB images without depth information. While equipping the robot with a high-cost camera could capture images with depth information, the cost would be too high, and this application is not applicable in this case.
[0018] Robots can be moved or fixed in one place, and their cameras can be fixed in one direction or rotate. For example, a robot can be fixed near a production line, with a fixed camera identifying the situation on the production line. Based on the identification results, the robot can be controlled to process items on the production line, such as packaging or moving them.
[0019] In this application, the robot can acquire RGB images at a preset frequency, such as acquiring one frame every preset time interval, or acquiring 24 frames per second. For the RGB images acquired by the robot, each image can be processed using the methods described in this application. Alternatively, the acquired RGB images can be filtered according to time period requirements, selecting RGB images within a preset time period, and then processing the selected images. Alternatively, for the acquired RGB images, the image clarity can be identified first, blurry images can be removed, and the remaining images can be processed. If blurry images are included, the position and angle of the robot's camera can be adjusted to reduce or avoid blurry images.
[0020] For RGB images acquired by the robot, this application first obtains the edge map of the RGB image. The edge map in this application is used to record the contours and boundary information of objects in the RGB image. By highlighting the contours and boundary information of objects in the RGB image through the edge map, the robot can recognize the RGB image and the edge map to generate control commands and control its own operation.
[0021] In this application, the process of generating an edge map from an RGB image involves multiple steps. The main steps include: Step 1: Filter the luminance channel of the RGB image to obtain the filtered image; In this step, the luminance channel of the RGB image is filtered, allowing for individual adjustment of the RGB image's brightness without needing to adjust the color information. Filtering the luminance channel alters the RGB image's brightness, smoothing the luminance information and removing noise and uneven lighting. Alternatively, this application can convert the RGB image to grayscale to remove color interference before filtering.
[0022] Step 2: Calculate the gradient information of the filtered image; After filtering an RGB image, the gradient information of the filtered image can be calculated. In this application, the gradient information may include the gradient direction and the gradient magnitude. The gradient direction indicates the direction of large gradient change, which is generally perpendicular to the edge. The gradient magnitude represents the degree of gradient change. Generally, the areas with large gradient magnitude changes are the edge regions of the image (the gradient magnitude changes greatly between edge regions and non-edge regions).
[0023] By calculating gradient information, the edges of different objects and regions in the filtered RGB image can be roughly located.
[0024] Step 3: Based on the gradient information, suppress the non-edge points of the filtered image to obtain the edge map of the RGB image.
[0025] After calculating the gradient information, this step uses the gradient direction and magnitude to adjust the originally wide edges to single-pixel widths, resulting in precisely narrower edges compared to the original wide edges. The image obtained after processing the edge width is then used as the edge map. The edge map preserves the edges of different objects and regions in the RGB image, and these edges are single-pixel wide. It is used along with the RGB image to input into the visual model, allowing the model to recognize the image and obtain control commands.
[0026] The visual model in this application can be located in the robot, which uses the visual model to recognize images and generate control commands to control its own operation. Alternatively, the visual model can be deployed on a server, where the robot uploads images to the server, which recognizes them and generates control commands. The robot then receives these control commands to control its own operation.
[0027] The solution provided in this application involves acquiring an RGB image collected by a robot; filtering the luminance channel of the RGB image to obtain a filtered image; calculating the gradient information of the filtered image; suppressing non-edge points of the filtered image according to the gradient information to obtain an edge map of the RGB image; inputting the RGB image and the edge map into a visual model; and controlling the robot according to the generated control commands. This allows the generation of an edge map based on the RGB image, providing depth information not found in the RGB image. By inputting both the RGB image and the edge map into the visual model for recognition, accurate recognition results can be obtained, further improving the control accuracy of the robot.
[0028] As an optional example, filtering the luminance channel of an RGB image to obtain a filtered image includes: converting the RGB image to a grayscale image; and filtering the grayscale image using Equations 1 and 2 to obtain the filtered image. (1) (2) in, This is the filtered image. It is a grayscale image. It is a two-dimensional Gaussian kernel. The standard deviation of the two-dimensional Gaussian kernel. This represents a convolution operation, where x and y are the coordinates of pixels in the image. It is a natural constant.
[0029] This embodiment introduces a scheme for filtering RGB images. The filtering method can be Gaussian filtering (but not limited to this; mean filtering, median filtering, bilateral filtering, etc., can also be used). Before filtering the RGB image, it is first converted to a grayscale image. Converting the RGB image to grayscale removes color interference. Then, the grayscale image can be filtered using the above formulas 1 and 2. Formula 1 uses a two-dimensional Gaussian kernel. This represents the standard deviation of the two-dimensional Gaussian kernel, commonly set to 1.4, but other values can also be used. Gaussian filtering filters each pixel in the grayscale image. Taking each pixel in the grayscale image as the current pixel, the two-dimensional Gaussian kernel for the current pixel is first calculated. Let x and y be the horizontal and vertical coordinates of the current pixel in the grayscale image, respectively. A two-dimensional Gaussian kernel is calculated and convolved with the grayscale value of the current pixel in the grayscale image. The result is used as the grayscale value of the corresponding pixel in the filtered image. After processing each pixel, the filtered image is obtained.
[0030] In this application, the resolution of the captured image can be 640x480 or higher. Therefore, for a size of 640x480, the RGB image includes 640x480 pixels. In this application, 3... Taking 3 pixels as an example, Figure 2 As shown, use formulas 1 and 2 to calculate 3. Each of the three pixels is filtered to obtain the filtered pixel.
[0031] In this application, the RGB image is filtered using the above formula, thereby reducing or removing interference in the RGB image, providing an accurate data basis for calculating the edge map in the RGB image, and improving the accuracy of the subsequently calculated edge map.
[0032] As an optional example, converting an RGB image to a grayscale image includes: converting an RGB image to a grayscale image using Formula 3: (3) in, It is a grayscale image. This represents the red value of the corresponding pixel in the RGB image. This represents the green value of the corresponding pixel in the RGB image. This is the blue value of the corresponding pixel in the RGB image.
[0033] This embodiment introduces a scheme for converting an RGB image to a grayscale image. Converting an RGB image to grayscale essentially involves fusing the color information from the three channels of the RGB image into the brightness information of a single channel. An RGB image includes three channels: red, green, and blue, each corresponding to a numerical value ranging from 0 to 255. In this embodiment, the RGB image can be converted to a grayscale image using Formula 3. In the formula, R, G, and B represent the color values in their respective channels; substituting these values into Formula 3 yields the grayscale values in the grayscale image.
[0034] Because the data in Formula 3 above contains decimals, if the calculated grayscale value is not an integer, it can normally be rounded to an integer. However, in this application, to make the grayscale values more distinct, grayscale values below 128 that contain decimals are rounded up. For example, if the grayscale value is 52.3, then the grayscale value is taken as 53. For grayscale values above 128 that contain decimals, the integer part is taken as the grayscale value. The purpose of this is to concentrate the grayscale values closer to 128, avoiding excessively dark or light edges and improving the accuracy of edge recognition.
[0035] In this embodiment, when using Formula 3, the above formula 3 is actually used to calculate the gray value of the corresponding pixel in the grayscale image for each pixel in the RGB image.
[0036] Continuing with the above RGB image, which includes 640x480 pixels, using 3 This example illustrates the case with 3 pixels. Figure 3 As shown, grayscale values are obtained by calculating the values of R, G, and B, thereby converting the RGB image into a grayscale image.
[0037] This application removes color interference from RGB images by first converting them to grayscale during the filtering process, thereby reducing the computational load of the filtering process and improving filtering efficiency.
[0038] As an optional example, calculating the gradient information of the filtered image includes: calculating the gradient information using Equations 4 and 5: (4) (5) in, For horizontal direction operators, For vertical direction operators, The gradient magnitude is the value in the gradient information. This refers to the gradient direction in the gradient information.
[0039] This embodiment introduces a scheme for calculating gradient information. The gradient information in this application includes gradient magnitude and gradient direction. The gradient direction represents the direction of large gradient changes, generally perpendicular to edges, while the gradient magnitude represents the degree of gradient change. Areas with large gradient magnitude changes are generally edge regions in the image. In this application, when calculating the gradient direction and gradient magnitude, [the following method is used]. and , and These are the horizontal and vertical operators, which are two matrices. They are convolved with the filtered image to detect horizontal and vertical edges, respectively. An intermediate value is calculated using Equation 4, and then substituted into Equation 5 to calculate the gradient direction and magnitude.
[0040] In this application, the gradient direction and gradient magnitude of the filtered image are calculated using the above formula, thereby determining the approximate edge position in the RGB image and providing a data basis for further accurate edge map calculation.
[0041] As an optional example, according to the gradient information, suppressing non-edge points of the filtered image to obtain the edge map of the RGB image includes: along the gradient direction, comparing the gradient value of each pixel with the gradient values of two adjacent pixels in the neighborhood and retaining the maximum value; for the retained pixels, dividing them into strong edge pixels and weak edge pixels by comparing the gradient value with a high threshold and a low threshold; and connecting the strong edge pixels and the weak edge pixels adjacent to the strong edge pixels to form a continuous image to obtain the edge map.
[0042] In this example, after converting the RGB image to grayscale, performing filtering, and calculating the gradient magnitude and direction, coarse edge information is obtained. This edge information is generally wide, which can be understood as a line segment formed by multiple rows or columns of pixels. Therefore, in this application, this edge information needs to be "slimmed down" to obtain a narrow edge. Since the gradient direction is perpendicular to the edge, this application compares the gradient value of each pixel in the edge information with the gradient values of two adjacent pixels along the gradient direction and retains the maximum value. This processing is performed for each pixel, thereby retaining a narrower edge line from the originally wide edge information.
[0043] In this application, the retained pixels are compared with high and low thresholds to distinguish strong edge pixels, weak edge pixels, and intermediate pixels. Strong edge pixels are those with pixel values greater than the high threshold, weak edge pixels are those with pixel values less than the low threshold, and intermediate pixels are those with pixel values between the two thresholds. In the comparison results, strong edge pixels can be considered as points on the edge lines, while intermediate and weak edge pixels are not considered as edge lines because their pixel values do not meet the thresholds.
[0044] The retained, narrower edge lines are typically one pixel wide. Therefore, compared to the original edge lines, a large number of pixels have been removed. Consequently, the edge lines formed by the remaining pixels may have breaks. To address this breakage issue, this application considers weak edge pixels adjacent to strong edge pixels as also pixels on the edge line. Although the weak edge pixels adjacent to strong edge pixels may not meet the pixel value requirement, they are connected to the strong edge pixels to form a joint edge map, recording the boundaries of objects and regions, thus resolving the issue of edge line breaks.
[0045] The method described above in this application enables the extraction of complete and accurate edge lines from rough edge lines, resulting in an edge map and improving the accuracy of the edge map.
[0046] As an optional example, connecting strong edge pixels and adjacent weak edge pixels to form a continuous image to obtain an edge map includes: determining each weak edge pixel as the current pixel; determining whether there are strong edge pixels within a preset neighborhood of the current pixel; if the determination result is yes, connecting the current pixel with the strong edge pixels; if the determination result is no, deleting the current pixel; and determining the remaining strong edge pixels and weak edge pixels to form the edge map.
[0047] In the example above, weak edge pixels adjacent to strong edge pixels are also considered pixels on the edge line. This example provides a scheme to determine whether strong and weak edge pixels are adjacent. Specifically, each weak edge pixel is designated as the current pixel, and it is determined whether a strong edge pixel exists within a preset neighborhood of that pixel. This neighborhood can be within N pixels directly at the weak edge pixel, where N is a positive integer. For example, traversing N pixels horizontally and vertically from the weak edge pixel as the center, the N pixels form a cross. The smallest polygon enclosing the cross is defined as the preset neighborhood of the weak edge pixel. If the neighborhood includes a strong edge pixel, the current weak edge pixel is also retained and treated as a pixel on the edge line, just like the strong edge pixel. Alternatively, if the neighborhood includes more than a preset number of strong edge pixels, the current weak edge pixel is also retained and treated as a pixel on the edge line, just like the strong edge pixel.
[0048] This application determines whether to delete a weak edge pixel by judging whether a strong edge pixel exists within a neighborhood of a preset size of the weak edge pixel. This avoids the existence of broken edges in the edge map and improves the accuracy of the edge map.
[0049] As an optional example, inputting the RGB image and edge map into the visual model and controlling the robot according to the generated control instructions includes: inputting the RGB image and edge map into the visual model, and having the visual model fuse them using Equation 6: (6) (7) in, To merge images, It is an RGB image. For edge maps; where, , and For learnable query matrices and learnable key matrices, The number of channels in an RGB image. The number of channels in the edge map, This represents the stitching features of the RGB image and the edge map, where D is the feature dimension. It is dynamic; the fused image is input into the visual model, which generates control commands to control the robot.
[0050] In this application, after obtaining the edge map of the RGB image, the edge map and the RGB image can be input into a visual model for recognition. Since the edge map records the boundary lines of objects and ranges in the RGB image, it can provide depth information for the RGB image, enabling the visual model to perform accurate image recognition operations.
[0051] Specifically, the RGB image and the edge map can be combined into a fused image using formula 6 above. These are dynamically variable parameters that are dynamically allocated by analyzing the complexity of the image and the salience of the edges using an attention model.
[0052] For fused images, a visual model performs recognition to obtain recognition results. These results can include, but are not limited to, data such as objects, regions, etc., in the RGB image. Furthermore, the robot or server can generate control commands based on the visual model's recognition results to control the robot. For example, controlling the robot to sort, filter, and move.
[0053] This application uses Formula 6 above to fuse RGB images and edge maps, thereby enabling the visual model to obtain color and depth information from the RGB images, further identify the results, and improve the accuracy of the recognition results.
[0054] As an optional example, the fused image is input into a visual model, and the visual model generates control instructions, including: the visual model extracts fused features from the fused image through a multi-layer encoder, identifies the scene and objects in the RGB image based on the fused features, and generates control instructions based on the scene and objects.
[0055] In this application, for the visual model, multiple encoders can be deployed. The output of each encoder layer serves as the input to the next encoder layer, and the first encoder layer receives the fused features. Thus, the multiple encoder layers can learn different aspects of the fused features. After recognition, the visual model can identify scenes and objects in RGB images and then generate corresponding control commands to control the robot.
[0056] Specifically, in this application, the robot's control commands can be determined based on the type of the identified scene and the numerical values of the objects within it. For example, the type of control command is determined based on the scene type, which may include, but is not limited to, movement commands, sorting commands, etc. The numerical values in the control commands, such as the magnitude of the force applied and the distance traveled, are determined based on the numerical values of the identified objects, such as size, shape, and material. For instance, if the current scene is determined to be sorting based on the identification result, the control command is determined to be a sorting command, and the force used during sorting is determined based on the material and size of the identified objects, thereby controlling the robot to sort the objects.
[0057] This application uses multi-layer encoders to learn and fuse features, thereby obtaining accurate recognition results and further improving the accuracy of the generated control commands.
[0058] The following example illustrates the process of controlling a robot by generating control commands. Taking an intelligent robot with a robotic arm as an example, control commands can be generated to control the robotic arm to perform actions. The robot is equipped with a low-cost camera that captures RGB images with a resolution typically 640x480 or higher to ensure sufficient visual detail. The acquired RGB image is represented as a three-channel matrix (...). ), where (H) and (W) are the height and width of the image, respectively, and 3 represents the RGB three channels (red, green, and blue).
[0059] An edge detection algorithm is applied to the acquired RGB image to generate an edge map, highlighting the geometric contour information of objects. Edge detection is a multi-stage image processing algorithm, and the specific steps are as follows: Noise Removal: Gaussian filtering is applied to the luminance channel of the RGB image (i.e., converting the RGB image to grayscale, which can be done using Formula 3 above) to reduce noise interference. The two-dimensional Gaussian kernel used is defined in Formula 1 above, where... The standard deviation of the Gaussian kernel is given. The filtered image is obtained using Formula 2 above.
[0060] Gradient calculation: The Sobel operator in edge detection algorithms is used to calculate the gradient of the filtered image to determine the edge strength and gradient direction. The Sobel operator includes a horizontal kernel. and vertical core
[0061] The horizontal and vertical gradients are calculated using Formula 4 above. The gradient magnitude and direction are calculated using Formula 5 above.
[0062] Non-maximum suppression: Along the gradient direction, local maximum gradient values are preserved while non-edge points are suppressed, resulting in a refined edge map. Interpolation is performed along the gradient direction, comparing the gradient value of each pixel with the gradient values of its two neighboring pixels, retaining only the maximum value, and setting the difference between non-maximum values to zero.
[0063] Dual threshold processing: Set high and low thresholds (e.g., high threshold...). ), low threshold ( (Based on an 8-bit grayscale image) to distinguish between strong and weak edges. Pixels with pixel values equal to a high threshold are considered strong edge pixels, while pixels with pixel values lower than a low threshold are considered weak edge pixels.
[0064] Edge tracking: Connects weak edge pixels located near strong edge pixels to the strong edge pixels to form a continuous edge map. This ultimately generates a binary edge map. The edge map is defined as follows: a value of 1 indicates an edge point, and a value of 0 indicates a non-edge point. The edge map highlights the shape features of the object and approximates the function of the depth map, but does not require 3D data.
[0065] The generated edge map As auxiliary input, along with the original RGB image They are input together into the visual model.
[0066] Furthermore, to enhance the visual model's ability to perceive shape features, this application introduces an adaptive weighted fusion mechanism that dynamically adjusts the feature contributions of the RGB image and the edge map to optimize the performance of the visual model in shape recognition and manipulation tasks. The visual model is based on the Transformer architecture and learns to extract the shape feature space from the edge map through pre-training and fine-tuning on large-scale visual-language datasets (such as ImageNet). The edge map is input as a single-channel image.
[0067] The input to the visual model is represented in a weighted fusion form, as shown in Formula 6 above. and For adaptive weights, dynamic calculation is performed through a lightweight attention module, see Equation 7. The edge map guides the model to focus on the object's geometry through a weighted fusion mechanism, which significantly enhances the model's robustness and specificity in shape recognition, object edge recognition (indirect depth), and manipulation tasks compared to directly using RGB images or depth maps.
[0068] The visual model outputs recognition results to obtain the scene and objects in the RGB image, generates control commands, and controls the robot's robotic arm to grasp, place, and sort.
[0069] Figure 4 This is a schematic diagram of a robot control device provided in an embodiment of this application. Figure 4 As shown, the robot control device includes: The acquisition module 402 is used to acquire RGB images collected by the robot; The filtering module 404 is used to filter the luminance channel of the RGB image to obtain the filtered image. The calculation module 406 is used to calculate the gradient information of the filtered image; The suppression module 408 is used to suppress non-edge points of the filtered image according to gradient information to obtain the edge map of the RGB image. The input module 410 is used to input RGB images and edge maps into the visual model and control the robot according to the generated control instructions.
[0070] Optionally, this application can be applied to scenarios involving robot control. The robot can be used for carrying people, carrying goods, transporting, manipulating cargo, interacting, learning, educating, and recognizing, but is not limited to these functions. The robot can be equipped with a camera to capture RGB images. The camera used in this application is a low-cost camera; low-cost cameras capture RGB images without depth information. While equipping the robot with a high-cost camera could capture images with depth information, the cost would be too high, and this application is not applicable in this case.
[0071] Robots can be moved or fixed in one place, and their cameras can be fixed in one direction or rotate. For example, a robot can be fixed near a production line, with a fixed camera identifying the situation on the production line. Based on the identification results, the robot can be controlled to process items on the production line, such as packaging or moving them.
[0072] In this application, the robot can acquire RGB images at a preset frequency, such as acquiring one frame every preset time interval, or acquiring 24 frames per second. For the RGB images acquired by the robot, each image can be processed using the methods described in this application. Alternatively, the acquired RGB images can be filtered according to time period requirements, selecting RGB images within a preset time period, and then processing the selected images. Alternatively, for the acquired RGB images, the image clarity can be identified first, blurry images can be removed, and the remaining images can be processed. If blurry images are included, the position and angle of the robot's camera can be adjusted to reduce or avoid blurry images.
[0073] For RGB images acquired by the robot, this application first obtains the edge map of the RGB image. The edge map in this application is used to record the contours and boundary information of objects in the RGB image. By highlighting the contours and boundary information of objects in the RGB image through the edge map, the robot can recognize the RGB image and the edge map to generate control commands and control its own operation.
[0074] In this application, the process of generating an edge map from an RGB image involves multiple steps. The main steps include: Step 1: Filter the luminance channel of the RGB image to obtain the filtered image; In this step, the luminance channel of the RGB image is filtered, allowing for individual adjustment of the RGB image's brightness without needing to adjust the color information. Filtering the luminance channel alters the RGB image's brightness, smoothing the luminance information and removing noise and uneven lighting. Alternatively, this application can convert the RGB image to grayscale to remove color interference before filtering.
[0075] Step 2: Calculate the gradient information of the filtered image; After filtering an RGB image, the gradient information of the filtered image can be calculated. In this application, the gradient information may include the gradient direction and the gradient magnitude. The gradient direction indicates the direction of large gradient change, which is generally perpendicular to the edge. The gradient magnitude represents the degree of gradient change. Generally, the areas with large gradient magnitude changes are the edge regions of the image (the gradient magnitude changes greatly between edge regions and non-edge regions).
[0076] By calculating gradient information, the edges of different objects and regions in the filtered RGB image can be roughly located.
[0077] Step 3: Based on the gradient information, suppress the non-edge points of the filtered image to obtain the edge map of the RGB image.
[0078] After calculating the gradient information, this step uses the gradient direction and magnitude to adjust the originally wide edges to single-pixel widths, resulting in precisely narrower edges compared to the original wide edges. The image obtained after processing the edge width is then used as the edge map. The edge map preserves the edges of different objects and regions in the RGB image, and these edges are single-pixel wide. It is used along with the RGB image to input into the visual model, allowing the model to recognize the image and obtain control commands.
[0079] The visual model in this application can be located in the robot, which uses the visual model to recognize images and generate control commands to control its own operation. Alternatively, the visual model can be deployed on a server, where the robot uploads images to the server, which recognizes them and generates control commands. The robot then receives these control commands to control its own operation.
[0080] The solution provided in this application involves acquiring an RGB image collected by a robot; filtering the luminance channel of the RGB image to obtain a filtered image; calculating the gradient information of the filtered image; suppressing non-edge points of the filtered image according to the gradient information to obtain an edge map of the RGB image; inputting the RGB image and the edge map into a visual model; and controlling the robot according to the generated control commands. This allows the generation of an edge map based on the RGB image, providing depth information not found in the RGB image. By inputting both the RGB image and the edge map into the visual model for recognition, accurate recognition results can be obtained, further improving the control accuracy of the robot.
[0081] For other examples of this embodiment, please refer to the examples above, which will not be repeated here.
[0082] like Figure 5 As shown in the figure, this application provides an electronic device, including a processor 111, a communication interface 112, a memory 113, and a communication bus 114, wherein the processor 111, the communication interface 112, and the memory 113 communicate with each other through the communication bus 114. Memory 113 is used to store computer programs; In one embodiment of this application, the processor 111, when executing the program stored in the memory 113, implements the robot control method provided in any of the foregoing method embodiments.
[0083] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the robot control method provided in any of the foregoing method embodiments.
[0084] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.
[0085] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software plus a general-purpose hardware platform, or of course, using hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.
[0086] It should be understood that the terminology used herein is for the purpose of describing particular exemplary embodiments only and is not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “described” as used herein may also include the plural forms. The terms “comprising,” “including,” “containing,” and “having” are inclusive and therefore indicate the presence of the stated features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or combinations thereof. The method steps, processes, and operations described herein are not construed as requiring them to be performed in a particular order described or illustrated unless the order of performance is explicitly indicated. It should also be understood that additional or alternative steps may be used.
[0087] The above description is merely a specific embodiment of the present invention, enabling those skilled in the art to understand or implement the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.
Claims
1. A robot control method, characterized in that, include: Acquire RGB images collected by the robot; The luminance channel of the RGB image is filtered to obtain the filtered image; Calculate the gradient information of the filtered image; Based on the gradient information, non-edge points in the filtered image are suppressed to obtain the edge map of the RGB image; The RGB image and the edge map are input into the visual model, and the robot is controlled according to the generated control instructions.
2. The method according to claim 1, characterized in that, The luminance channel of the RGB image is filtered to obtain a filtered image, including: Convert the RGB image to a grayscale image; The grayscale image is filtered using Formula 1 and Formula 2 to obtain the filtered image: （1） （2） in, The filtered image, For the grayscale image, It is a two-dimensional Gaussian kernel. The standard deviation of the two-dimensional Gaussian kernel is... This represents a convolution operation, where x and y are the coordinates of pixels in the image. It is a natural constant.
3. The method according to claim 2, characterized in that, Converting the RGB image to a grayscale image includes: The RGB image is converted to a grayscale image using formula 3: （3） in, For the grayscale image, The red value of the corresponding pixel in the RGB image. The green value is the corresponding pixel value in the RGB image. The value is the blue value of the corresponding pixel in the RGB image.
4. The method according to claim 3, characterized in that, Calculating the gradient information of the filtered image includes: The gradient information is calculated using formulas 4 and 5: （4） （5） in, For horizontal direction operators, For vertical direction operators, The gradient magnitude is the gradient information. The gradient direction is defined in the gradient information.
5. The method according to claim 4, characterized in that, Based on the gradient information, suppressing non-edge points in the filtered image to obtain the edge map of the RGB image includes: Along the gradient direction, the gradient value of each pixel is compared with the gradient values of two adjacent pixels in the neighborhood, and the maximum value is retained; For the retained pixels, they are divided into strong edge pixels and weak edge pixels by comparing the gradient value with the high threshold and the low threshold. The edge map is obtained by connecting strong edge pixels and adjacent weak edge pixels to form a continuous image.
6. The method according to claim 5, characterized in that, The edge map is obtained by connecting strong edge pixels and adjacent weak edge pixels to form a continuous image, including: Each weak edge pixel is identified as the current pixel, and it is determined whether there are strong edge pixels within a neighborhood of a preset size for the current pixel. If the determination result is yes, connect the current pixel to the strong edge pixel; If the determination result is negative, delete the current pixel. The remaining strong edge pixels and weak edge pixels are used to define the edge map.
7. The method according to claim 1, characterized in that, Inputting the RGB image and the edge map into a visual model, and controlling the robot according to the generated control instructions includes: The RGB image and the edge map are input into the visual model, and the visual model fuses them using Equation 6: （6） in, To merge images, For the RGB image, The edge map; in, , , and For learnable query matrices and learnable key matrices, The number of channels in an RGB image. The number of channels in the edge map, This represents the stitching features of the RGB image and the edge map, where D is the feature dimension. It is dynamic, where x and y are the coordinates of pixels in the image; The fused image is input into the visual model, which then generates the control commands to control the robot.
8. The method according to claim 7, characterized in that, The fused image is input into the visual model, and the visual model generates the control commands, including: The visual model extracts fusion features from the fused image through a multi-layer encoder, and identifies scenes and objects in the RGB image based on the fusion features. The control commands are generated based on the scene and objects.
9. A robot control device, characterized in that, include: The acquisition module is used to acquire RGB images collected by the robot; The filtering module is used to filter the luminance channel of the RGB image to obtain a filtered image. A calculation module is used to calculate the gradient information of the filtered image; The suppression module is used to suppress non-edge points of the filtered image according to the gradient information to obtain the edge map of the RGB image; The input module is used to input the RGB image and the edge map into the visual model, and control the robot according to the generated control instructions.
10. An electronic device, characterized in that, include: At least one communication interface; At least one bus connected to the at least one communication interface; At least one processor connected to the at least one bus; At least one memory connected to the at least one bus, wherein the memory stores a computer program, and the processor executes the computer program to implement the robot control method according to any one of claims 1 to 8.