Image processing method and apparatus, and demolition robot and computer-readable storage medium

Abstract

The present disclosure relates to the technical field of computers, and relates to an image processing method and apparatus, and a demolition robot and a computer-readable storage medium. The image processing method comprises: performing image segmentation processing on an image to be processed of a current operation scene, so as to acquire a plurality of image regions in said image, wherein different image regions correspond to different material categories; on the basis of a conversion relationship between an image coordinate system where said image is located and a world coordinate system, projecting point cloud data of the current operation scene into image regions, which correspond to the point cloud data, among the plurality of image regions; and on the basis of a plurality of pieces of point cloud data corresponding to each image region, estimating pose information of each image region.

Description

Image processing method, device, demolition robot and computer-readable storage medium

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is based on the application with CN application number 202410364379.8 and application date March 27, 2024, and claims its priority. The disclosed content of the CN application is hereby introduced as a whole into this application. Technical Field

[0003] The present disclosure relates to the field of computer technology, and in particular to an image processing method, an image processing device, a demolition robot, a computer-readable storage medium, and a computer program product. Background Art

[0004] Robots, with their compact size, powerful capabilities, flexibility, safety, and reliability, are widely used in mining, tunneling, building demolition, the cement industry, and emergency rescue operations. For example, robots can ensure the safety of construction workers in harsh working environments and effectively improve work efficiency. With the continued development of cutting-edge technologies such as artificial intelligence, computer vision, and deep learning algorithms, intelligent and automated robotic operations are an inevitable trend.

[0005] In related technologies, robots are mostly operated by manual remote control, or image processing is performed based on two-dimensional images of the operating scene to control the robot to perform the operation.

[0006] Summary of the Invention

[0007] According to some embodiments of the present disclosure, an image processing method is provided, including: performing image segmentation processing on an image to be processed of a current work scene to obtain multiple image areas in the image to be processed, where different image areas correspond to different material categories; projecting point cloud data of the current work scene into image areas corresponding to the point cloud data in multiple image areas based on a conversion relationship between an image coordinate system where the image to be processed is located and a world coordinate system; and estimating the posture information of each image area based on multiple point cloud data corresponding to each image area.

[0008] In some embodiments, estimating the pose information of each image region includes: determining at least one target plane of each image region based on multiple point cloud data corresponding to each image region; and determining the pose information of each image region based on the at least one target plane.

[0009] In some embodiments, at least one target plane includes multiple target planes, the posture information includes a normal vector, and estimating the posture information of each image area includes: selecting multiple candidate point cloud data from the corresponding multiple point cloud data; determining a target plane based on the multiple candidate point cloud data; repeating the above steps until multiple target planes are determined; and determining the normal vector of each image area based on the mean of the normal vectors of the multiple target planes.

[0010] In some embodiments, the pose information includes a center of mass, and estimating the pose information of each image region includes: determining the center of mass of each image region based on the corresponding plurality of point cloud data.

[0011] In some embodiments, projecting the point cloud data of the current operation scene into an image area corresponding to the point cloud data in multiple image areas includes: aligning the radar coordinate system used to collect the point cloud data with the world coordinate system; and jointly calibrating the world coordinate system and the image coordinate system to determine the conversion relationship.

[0012] In some embodiments, image segmentation processing of the image to be processed in the current work scene includes: using the attention mechanism module in the machine learning model to encode the image to be processed; and using the dilated convolution module in the machine learning model to decode the encoding processing result to obtain multiple image areas.

[0013] In some embodiments, the dilated convolution module includes multiple dilated convolution layers, and different dilated convolution layers have different dilation rates.

[0014] In some embodiments, the machine learning model further includes a dropout layer connected after the dilated convolution module.

[0015] In some embodiments, the machine learning model also includes a convolution module, which includes multiple convolution layers. The machine learning model is trained in the following manner: calculating the loss function value of each convolution layer based on the output of each convolution layer in the multiple convolution layers; calculating the loss function value of the dilated convolution module based on the output of the dilated convolution module; and fusing the loss function value of each convolution layer and the loss function value of the dilated convolution module to determine the comprehensive loss function value used to train the machine learning model.

[0016] In some embodiments, the current operation scene includes a demolition operation scene, and the material category includes a building material category.

[0017] According to other embodiments of the present disclosure, a demolition robot is provided, comprising: a processor for executing the image processing method in any one of the above embodiments; and a controller for controlling the demolition equipment to perform operations based on the material category corresponding to the image area and the position information of the image area determined by the image processing device.

[0018] In some embodiments, the demolition robot further includes: an image acquisition device for acquiring an image to be processed of the current working scene; and a radar device for acquiring point cloud data of the current working scene.

[0019] In some embodiments, the processor executes the image processing method through the main thread, the controller controls the demolition equipment through the main thread, the image acquisition device obtains the image to be processed of the current working scene through the main thread, and the radar device obtains the point cloud data of the current working scene through the sub-thread.

[0020] In some embodiments, the main thread and the child threads run in parallel.

[0021] According to some further embodiments of the present disclosure, an image processing device is provided, including: a segmentation unit, used to perform image segmentation processing on an image to be processed of a current work scene, so as to obtain multiple image areas in the image to be processed, where different image areas correspond to different material categories; a projection unit, used to project point cloud data of the current work scene into an image area corresponding to the point cloud data in multiple image areas according to a conversion relationship between an image coordinate system where the image to be processed is located and a world coordinate system; and an estimation unit, used to estimate the posture information of each image area based on multiple point cloud data corresponding to each image area.

[0022] In some embodiments, the estimation unit determines at least one target plane for each image region based on a plurality of point cloud data corresponding to each image region, and determines the pose information of each image region based on the at least one target plane.

[0023] In some embodiments, at least one target plane includes multiple target planes, the posture information includes a normal vector, the estimation unit selects multiple candidate point cloud data from the corresponding multiple point cloud data, determines a target plane based on the multiple candidate point cloud data, repeats the above steps until multiple target planes are determined, and determines the normal vector of each image area based on the average of the normal vectors of the multiple target planes.

[0024] In some embodiments, the pose information includes a center of mass, and the estimation unit determines the center of mass of each image region based on the corresponding plurality of point cloud data.

[0025] In some embodiments, the projection unit overlaps the radar coordinate system used to collect point cloud data with the world coordinate system; and jointly calibrates the world coordinate system and the image coordinate system to determine a conversion relationship.

[0026] In some embodiments, the segmentation unit uses the attention mechanism module in the machine learning model to encode the image to be processed, and uses the dilated convolution module in the machine learning model to decode the encoding processing results to obtain multiple image regions.

[0027] In some embodiments, the dilated convolution module includes multiple dilated convolution layers, and different dilated convolution layers have different dilation rates.

[0028] In some embodiments, the machine learning model further includes a dropout layer connected after the dilated convolution module.

[0029] In some embodiments, the machine learning model also includes a convolution module, which includes multiple convolution layers. The machine learning model is trained in the following manner: calculating the loss function value of each convolution layer based on the output of each convolution layer in the multiple convolution layers; calculating the loss function value of the dilated convolution module based on the output of the dilated convolution module; and fusing the loss function value of each convolution layer and the loss function value of the dilated convolution module to determine the comprehensive loss function value used to train the machine learning model.

[0030] In some embodiments, the current operation scene includes a demolition operation scene, and the material category includes a building material category.

[0031] According to some further embodiments of the present disclosure, an image processing device is provided, comprising: a memory; and a processor coupled to the memory, wherein the processor is configured to execute the image processing method in any one of the above embodiments based on instructions stored in the memory device.

[0032] According to some further embodiments of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the image processing method in any of the above embodiments is implemented.

[0033] According to some further embodiments of the present disclosure, a computer program product is provided, comprising instructions, which, when executed by a processor, cause the processor to perform the image processing method according to any one of the above embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0035] The present disclosure can be more clearly understood from the following detailed description with reference to the accompanying drawings:

[0036] FIG1 shows a flowchart of some embodiments of the image processing method of the present disclosure;

[0037] FIG2 a shows a schematic diagram of some embodiments of the image segmentation process disclosed herein;

[0038] FIG2 b shows a schematic diagram of some embodiments of the machine learning model of the present disclosure;

[0039] 2c and 2d are schematic diagrams showing some embodiments of image sample annotation according to the present disclosure;

[0040] FIG2e is a schematic diagram showing some embodiments of pose estimation according to the present disclosure;

[0041] FIG3 shows a flowchart of another embodiment of the image processing method of the present disclosure;

[0042] FIG4a is a block diagram of some embodiments of the demolition robot disclosed herein;

[0043] FIG4b is a block diagram of some embodiments of the image processing apparatus disclosed herein;

[0044] FIG5 is a block diagram showing some other embodiments of the image processing apparatus disclosed herein;

[0045] FIG6 shows a block diagram of still further embodiments of the image processing apparatus disclosed herein. DETAILED DESCRIPTION

[0046] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that unless otherwise specifically stated, the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure.

[0047] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0048] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0049] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.

[0050] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0051] It should be noted that like reference numerals and letters refer to like items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0052] The inventors of the present disclosure have discovered that the above-mentioned related technologies have the following problems: it is difficult to automatically identify the posture of each area in the work scene, resulting in poor image processing effects.

[0053] In view of this, the present disclosure proposes an image processing technology solution that can improve the effect of image processing.

[0054] As mentioned above, most demolition robots rely on manual remote control, which prevents them from achieving intelligent, automated control. For example, demolition robots cannot adaptively identify the material of the object they are working on, and therefore cannot accurately estimate the object's position based on material recognition.

[0055] In order to solve the above-mentioned technical problems such as low intelligence and automation control level of robot operations and low operation efficiency, this paper proposes a posture estimation technology solution based on material recognition.

[0056] For example, the present invention designs a deep material segmentation model to realize material area segmentation of images of work scenes; the work object pose estimation method using multi-sensor information fusion can provide accurate three-dimensional coordinates and pose estimation for control systems such as robotic arms, which can effectively improve the accuracy and efficiency of demolition operations.

[0057] For example, the following embodiments may be used to implement the technical solutions of the present disclosure.

[0058] FIG1 shows a flowchart of some embodiments of the image processing method disclosed herein.

[0059] As shown in FIG1 , in step 110 , image segmentation processing is performed on the image to be processed of the current working scene to obtain multiple image regions in the image to be processed, where different image regions correspond to different material categories.

[0060] In some embodiments, the current operation scene includes a demolition operation scene, the material category includes a building material category, and the building material category may include concrete, bricks, metal, wood, plastic, etc.

[0061] For example, an image acquisition device (such as a camera, a webcam, etc.) can be used to capture a video of the demolition robot's working scene, and the image to be processed can be intercepted at a certain frame interval.

[0062] The following describes the image regions corresponding to different material categories obtained after image segmentation processing through the embodiment in FIG. 2 a .

[0063] FIG. 2 a shows a schematic diagram of some embodiments of the image segmentation process disclosed herein.

[0064] As shown in FIG2a , the image to be processed in the demolition operation scene includes objects belonging to various building material categories, such as plastic, metal, stone, wood, and other materials.

[0065] For example, after performing image segmentation on the image to be processed, image regions 21a, 22a, 23a, and 24a can be obtained. Image region 21a corresponds to the region where objects of the metal material category are located, image region 22a corresponds to the region where objects of the stone material category are located, image region 23a corresponds to the region where objects of the plastic material category are located, and image region 24a corresponds to the region where objects of the wood material category are located.

[0066] In some embodiments, machine learning models can be used to segment images. For example, in a demolition scenario, appropriate deep semantic segmentation algorithms and models can be selected based on the actual operation conditions and adapted into a machine learning model capable of material region segmentation.

[0067] In some embodiments, an attention mechanism module in a machine learning model is used to encode the image to be processed. For example, the attention mechanism module may include a self-attention mechanism module, a mutual attention mechanism module, etc.

[0068] For example, deep semantic segmentation models with Transformer encoding blocks, such as SegFormer, can be used as machine learning models. SegFormer has a six-layer Transformer encoder block and uses a simple decoder to fuse the feature maps output by the last four layers of encoder blocks using a two-layer MLP (Multilayer Perceptron).

[0069] In some embodiments, a dilated convolution module in a machine learning model is used to decode the encoding result to obtain multiple image regions. For example, the dilated convolution module includes multiple dilated convolution layers, and different dilated convolution layers have different dilation rates.

[0070] For example, the dilated convolution module includes one convolution layer, three dilated convolution layers, and one convolution layer in sequence. The dilation rates of the three dilated convolution layers can be set according to actual needs, such as 2, 4, and 6 respectively.

[0071] In this way, by using dilated convolutional layers with different dilation rates to expand the receptive field (Receptive Field), a wider range of contextual information is captured; by cascading three layers of different dilated convolutional layers, the machine learning model is helped to better understand the contextual information of the input data, further enhancing the model's generalization ability and stability.

[0072] In some embodiments, the machine learning model further includes a dropout layer connected after the dilated convolution module. For example, the dropout layer can be connected at the end of the dilated convolution module to temporarily "turn off" some neurons during the training process.

[0073] In this way, the neural network's dependence on specific neurons can be reduced, preventing the model from learning detailed features in the training data and causing overfitting problems, making the model more robust and generalized.

[0074] In the above embodiment, the decoder part of the machine learning model based on the attention mechanism is improved, so that the machine learning model can adapt to image segmentation in specific scenarios, such as the segmentation of material areas (concrete, wood, metal, plastic, etc.) in demolition scenes, thereby improving the material recognition performance through adaptation and parameter adjustment.

[0075] In some embodiments, the machine learning model further includes a convolution module, which includes multiple convolution layers for decoding processing.

[0076] The structure of the machine learning model is described below using the embodiment in FIG2 b .

[0077] FIG2 b shows a schematic diagram of some embodiments of the machine learning model of the present disclosure.

[0078] As shown in Figure 2b, the machine learning model consists of an encoder and a decoder. For example, the encoder includes an embedding module for overlapping patch embedding, a multi-layer transformer encoder module 21-24 based on the attention mechanism, and an MLP layer. The decoder includes a convolution module and a dilated convolution cascade module.

[0079] For example, each transformer encoder module includes an attention module, a Mix-FFN (mixed feedforward network) module, and a fusion module for overlapping patch fusion processing; the MLP layer includes an MLP layer and an upsample layer.

[0080] For example, the convolution module includes multiple convolutional layers, each of which outputs a decoding result. As shown in Figure 2b, this outputs four decoding results. The dilated convolution module includes two convolutional layers (CL) and three dilated convolutional layers (DC). The dilation rates of the three dilated convolutional layers are 2, 4, and 6, respectively. The dilated convolution module also outputs one decoding result.

[0081] In some embodiments, the machine learning model can be improved based on the SegFormer semantic segmentation model by using a dilated convolution module in the decoder instead of a separate convolution module. This not only enhances the model's generalization and stability, but also speeds up network training and prevents overfitting.

[0082] In the above embodiment, the decoder part of the machine learning model based on the attention mechanism is improved, so that the machine learning model can adapt to image segmentation in specific scenarios, such as the segmentation of material areas (concrete, wood, metal, plastic, etc.) in demolition scenes, thereby improving the material recognition performance through adaptation and parameter adjustment.

[0083] In some embodiments, a machine learning model may be trained by calculating a loss function value for each convolutional layer based on the output of each convolutional layer; calculating a loss function value for the dilated convolution module based on the output of the dilated convolution module; and fusing the loss function values ​​for each convolutional layer and the loss function value for the dilated convolution module to determine a comprehensive loss function value for training the machine learning model. For example, the loss function may be a cross-entropy loss function.

[0084] For example, the convolution module in Figure 2b outputs four decoding results, and the dilated convolution module also outputs one decoding result. Based on these five processing results, multi-level supervision loss is introduced to calculate the five losses separately. The five losses are fused and used to train the machine learning model.

[0085] In the above embodiment, a multi-layer supervised loss mechanism is used to introduce additional supervisory signals (such as the cross-entropy loss function) at different levels of the machine learning model to guide training. The losses at each level are then combined, and the final segmentation result is generated through a convolution operation. This allows the training process to better control the representation of features, more effectively capture the complex structure of the data, enhance the robustness of the model, and thus improve the model's performance in segmentation tasks.

[0086] In the above embodiment, an expanded convolution module and a dropout layer are added, and a multi-layer supervised loss is introduced, which can adapt to and improve the image segmentation performance of the machine learning model, and can be suitable for material recognition in various operation scenarios such as demolition operation scenarios to improve the operation quality.

[0087] In some embodiments, before training a machine learning model, it is necessary to construct a sample image set, i.e., to label the sample images. For example, a video of an actual work scene can be captured and then cut into images at regular frame intervals to form a sample image set.

[0088] For example, for demolition scenarios, we can construct a collection of image samples of objects at different stages based on their morphology. These objects can include fully standing buildings (such as buildings), partially collapsed buildings, and completely collapsed buildings. Sufficient image samples are collected for each type of object at each stage. These image samples are then divided into training and test sets according to a specific ratio, used for training and testing machine learning models, respectively.

[0089] For example, labeling tools can be used to label the material category of work objects in image samples. Based on the SAM deep image segmentation model, massive image segmentation processing can be performed for labeling. For example, open-source software such as Anylabeling can be used for labeling.

[0090] For example, a demolition robot can be captured in a video of its operation and captured as image samples at regular intervals. The image samples are then labeled with different image regions, such as concrete, brick, metal, wood, plastic, and background areas. Different material categories are then labeled with category names and serial numbers.

[0091] In this way, natural objects can be segmented from background areas quickly and accurately, and by receiving various manually given prompt information, the area that people want to segment can be distinguished from the surrounding area, thereby improving the accuracy and efficiency of image annotation.

[0092] The following describes the labeling process of image samples through the embodiments in Figures 2c and 2d.

[0093] 2c and 2d are schematic diagrams showing some embodiments of image sample annotation according to the present disclosure.

[0094] As shown in Figure 2c, for image regions containing materials of different shapes, different segmentation hints can be given to the machine learning model used for annotation, such as rectangular boxes, polygonal boxes, and points, lines, or area boxes within the image region to be segmented. Users can provide hints such as the hint box 21b in Figure 2c, and deep segmentation models such as SAM perform image segmentation and annotation based on the hint box.

[0095] For example, for simple scenes, you can choose to have the software automatically segment the entire image. Furthermore, you can set shortcut keys to help users quickly browse images, zoom in and out, update prompt points (such as deleting points on a polygon border), modify annotations, and automatically generate a .json file with the same name as the file after annotation. You can also use the json2png.py file format conversion function to convert image files into PNG format.

[0096] As shown in FIG2 d , a depth segmentation model such as SAM performs image segmentation based on the prompt box 21 b to obtain an image region 21 c .

[0097] In some embodiments, in addition to calling SAM to annotate material regions, various auxiliary editing functions can be added to improve annotation efficiency. For example, the annotated image sample set can be formatted to match the machine learning model used for material image segmentation, and the algorithm configuration file can be modified for model training. Ultimately, a deep semantic segmentation model is obtained, and the material segmentation model is lightweighted and integrated into the software for deployment.

[0098] In the above example, while the robot is performing tasks such as demolition, it uses a camera to capture 2D images in real time. The large model efficiently annotates these captured 2D images to obtain true values ​​for image segmentation processing, thereby constructing sufficient training samples for model training. The trained material segmentation model is then deployed in software to obtain accurate material segmentation results in real time.

[0099] In addition, through large deep image segmentation models (such as SAM, etc.), experience can be learned from massive image samples; large deep image segmentation models can accept a variety of targeted manual prompts to improve the efficiency of labeling the true value of the category of irregular material areas and reduce the burden of manual labeling.

[0100] After image segmentation is performed to obtain different image regions corresponding to different materials, image processing can be continued through steps 120 and 130 in FIG. 1 .

[0101] In step 120 , based on the conversion relationship between the image coordinate system of the image to be processed and the world coordinate system, the point cloud data of the current working scene is projected into an image area corresponding to the point cloud data among the multiple image areas.

[0102] In some embodiments, the radar coordinate system used to collect point cloud data is overlapped with the world coordinate system; the world coordinate system and the image coordinate system are jointly calibrated to determine the conversion relationship.

[0103] For example, a camera or other image acquisition device can capture a 2D image of the current work scene, while a fixed radar or other radar device can capture 3D point cloud data of the current work scene. To project the 3D point cloud data onto the 2D image, a transformation from the point cloud data to the row and column coordinates of the image is required.

[0104] For example, the conversion relationship can be determined by jointly calibrating the camera and solid-state radar. The joint calibration of the camera and solid-state radar can be understood as camera extrinsic calibration, but it requires that the world coordinate system and the radar coordinate system coincide.

[0105] In some embodiments, joint calibration may be performed using calibration methods such as a checkerboard calibration method and a Tsai calibration method.

[0106] For example, the Tsai calibration method first requires obtaining the three-dimensional coordinates of multiple point cloud data and the corresponding two-dimensional image row and column coordinates of multiple image pixels, as well as the camera's target surface size and pixel size. The image row and column coordinates of the camera's optical center in the internal parameters are u0 and v0, which are the image center. Using the triangle similarity relationship in the pinhole imaging principle, the point pair equations are substituted into solve the rotation matrix R = {r i,j} and t in the translation matrix T x and t y Component; solve the translation matrix t z Equivalent focal length f in components and internal parameters x and f y ; Realize u0, v0, t through nonlinear optimization z 、f x 、f y and the joint optimization of the distortion coefficient k value (the initial value of k can be 0).

[0107] For example, by calculating the joint calibration parameters, we can obtain the conversion relationship between the three-dimensional coordinates of the point cloud data and the row and column coordinates of the two-dimensional image. For a point cloud data with three-dimensional coordinates (X, Y, Z), the converted two-dimensional image row and column coordinates (u, v) can be calculated using the following conversion formula:

[0108] λ is the scale factor.

[0109] In step 130, the pose information of each image region is estimated based on the plurality of point cloud data corresponding to each image region. For example, the pose information may include information such as the normal vector and the center of mass of the image region.

[0110] In the above embodiment, the three-dimensional point cloud data of the current work scene is projected onto image regions corresponding to different materials in the two-dimensional image to estimate the pose information of the regions where each material resides. This enables automatic position recognition of the regions where each material resides in the work scene, thereby improving the image processing effect.

[0111] In some embodiments, each image region corresponds to a material type, and based on the calculated conversion relationship between the three-dimensional point cloud data and the two-dimensional image row and column coordinates, a corresponding point cloud cluster (including multiple point cloud data) projected onto each image region can be obtained. For example, the center of mass and normal vector of the image region can be calculated based on the point cloud cluster corresponding to the image region.

[0112] For example, the N image regions obtained by image segmentation processing based on the machine learning model are After converting the 3D point cloud data to 2D image coordinates, the point cloud cluster corresponding to each image area can be obtained. The i-th point cloud cluster P i Corresponding to image area C i , including n i The three-dimensional coordinates of point cloud data:

[0113] In some embodiments, at least one target plane of each image region is determined based on a plurality of point cloud data corresponding to each image region; and based on the at least one target plane, the position information of each image region is determined.

[0114] For example, the at least one target plane includes multiple target planes, and the pose information includes a normal vector. Multiple candidate point cloud data are selected from the corresponding multiple point cloud data; a target plane is determined based on the multiple candidate point cloud data; the above steps are repeated until multiple target planes are determined; and a normal vector for each image region is determined based on the average of the normal vectors of the multiple target planes.

[0115] For example, for a point cloud cluster corresponding to an image area, at least multiple point cloud data are selected in each round to form a point pair; the coordinate information of multiple point cloud data in the point pair is used to determine the target plane of this round, and its normal vector is calculated; the above steps are iteratively performed to obtain multiple target planes and multiple normal vectors; multiple normal vectors are weighted averaged to calculate the normal vector of the image area, which is used to guide the demolition device such as the robotic arm demolition head to operate at an angle as perpendicular to the demolition surface as possible, thereby improving the operation effect.

[0116] For example, for each point cloud cluster, two point cloud data are randomly selected from it and Form a vector

[0117] according to The normal vector to the target plane vertical The normal vector equation of the target plane can be determined:

[0118] For multiple point cloud data pairs selected from the point cloud cluster, multiple normal vector equations can be determined to form an equation group; the least squares method is used to solve the equation group to calculate the normal vector of the image area corresponding to the point cloud cluster.

[0119] In some embodiments, the pose information includes a center of mass. The center of mass of each image region is determined based on the corresponding plurality of point cloud data. For example, image region C i Corresponding point cloud cluster C i The center of mass coordinates of It can be calculated by the following formula:

[0120] In the above embodiment, the target category semantics of the image depth semantic segmentation model are transformed into material category semantics, thereby realizing the material area segmentation of the scene; by utilizing the joint calibration of the high-precision camera and solid-state radar, the radar point cloud data can be matched with the image material segmentation results; the projection matrix is ​​constructed through the calibration parameters, and the radar point cloud data items are projected into a two-dimensional image, thereby realizing the material segmentation of the point cloud data; the point cloud clusters within each segmented material area can be used for center of mass positioning and posture estimation.

[0121] The following describes the processing method of pose estimation through the embodiment in Figure 2e.

[0122] FIG2e shows a schematic diagram of some embodiments of pose estimation according to the present disclosure.

[0123] As shown in Figure 2e, based on the joint calibration results of the camera and radar, each point cloud data can be projected into a two-dimensional image to determine the image area (the white area box in Figure 2e) to which each point cloud data (the dotted figure in Figure 2e) belongs and the corresponding material category, and then calculate the pose information of each image area.

[0124] In some embodiments, the center of mass and normal vector of the image region are calculated based on the point cloud data belonging to the same image region (i.e., a point cloud cluster). For example, the center of mass of the image region is calculated using the three-dimensional coordinates of all the point cloud data in the point cloud cluster. At least three points are randomly selected from the point cloud cluster multiple times to estimate the normal vector of the target plane in the image region. Based on this, the normal vector of the target plane is determined to guide the robotic arm demolition head to operate perpendicular to the demolition surface.

[0125] For example, through the above method, it can be determined that the centroids of the two image regions in FIG. 2e are centroid 21d and centroid 23d, and the normal vectors of the two image regions are vector 22d and vector 24d respectively.

[0126] In the above examples, an image processing technology solution suitable for various application scenarios (such as demolition robot operations) was proposed, enabling material recognition and pose estimation in images. For practical application scenarios such as robot tool selection and parameter adjustment, a deep material segmentation network model was designed, and combined with dense radar point cloud data, pose estimation of material regions was achieved. This technical solution can include the design of a deep material segmentation model, the construction and annotation of image sample sets, the joint calibration of cameras and solid-state radars, and the pose estimation of material regions.

[0127] In some embodiments, the image data collected from the current working scene can be used to construct an image sample set for training a deep semantic segmentation model; the model is integrated into the software, and the software is deployed on the demolition robot.

[0128] For example, the present image processing technology solution is described below through the embodiment in FIG3 .

[0129] FIG3 shows a flowchart of other embodiments of the image processing method disclosed herein.

[0130] As shown in Figure 3, after all variables of the program integrating the technical solution of the present disclosure are initialized, the program can be divided into a main thread and a sub-thread to run simultaneously. For example, the sub-thread is responsible for collecting radar data (i.e., point cloud data), and the main thread is responsible for: collecting visible light images (i.e., image samples, images to be processed, etc.); material region segmentation based on the SegFormer model; extracting the point cloud data at the current moment and projecting it into a two-dimensional image coordinate system to obtain the image region to which the point cloud data belongs; calculating the center of mass and normal vector of each image region to guide the demolition head to operate perpendicular to the demolition surface, and visualizing the material segmentation, center of mass, and posture results.

[0131] In step 310, an image acquisition device (such as a camera, a webcam, etc.) may be used to capture a video of the demolition robot's working scene, and images to be processed or image samples may be captured at a certain frame interval.

[0132] In step 320 , three-dimensional point cloud data of the current operation scene is acquired by a radar device such as a fixed radar.

[0133] In step 330, a machine learning model such as a SegFormer model is used to perform image segmentation on the image to be processed, and different image regions correspond to different material categories (image regions can also be called material regions).

[0134] Material recognition requires not only capturing a 2D image of the scene with a camera and segmenting the material regions using a material segmentation model, but also locating the pose of each material region (e.g., its center of mass and surface normal in 3D space). Therefore, radar is also required to detect 3D points on the surface of the object corresponding to each material region to predict the center of mass and pose.

[0135] In step 340, to determine the material region to which each 3D point cloud data point is projected, the point cloud data is converted to a 2D image through a series of transformations. The radar point data projection is performed based on the material region in which the corresponding 2D image coordinates of the converted 3D point cloud data lie.

[0136] For example, the above series of transformations include three-dimensional rigid body transformation (such as rotation and translation, etc.) from the radar coordinate system to the camera coordinate system, and perspective projection transformation from the camera coordinate system to the image target coordinate system and the image row and column coordinate system.

[0137] Step 320 may be executed in parallel with any one or more of steps 310 , 330 , 340 , and 350 .

[0138] In step 350, the pose information of each image region is estimated based on the multiple point cloud data corresponding to each image region. For example, the pose information may include the normal vector and center of mass of the image region. The estimated position information can be output as a display result to guide the operation.

[0139] FIG4 a is a block diagram of some embodiments of the demolition robot disclosed herein.

[0140] As shown in Figure 4a, the demolition robot 4a includes: a processor 41a, used to execute the image processing method in any of the above embodiments; a controller 42a, used to control the demolition equipment to perform operations based on the material category corresponding to the image area and the position information of the image area determined by the image processing device.

[0141] In some embodiments, the demolition robot 4a further includes: an image acquisition device 43a for acquiring an image to be processed of the current working scene; and a radar device 44a for acquiring point cloud data of the current working scene.

[0142] In some embodiments, the processor 41a executes the image processing method through the main thread, the controller 42a controls the demolition equipment through the main thread, the image acquisition device 43a obtains the image to be processed of the current working scene through the main thread, and the radar device 44a obtains the point cloud data of the current working scene through the sub-thread.

[0143] In some embodiments, the main thread and the child threads run in parallel.

[0144] In the above embodiment, the deep semantic segmentation model method is applied to the material recognition of the work object according to the actual working conditions of the robot; the model is trained using supervised learning, and by collecting data sets of actual work scenes, it can meet the working needs of the robot in different work scenes, and has universality and robustness.

[0145] In addition, a material recognition and posture estimation method and system combining images and radars was designed using cameras and solid-state radars. This method can obtain the three-dimensional coordinates of the target point of the work object and estimate its posture. It can transform the robot's demolition operations and other work behaviors into intelligent and automated operations, thereby improving work efficiency and ensuring the safety of personnel in harsh construction environments.

[0146] FIG4 b is a block diagram of some embodiments of the image processing apparatus disclosed herein.

[0147] As shown in Figure 4b, the image processing device 4b includes: a segmentation unit 41b, which is used to perform image segmentation processing on the image to be processed of the current work scene to obtain multiple image areas in the image to be processed, and different image areas correspond to different material categories; a projection unit 42b, which is used to project the point cloud data of the current work scene into the image area corresponding to the point cloud data in the multiple image areas according to the conversion relationship between the image coordinate system of the image to be processed and the world coordinate system; an estimation unit 43b, which is used to estimate the posture information of each image area based on the multiple point cloud data corresponding to each image area.

[0148] In some embodiments, the estimation unit 43b determines at least one target plane for each image region based on a plurality of point cloud data corresponding to each image region, and determines the pose information of each image region based on the at least one target plane.

[0149] In some embodiments, at least one target plane includes multiple target planes, the posture information includes a normal vector, the estimation unit 43b selects multiple candidate point cloud data from the corresponding multiple point cloud data, determines a target plane based on the multiple candidate point cloud data, repeats the above steps until multiple target planes are determined, and determines the normal vector of each image area based on the mean of the normal vectors of the multiple target planes.

[0150] In some embodiments, the pose information includes a centroid, and the estimation unit 43b determines the centroid of each image region based on the corresponding plurality of point cloud data.

[0151] In some embodiments, the projection unit 42b overlaps the radar coordinate system used to collect point cloud data with the world coordinate system; and jointly calibrates the world coordinate system and the image coordinate system to determine the conversion relationship.

[0152] In some embodiments, the segmentation unit 41b uses the attention mechanism module in the machine learning model to encode the image to be processed, and uses the dilated convolution module in the machine learning model to decode the encoding processing result to obtain multiple image regions.

[0153] In some embodiments, the dilated convolution module includes multiple dilated convolution layers, and different dilated convolution layers have different dilation rates.

[0154] In some embodiments, the machine learning model further includes a dropout layer connected after the dilated convolution module.

[0155] In some embodiments, the machine learning model also includes a convolution module, which includes multiple convolution layers. The machine learning model is trained in the following manner: calculating the loss function value of each convolution layer based on the output of each convolution layer in the multiple convolution layers; calculating the loss function value of the dilated convolution module based on the output of the dilated convolution module; and fusing the loss function value of each convolution layer and the loss function value of the dilated convolution module to determine the comprehensive loss function value used to train the machine learning model.

[0156] In some embodiments, the current operation scene includes a demolition operation scene, and the material category includes a building material category.

[0157] FIG5 is a block diagram showing some other embodiments of the image processing apparatus disclosed herein.

[0158] As shown in FIG5 , the image processing device 5 of this embodiment includes: a memory 51 and a processor 52 coupled to the memory 51 . The processor 52 is configured to execute the image processing method in any one embodiment of the present disclosure based on instructions stored in the memory 51 .

[0159] The memory 51 may include, for example, a system memory, a fixed non-volatile storage medium, etc. The system memory may store, for example, an operating system, application programs, a boot loader, a database, and other programs.

[0160] FIG6 shows a block diagram of still further embodiments of the image processing apparatus disclosed herein.

[0161] As shown in FIG6 , the image processing device 6 of this embodiment includes: a memory 610 and a processor 620 coupled to the memory 610 . The processor 620 is configured to execute the image processing method of any of the aforementioned embodiments based on instructions stored in the memory 610 .

[0162] The memory 610 may include, for example, a system memory, a fixed non-volatile storage medium, etc. The system memory may store, for example, an operating system, application programs, a boot loader, and other programs.

[0163] The image processing device 6 may further include an input / output interface 630, a network interface 640, a storage interface 650, and the like. These interfaces 630, 640, 650, as well as the memory 610 and the processor 620, may be connected, for example, via a bus 660. The input / output interface 630 provides a connection interface for input / output devices such as a display, mouse, keyboard, touch screen, microphone, and speakers. The network interface 640 provides a connection interface for various networked devices. The storage interface 650 provides a connection interface for external storage devices such as SD cards and USB flash drives.

[0164] Those skilled in the art will appreciate that embodiments of the present disclosure may be provided as methods, systems, or computer program products. Thus, the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present disclosure may take the form of a computer program product implemented on one or more computer-usable non-transitory storage media, including but not limited to magnetic disk storage, CD-ROMs, optical storage, and the like, containing computer-usable program code.

[0165] The image processing method, image processing device, demolition robot, computer-readable storage medium, and computer program product according to the present disclosure have been described in detail. To avoid obscuring the concepts of the present disclosure, some details well known in the art have been omitted. Based on the above description, those skilled in the art will fully understand how to implement the technical solutions disclosed herein.

[0166] The methods and systems of the present disclosure may be implemented in many ways. For example, the methods and systems of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is for illustration only, and the steps of the method of the present disclosure are not limited to the order specifically described above unless otherwise specified. In addition, in some embodiments, the present disclosure may also be implemented as programs recorded in a recording medium, which include machine-readable instructions for implementing the methods according to the present disclosure. Therefore, the present disclosure also covers recording media that store programs for executing the methods according to the present disclosure.

[0167] Although some specific embodiments of the present disclosure have been described in detail by way of examples, those skilled in the art will appreciate that the above examples are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Those skilled in the art will appreciate that modifications may be made to the above embodiments without departing from the scope and spirit of the present disclosure. The scope of the present disclosure is defined by the appended claims.

Claims

1. An image processing method, comprising: Performing image segmentation processing on the image to be processed of the current working scene to obtain multiple image regions in the image to be processed, where different image regions correspond to different material categories; Projecting the point cloud data of the current operation scene into image areas corresponding to the point cloud data in the multiple image areas according to a conversion relationship between the image coordinate system of the image to be processed and the world coordinate system; Estimate the position and posture information of each image region based on the plurality of point cloud data corresponding to each image region.

2. The image processing method according to claim 1, wherein: The estimating the pose information of each image region comprises: determining at least one target plane of each image region according to the plurality of point cloud data corresponding to each image region; Determine the pose information of each image region according to the at least one target plane.

3. The image processing method according to claim 2, wherein: The at least one target plane includes a plurality of target planes, the pose information includes a normal vector, The estimating the pose information of each image region comprises: Selecting a plurality of candidate point cloud data from the corresponding plurality of point cloud data; Determining a target plane based on the plurality of candidate point cloud data; Repeat the above steps until the multiple target planes are determined; The normal vector of each image region is determined according to the average of the normal vectors of the multiple target planes.

4. The image processing method according to any one of claims 1 to 3, wherein: The posture information includes the center of mass, The estimating the pose information of each image region comprises: The centroid of each image region is determined according to the corresponding plurality of point cloud data.

5. The image processing method according to any one of claims 1 to 4, wherein: The step of projecting the point cloud data of the current operation scene into the image areas corresponding to the point cloud data in the plurality of image areas includes: aligning a radar coordinate system used to collect the point cloud data with the world coordinate system; The world coordinate system and the image coordinate system are jointly calibrated to determine the conversion relationship.

6. The image processing method according to any one of claims 1 to 5, wherein: The image segmentation processing of the image to be processed in the current operation scene includes: Encoding the image to be processed using an attention mechanism module in a machine learning model; The encoding processing result is decoded using the dilated convolution module in the machine learning model to obtain the multiple image regions.

7. The image processing method according to claim 6, wherein: The dilated convolution module includes multiple dilated convolution layers, and different dilated convolution layers have different dilation rates.

8. The image processing method according to claim 6 or 7, wherein: The machine learning model also includes a dropout layer connected after the dilated convolution module.

9. The image processing method according to any one of claims 6 to 8, wherein: The machine learning model further includes a convolution module, which includes multiple convolution layers. The machine learning model is trained in the following manner: Calculating a loss function value of each convolutional layer according to an output of each convolutional layer in the plurality of convolutional layers; Calculating a loss function value of the dilated convolution module according to an output of the dilated convolution module; The loss function value of each convolutional layer and the loss function value of the dilated convolution module are fused to determine a comprehensive loss function value for training the machine learning model.

10. The image processing method according to any one of claims 1 to 9, wherein: The current operation scene includes a demolition operation scene, and the material category includes a building material category.

11. A demolition robot comprising: A processor, configured to execute the image processing method according to any one of claims 1 to 10; The controller is used to control the demolition equipment to operate according to the material category corresponding to the image area determined by the image processing device and the position information of the image area.

12. The demolition robot according to claim 11, further comprising: An image acquisition device, used to obtain the image to be processed of the current operation scene; The radar device is used to obtain point cloud data of the current operation scene.

13. The demolition robot according to claim 12, wherein: The processor executes the image processing method through the main thread, the controller controls the demolition equipment through the main thread, the image acquisition device obtains the image to be processed of the current working scene through the main thread, and the radar device obtains the point cloud data of the current working scene through the sub-thread.

14. The demolition robot according to claim 13, wherein: The main thread and the sub-thread run in parallel.

15. An image processing apparatus, comprising: a segmentation unit, configured to perform image segmentation processing on the image to be processed of the current operation scene to obtain a plurality of image regions in the image to be processed, wherein different image regions correspond to different material categories; a projection unit, configured to project the point cloud data of the current operation scene into an image region corresponding to the point cloud data in the plurality of image regions according to a conversion relationship between the image coordinate system of the image to be processed and the world coordinate system; The estimation unit is used to estimate the pose information of each image area based on the multiple point cloud data corresponding to each image area.

16. An image processing apparatus, comprising: Memory; and a processor coupled to the memory, wherein the processor is configured to execute the image processing method according to any one of claims 1 to 10 based on instructions stored in the memory.

17. A computer-readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the image processing method according to any one of claims 1 to 10 is implemented.

18. A computer program product comprising instructions, which, when executed by a processor, cause the processor to perform the image processing method according to any one of claims 1 to 10.

Images