Paving robot, plate paving control method, device and system thereof and medium

Abstract

The invention discloses a paving robot, a plate paving control method, device and system thereof and a medium, and relates to the technical field of robots. The method comprises the following steps: acquiring an original image obtained by shooting a current plate paving position by a camera module of a paving robot, wherein the original image comprises position information of a to-be-paved plate and a reference plate; performing image preprocessing on the original image to obtain a to-be-segmented image; the to-be-segmented image is processed through a preset image instance segmentation model to obtain a corresponding instance segmentation result, and the image instance segmentation model is configured to perform local refinement processing on the to-be-segmented image in combination with plate prior information of flat and straight plate edges; performing image post-processing on the instance segmentation result to obtain a position deviation between the to-be-paved plate and the reference plate; and correcting the plate paving position according to the position deviation. The plate paving position identification method provided by the invention is low in cost and high in adaptability.

Description

Technical Field

[0001] The present application relates to the field of robotics technology, and in particular to a paving robot and a plate paving control method, device, system and medium thereof. Background Art

[0002] At present, the application of automated construction robots is becoming more and more widespread in the process of paving panels in construction. For example, paving robots, as automated construction robots used for indoor floor tile paving, will be widely used in residential areas, high-speed rail stations, airports, office buildings, schools and other scenarios. Taking paving robots as an example, this type of automated construction robot should have advantages such as high efficiency and high quality. Specific requirements include: being able to achieve long-term continuous operation; being able to achieve automatic alignment and positioning to ensure paving quality and accuracy; being able to automatically identify and avoid obstacles to avoid harm to the surrounding environment and personnel; and being able to solve problems such as unstable quality and safety hazards in traditional manual paving.

[0003] As such, very high requirements are placed on the accuracy of the plate paving position recognition of the paving robot. In the prior art, most of them are aimed at paving robots. After obtaining the tile paving position image, they usually use traditional canny operators, threshold segmentation, binarization and line detection algorithms to segment the tile edges from the background image, and then obtain multiple tile edge straight lines to finally determine the distance and angle relationship between the tile edges to guide the paving of tiles. However, this process is usually based on a fixed pixel threshold to complete the segmentation of tiles and background. If the light at the paving site changes, the segmentation accuracy will be poor, making it difficult to adapt to the actual working conditions of complex and changeable light intensity in the construction industry scene. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a paving robot and a plate paving control method, device, system and medium thereof, so as to at least partially solve the above-mentioned technical problems.

[0005] In order to achieve the above-mentioned purpose, the first aspect of the present application provides a plate paving control method, comprising: obtaining an original image obtained by photographing the current plate paving position by a camera module of a paving robot, wherein the original image includes position information of a plate to be paved and a reference plate; performing image preprocessing on the original image to obtain an image to be segmented; processing the image to be segmented by a preset image instance segmentation model to obtain a corresponding instance segmentation result, wherein the image instance segmentation model is configured to perform local refinement processing on the image to be segmented in combination with prior information of the plate that the plate edge is straight; performing image post-processing on the instance segmentation result to obtain a position deviation between the plate to be paved and the reference plate; and correcting the plate paving position according to the position deviation.

[0006] In the embodiment of the present application, the reference plate is a preset plate with straight edges or a plate that is adjacent to the paving position of the plate to be paved and has straight edges.

[0007] In an embodiment of the present application, processing the image to be segmented by the image instance segmentation model includes: extracting feature vectors of a pre-labeled plate instance segmentation image with straight plate edges to obtain a position memory vector representing the prior information of the plate; obtaining a feature vector to be queried corresponding to the image to be segmented; performing attention interaction processing on the position memory vector and the feature vector to be queried to obtain global image information; and performing local refinement processing on the global image information to obtain a final instance segmentation result.

[0008] In the embodiment of the present application, the plate instance segmentation image is annotated offline under different environmental backgrounds and different environmental lighting.

[0009] In an embodiment of the present application, the image post-processing includes: determining the corner points of the plate in the image based on the instance segmentation result; and determining the position deviation between the plate to be laid and the reference plate based on the pixel position of the determined corner points of the plate, wherein the position deviation includes the corner point distance and angle deviation between the plate to be laid and the reference plate.

[0010] In an embodiment of the present application, determining the corner points of the plate in the image includes: extracting a set of multiple straight lines in the instance segmentation result through straight line fitting; calculating the intersection points between each straight line and determining whether the corresponding intersection points are the corner points of the plate; and traversing the set to determine the corner points of the plate.

[0011] In an embodiment of the present application, the camera module includes a first camera and a second camera respectively installed on both sides of the actuator end of the robotic arm of the paving robot, and the first camera and the second camera are configured to be able to capture different plate corner points of the plate to be laid and the reference plate.

[0012] In an embodiment of the present application, the correction of the plate laying position according to the position deviation includes: obtaining a first position deviation and a second position deviation obtained by processing the original images taken by the first camera and the second camera respectively; comprehensively processing the first position deviation and the second position deviation in combination with the current laying height to determine the position correction amount and angle correction amount for plate laying; and controlling the robotic arm to correct the plate laying position according to the determined position correction amount and angle correction amount.

[0013] The second aspect of the present application provides a board paving control device, comprising: a memory configured to store instructions; and a processor configured to call the instructions from the memory and implement any of the above-mentioned board paving control methods when executing the instructions.

[0014] A third aspect of the present application provides a plate paving control system, comprising: any of the above-mentioned plate paving control devices, used to control a paving robot to perform plate paving; and a camera module, including a first camera and a second camera respectively installed on both sides of the actuator end of the mechanical arm of the paving robot, and the first camera and the second camera are configured to capture original images including different plate corner points of the plate to be laid and the reference plate, and provide them to the plate paving control device.

[0015] A fourth aspect of the present application provides a paving robot, comprising any of the above-mentioned plate paving control systems.

[0016] A fifth aspect of the present application provides a machine-readable storage medium, on which instructions are stored, and the instructions are used to enable a machine to execute any of the above-mentioned board paving control methods.

[0017] Through the above technical solution, the embodiment of the present application uses the prior information of the plate with straight edges to perform image instance segmentation, which is equivalent to forming an improved plate instance segmentation algorithm based on position memory and edge refinement. The algorithm has low computational cost and good foreground and background segmentation capabilities, and can adapt to the actual working conditions of complex and variable light intensity in the construction industry scene.

[0018] Other features and advantages of the embodiments of the present application will be described in detail in the subsequent specific implementation section. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the following specific implementations, they are used to explain the embodiments of the present application, but do not constitute a limitation on the embodiments of the present application. In the accompanying drawings:

[0020] Figure 1 A schematic diagram of a process flow of a plate paving control method according to an embodiment of the present application is schematically shown;

[0021] Figures 2(a) and 2(b) schematically illustrate the first / single-pass slab laying scenario and the multi-pass slab laying scenario, respectively;

[0022] Figure 3 A schematic diagram of a workflow of image segmentation using an image instance segmentation model according to an embodiment of the present application is shown;

[0023] Figure 4The schematic diagram of the implementation process of the improved plate instance segmentation algorithm according to the embodiment of the present application is shown schematically;

[0024] FIG. 5( a ) and FIG. 5( b ) schematically illustrate the effects of using a traditional image instance segmentation algorithm and using the “position memory + edge refinement” segmentation algorithm of an embodiment of the present application, respectively;

[0025] Figure 6 A structural block diagram of a plate paving control device according to an embodiment of the present application is schematically shown; and

[0026] Figure 7 The structural block diagram of a board paving control system according to an embodiment of the present application is schematically shown. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the specific implementation methods described herein are only used to illustrate and explain the embodiments of the present application, and are not used to limit the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0028] It should be noted that the acquisition, transmission, storage, use, and processing of data in the technical solution of this application are in compliance with the relevant provisions of national laws and regulations. In the embodiments of this application, some existing solutions in the industry such as certain software, components, and models may be mentioned, which should be considered as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of this application, but it does not mean that the applicant has or will necessarily use the solution.

[0029] It should be noted that if the embodiments of the present application involve directional indications (such as up, down, left, right, front, back...), the directional indications are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0030] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present application, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or suggesting their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0031] Figure 1 The following schematically shows a flow chart of a method for controlling the paving of a plate according to an embodiment of the present application. Figure 1 As shown, an embodiment of the present application provides a board paving control method, which may include the following steps S100-500.

[0032] Step S100, obtaining an original image obtained by taking a picture of the current plate paving position by a camera module of the paving robot.

[0033] Among them, the camera module includes at least one camera, and it can be a 3D camera or a 2D camera. The original image captured is required to include the position information of the plate to be laid and the reference plate, especially the plate edge information and corner point information, so as to obtain the position deviation between the plate to be laid and the reference plate through subsequent image processing. In the example, only one plane camera can be set, and the camera can be guaranteed to capture the two plate corner points forming a diagonal between the plate to be laid and the reference plate by diagonal installation or omnidirectional installation. The position deviation between the plate to be laid and the reference plate is then characterized by calculating the diagonal distance. However, in the process of laying the first plate (i.e., laying the first plate), there is no diagonal between the plate to be laid and the reference plate, so that the diagonal arrangement of the plane camera will result in no reference during the laying of the first plate. In addition, this diagonal installation or omnidirectional installation method makes the camera utilization efficiency low and the cost high.

[0034] Therefore, considering that the plate has a regular shape and straight edges, and the position and posture of the plate to be laid relative to the actuator end of the robot arm are relatively fixed, in a preferred embodiment, the camera layout is adjusted. Specifically, the camera module includes a first camera and a second camera respectively installed on both sides of the actuator end of the robot arm of the paving robot, and the first camera and the second camera are configured to be able to capture different plate corners of the plate to be laid and the reference plate. Taking the first-pass / single-pass tile laying scene of Figure 2(a) and the multi-pass tile laying scene of Figure 2(b) as examples, camera A and camera B represent the first camera and the second camera, respectively. In Figure 2(a), the reference brick is a preset tile with a straight edge. After the reference brick is laid and fixed, camera A and camera B are fixed on the end of the actuator. The two cameras are installed facing one side of the reference brick. At the same time, the camera field of view needs to ensure that the right angles and corners of the reference brick and the tile to be laid can be captured simultaneously within the effective working distance; in Figure 2(b), the reference brick is a preset tile with a straight edge or a tile with a straight edge adjacent to the laying position of the tile to be laid, that is, tile 1 can be understood as another reference brick relative to the tile to be laid. In this way, the right angles and corners of the tile to be laid and tile 1 can be captured by camera A and camera B, and are not limited to the diagonal angle between the original reference brick and the tile to be laid (marked with a red line in the figure).

[0035] Further considering that 3D cameras have deficiencies in cost and real-time performance, and that relying solely on 3D visual information to determine the height of the plate laying locally will inevitably lead to error accumulation, the first camera and the second camera in the above embodiment are both planar 2D cameras. Therefore, the preferred embodiment of the present application adopts a layout scheme in which two planar 2D cameras are fixed on one side (one on the left and one on the right) at the end of the actuator along the plate laying direction, which saves costs while reducing the complexity of data transmission and processing.

[0036] Based on the above camera layout, the sheet laying actuator carries the two cameras fixed on it and moves to the predetermined shooting position. After reaching the position, the cameras are driven to take pictures to obtain the original images of the left and right cameras. After obtaining the preliminary original images, the image processing flow involved in the subsequent steps is entered.

[0037] Step S200: performing image preprocessing on the original image.

[0038] For example, image preprocessing includes: dedistorting the original image using the distortion parameters obtained through offline calibration to eliminate image distortion caused by the lens; obtaining edge-sharpened images using Gaussian filtering and Laplace transform; and increasing the brightness difference between bright and dark areas of the image through gamma correction, making the plate area image and the ground background more distinguishable.

[0039] Step S300: Process the image to be segmented by using a preset image instance segmentation model to obtain a corresponding instance segmentation result.

[0040] The image instance segmentation model is configured to perform local refinement processing on the image to be segmented in combination with prior information of the plate with straight plate edges, wherein the prior information of the plate with straight plate edges refers to prior information corresponding to a scene where the plate edges are laid straight.

[0041] In the example, Figure 3 As shown, the step S300 may include the following steps S310-S340.

[0042] Step S310 , extracting feature vectors from the pre-marked plate instance segmentation image with straight plate edges, so as to obtain a position memory vector representing the prior information of the plate.

[0043] The plate instance segmentation image is annotated offline under different environmental backgrounds and different environmental lighting.

[0044] Step S320, obtaining a feature vector to be queried corresponding to the image to be segmented.

[0045] Step S330, performing attention interaction processing on the position memory vector and the feature vector to be queried to obtain global image information.

[0046] Step S340: performing local refinement processing on the global information of the image to obtain a final instance segmentation result.

[0047] The above steps S310-S340 are performed by the image instance segmentation model, which essentially forms an improved plate instance segmentation algorithm based on position memory and edge refinement on the basis of the traditional image instance segmentation algorithm. The improved plate instance segmentation algorithm makes full use of the prior information such as the approximate position, number and shape of the plate when laying the tiles, avoids the model calculation load caused by predicting each frame of the plate image from scratch, and helps to improve the stability of the calculation.

[0048] Below through Figure 4 The following is a detailed introduction to the example implementation process of this improved plate instance segmentation algorithm. This example takes tiles as an example and specifically includes the following steps S1-S6:

[0049] Step S1: Use a common public data set for deep learning pre-training to obtain Figure 4 Pre-trained weights for the encoding, decoding and attention interactions involved;

[0050] Step S2, manually annotating a small number of tile instance segmentation images under different backgrounds and ambient lighting factors offline. For example, for each ambient background and ambient lighting, annotate 1-3 tile instance segmentation images with straight tile edges.

[0051] Step S3, extracting and storing the feature vector of the labeled actual tile instance segmentation image using the pre-trained weights, and obtaining the position memory vector representation of the tile prior information.

[0052] Step S4, the new image to be processed is input into the query encoder module to obtain the feature vector to be queried. It should be noted that in the embodiment of the present application, the image to be processed, the picture to be predicted and the image to be segmented can be understood equivalently.

[0053] Step S5, performing L attention interactions between the feature vector to be queried and the tile image feature vector obtained after actual annotation (i.e., the position memory vector), where L is generally 2 to 4.

[0054] In step S6, a multi-layer perceptron is used to locally refine the global information of attention interaction to obtain the final instance segmentation result.

[0055] The effect diagram of the instance segmentation result obtained through steps S1-S6 is shown in Figure 5(b), where Figure 5(a) is the ordinary segmentation effect obtained by using the traditional image instance segmentation algorithm. It is easy to see that the segmentation edge is rough and wavy, while Figure 5(b) is the innovative "position memory + edge refinement" segmentation effect of the embodiment of the present application. It is easy to see that the segmentation edge is neat and uniform.

[0056] In this way, the embodiment of the present application adopts a deep learning method to ensure the good foreground and background segmentation capabilities of the image instance segmentation model, which provides high-quality perception data for the plate corner positioning algorithm process involved in subsequent steps.

[0057] Step S400: performing image post-processing on the instance segmentation result to obtain a position deviation between the plate to be laid and the reference plate.

[0058] In a preferred embodiment of the present application, the image post-processing of step S400 further includes: determining the corner points of the plate in the image based on the instance segmentation result; and determining the position deviation between the plate to be laid and the reference plate based on the pixel position of the determined corner points of the plate, wherein the position deviation includes the corner point distance and angle deviation between the plate to be laid and the reference plate.

[0059] That is, the post-processing involved in step S400 includes two parts: corner point positioning and position deviation calculation. These two parts are described in detail below.

[0060] 1. Corner point positioning.

[0061] In an embodiment of the present application, determining the corner points of the plate in the image includes: extracting a set of multiple straight lines in the instance segmentation result through straight line fitting; calculating the intersection points between each straight line and determining whether the corresponding intersection points are the corner points of the plate; and traversing the set to determine the corner points of the plate.

[0062] For example, combining Figure 4 , which is different from the traditional operation mode of obtaining a mask by processing a plate image, the embodiment of the present application directly obtains the plate instance segmentation result of the visible area and obtains the plate absolute area mask, thereby avoiding the traditional binarization operation process; and compared with the traditional threshold segmentation, there is no need to manually compare the detection accuracy of different thresholds for the plate edge to determine the fixed threshold experience value, which can avoid the scene limitations caused by the fixed threshold. Further, the post-processing process steps for determining the corner point in the example include: for the plate instance segmentation result, extracting the main straight lines contained in the edge of the plate instance absolute area mask by PCA straight line fitting, and then calculating the intersection between the straight lines based on the obtained local straight line slope information, if the slopes of the two straight lines at the intersection are perpendicular and the intersection is on the inside of the rough position of the corresponding corner point of the plate absolute area, then the pixel position of the intersection is determined to be the visible corner point of the plate. If all the straight line sets within the absolute area mask range of the plate are traversed and no suitable corner point is found, then extract the local texture map of the corresponding position of the pre-processed enhanced image according to the mask, and re-execute the post-processing process steps until the required plate corner point is finally determined.

[0063] 2. Calculation of position deviation.

[0064] For example, for the layout of the two cameras in Figure 2(a) and Figure 2(b), after the previous step realizes the detection of all visible tile corner points and the determination of pixel positions within the field of view of the left and right cameras, the actual corner point distance is determined by the pixel accuracy corresponding to the camera height, and the position deviation between the corner points of the tiles to be laid and the corresponding reference tile corner points within the field of view of each camera is calculated through the fixed corner point position correspondence within a single camera. The position deviation includes the corresponding corner point distance and angle deviation.

[0065] Step S500, correcting the plate laying position according to the position deviation.

[0066] Taking the layout of the two cameras in Figures 2(a) and 2(b) as an example, the correction includes: obtaining a first position deviation and a second position deviation obtained by processing the original images taken by the first camera and the second camera respectively; comprehensively processing the first position deviation and the second position deviation in combination with the current paving height to determine the position correction amount and angle correction amount for paving the plate; and controlling the robot arm to correct the paving position of the plate according to the determined position correction amount and angle correction amount.

[0067] For example, the obtained position deviation is converted into the two-dimensional coordinate system where the reference plate is located to establish the relative position relationship between the plate to be laid and the reference plate in the plane, and the translation position correction and rotation angle correction required to align the parallel side surfaces of the two plate edges are calculated as the current plate posture change that needs to be adjusted.

[0068] In the actual operation process, the visual recognition and positioning algorithm process represented by steps S100-S500 runs in real time, and the calibrated external parameters of the two original images collected are continuously converted to the same plane of the virtual camera (i.e., the virtual plane) at different paving heights to ensure that the benchmarks for subsequent calculations of the original images of the two cameras are consistent; the position correction amount and the angle deviation amount are calculated, and the paving height is determined in combination with the global laser collimator until the paving effect reaches the final plate gap spacing and height requirements. Corresponding to Figure 2(a) and Figure 2(b), this visual recognition and positioning algorithm solution is suitable for the first / single-pass paving scenario and the multi-pass paving scenario.

[0069] It should be noted that the above examples mainly take ceramic tiles as an example, but the boards targeted by the embodiments of the present application are not limited to ceramic tiles, but can also include any type of boards such as stone, wood, gypsum board, cement board, foam board, fiber board, synthetic board, etc.

[0070] In summary, the embodiments of the present application provide a low-cost plate laying control method with strong environmental adaptability, which has at least the following advantages:

[0071] First, the plate paving position recognition method of the embodiment of the present application utilizes the plate prior information of the straight edge of the plate to perform image instance segmentation, which is equivalent to forming an improved plate instance segmentation algorithm based on position memory and edge refinement, avoiding the model calculation load caused by predicting each frame of the plate image from scratch, with low calculation cost, and can provide detection images with higher stability and segmentation accuracy for image post-processing. Moreover, compared with the traditional canny operator, threshold segmentation, binarization and line detection algorithms, the improved plate instance segmentation algorithm of the embodiment of the present application has good foreground and background segmentation capabilities, can provide high-quality perception data, and thus adapt to the actual working conditions of complex and changeable light intensity in the construction industry scene.

[0072] Second, the embodiment of the present application involves a single-sided layout of dual-vision cameras and a matching intelligent visual recognition and positioning algorithm, which can obtain the posture correction value (i.e., position correction value and angle correction value) of the plate to be laid in the plane, thereby realizing high-precision plate laying, which is more practical for single-channel and multi-channel plate laying scenarios.

[0073] Third, unlike diagonal or omnidirectional installation, the embodiment of the present application takes into account the cost and data processing complexity through a reasonable one-sided layout of dual-vision cameras, while ensuring the calculation accuracy. This camera layout has good adaptability and high utilization efficiency. In addition, the embodiment of the present application can adopt a pure 2D vision method of visual recognition and positioning algorithm, which is lower in cost and relatively simple in data processing.

[0074] Fourth, the image post-processing position deviation calculation involved in the embodiment of the present application avoids the traditional binarization operation process. In addition, the main straight lines are extracted using the prior information of the straight edge of the plate, and the plate position deviation is determined in the form of intersections, which has higher accuracy and reliability.

[0075] Figure 6 The structure block diagram of a board paving control device according to an embodiment of the present application is schematically shown. Figure 6 As shown, the device may include: a memory configured to store instructions; and a processor configured to call instructions from the memory and implement the above-mentioned board paving control method when executing the instructions.

[0076] The board paving control device may be a built-in controller or a remote controller of the paving robot. In addition, other implementation details and effects of the board paving control device may also refer to the embodiments of the board paving control method, which will not be described in detail here.

[0077] Figure 7 A structural block diagram of a plate paving control system according to an embodiment of the present application is schematically shown, the system comprising: the above-mentioned plate paving control device, used to control a paving robot to perform plate paving; and a camera module, comprising a first camera and a second camera respectively installed on both sides of the actuator end of the mechanical arm of the paving robot, and the first camera and the second camera are configured to capture original images including different plate corner points of the plate to be laid and the reference plate, and provide the images to the plate paving control device.

[0078] Among them, regarding the camera layout, reference may be made to the above-mentioned embodiment of the plate paving control method, which will not be described in detail here.

[0079] The panel paving control system is used in conjunction with other systems of the paving robot to control the robot to automatically lay tiles. For example, for the paving robot, its chassis and perception execution system comprehensively utilize visual correction, vibration leveling and laser alignment auxiliary systems to achieve automatic alignment and positioning of panel paving, and complete indoor high-precision mapping and navigation through SLAM navigation radar and omnidirectional mobile chassis system, thereby realizing indoor paving path planning. The autonomous mobile robot platform constructed by the panel paving control device realizes the automation of the traditional manual tile paving process. The end effector needs to detect the position deviation and angle deviation between the tiles that have been laid and the tiles to be laid through the intelligent perception of the camera module. At the same time, in order to ensure the flatness of the entire surface of the indoor paving, a laser collimator is also required to provide tile paving height reference data.

[0080] An embodiment of the present application also provides a paving robot, comprising any of the above-mentioned plate paving control systems.

[0081] An embodiment of the present application also provides a machine-readable storage medium, on which instructions are stored, and the instructions are used to enable a machine to execute the above-mentioned board paving control method.

[0082] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0083] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0084] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0085] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0086] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0087] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0088] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0089] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0090] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A board paving control method, characterized in that: include: Acquire an original image obtained by a camera module of the paving robot taking a picture of the current plate paving position, wherein the original image includes position information of the plate to be paved and the reference plate; Performing image preprocessing on the original image to obtain an image to be segmented; Processing the image to be segmented by a preset image instance segmentation model to obtain a corresponding instance segmentation result, wherein the image instance segmentation model is configured to perform local refinement processing on the image to be segmented in combination with prior information of a plate having a straight edge; Performing image post-processing on the instance segmentation result to obtain a position deviation between the to-be-laid plate and the reference plate; and The plate laying position is corrected according to the position deviation.

2. The plate paving control method according to claim 1, characterized in that: The reference plate is a preset plate with straight edges or a plate adjacent to the paving position of the plate to be paved and with straight edges.

3. The plate paving control method according to claim 1, characterized in that: Processing the image to be segmented by the image instance segmentation model includes: Extracting feature vectors from a pre-marked plate instance segmentation image with straight plate edges to obtain a position memory vector representing prior information of the plate; Obtaining a feature vector to be queried corresponding to the image to be segmented; Performing attention interaction processing on the position memory vector and the feature vector to be queried to obtain global image information; and The global information of the image is locally refined to obtain a final instance segmentation result.

4. The plate paving control method according to claim 3, characterized in that: The plate instance segmentation images are annotated offline under different environmental backgrounds and different environmental illuminations.

5. The plate paving control method according to claim 1, characterized in that: The image post-processing includes: Determine the plate corner points in the image according to the instance segmentation result; and According to the determined pixel positions of the corner points of the plate, the position deviation between the plate to be laid and the reference plate is determined, wherein the position deviation includes the corner point distance and angle deviation between the plate to be laid and the reference plate.

6. The plate paving control method according to claim 5, characterized in that: Determining the plate corner points in the image includes: Extracting a set of multiple straight lines in the instance segmentation result by straight line fitting; Calculate the intersection points between the straight lines and determine whether the corresponding intersection points are the corner points of the plate; and The set is traversed to determine the plate corner points.

7. The plate paving control method according to claim 1, characterized in that: The camera module includes a first camera and a second camera respectively installed on both sides of the actuator end of the mechanical arm of the paving robot, and the first camera and the second camera are configured to be able to capture different plate corner points of the plate to be laid and the reference plate.

8. The plate paving control method according to claim 7, characterized in that: The correcting of the plate paving position according to the position deviation comprises: Acquire a first position deviation and a second position deviation obtained by processing original images respectively taken by the first camera and the second camera; Comprehensively processing the first position deviation and the second position deviation in combination with the current paving height to determine a position correction amount and an angle correction amount for paving the plate; and The robot arm is controlled to correct the plate laying position according to the determined position correction amount and angle correction amount.

9. A board paving control device, characterized in that: include: a memory configured to store instructions; as well as A processor is configured to call the instruction from the memory and implement the board paving control method according to any one of claims 1 to 8 when executing the instruction.

10. A board paving control system, characterized in that: include: The board paving control device as described in claim 9 is used to control a paving robot to perform board paving; as well as The camera module includes a first camera and a second camera respectively installed on both sides of the actuator end of the mechanical arm of the paving robot, and the first camera and the second camera are configured to capture original images including different plate corner points of the plate to be paved and the reference plate, and provide them to the plate paving control device.

11. A paving robot, characterized in that: A board paving control system comprising claim 10.

12. A machine-readable storage medium, characterized in that: The machine-readable storage medium stores instructions for enabling the machine to execute the board paving control method according to any one of claims 1 to 8.

Images