An indoor decoration robot site planning method and device based on a three-dimensional semantic map

Abstract

The present invention discloses a method and device for indoor decoration robot station planning based on a three-dimensional semantic map, which can solve the technical problems existing in current decoration robots that during the path planning process, the calculated work stations are only applicable to a single scenario and lack generality, and the accuracy of the stations planned based on experience is not high. The method includes: generating a three-dimensional semantic map of the to-be-decorated task; obtaining the target object parameters and working mode of the room where the robot is located; deriving points according to the target object parameters, including: calculating the initial point and end point poses in the manipulator coordinate system through the inverse kinematics algorithm, and obtaining the key point coordinates by using the multi-agent reinforcement learning algorithm; generating the point planning data for this room; and performing point planning for all rooms under the to-be-decorated task according to the generated single-room planned points.

Description

Technical Field

[0001] The present invention belongs to the field of indoor decoration robots, and particularly relates to a method and device for site planning of indoor decoration robots based on a three-dimensional semantic map. Background Art

[0002] In the fields of indoor decoration and auxiliary construction, the application of robots is becoming more and more extensive. Therefore, researching the working path planning of robots is of great significance in improving operation efficiency, ensuring construction quality, reducing safety risks, saving labor costs, and enhancing flexibility.

[0003] In previous path planning, the working sites of robots were often calculated manually relying on past experience. Generally, workers or technicians relied on past experience to calculate specific working sites. The disadvantages of this method are as follows: (1) The calculated working sites are only applicable to a single scenario and do not have universality. Each time the scenario is changed, the working sites need to be recalculated, which is costly and very time-consuming. (2) The accuracy of the sites planned based on experience is not high and the error is large. Summary of the Invention

[0004] Aiming at the defects of the related technology, the purpose of the present invention is to provide a method and device for site planning of indoor decoration robots based on a three-dimensional semantic map, aiming to solve the technical problems of single applicable scenario and low accuracy in the site planning of existing decoration robots.

[0005] To solve the above technical problems, the present invention is implemented as follows.

[0006] According to one aspect of the present invention, there is provided a method for site planning of indoor decoration robots based on a three-dimensional semantic map, including the following steps:

[0007] S1. Generate a three-dimensional semantic map of the to-be-decorated task, specifically including the following steps:

[0008] Collect indoor environmental data and extract geographical images;

[0009] Extract features and classify the collected geographical images to identify different types of entities.

[0010] Based on the types of each entity, determine the pixel values of each entity;

[0011] Extract the contours of each entity object through the geographical images, and perform semantic fusion on the pixel values of each entity and the corresponding contours to obtain a quadruple SCR<D, V, N, R> for each entity, where D represents the entity object type, V represents the set of three-dimensional coordinate information of the outer edge of the contour, N represents the set of three-dimensional coordinate information of each pixel point in the contour layer, and R represents the pixel value;

[0012] Generate a three-dimensional semantic map through each entity quadruple;

[0013] Based on the decoration task to be performed, plan the path of the robot to each door of this task through the SLAM path planning algorithm;

[0014] Drive the robot to move to each door, and repeat step S1 after moving to each door until the three-dimensional semantic map of the entire task is obtained;

[0015] S2. Obtain the target object parameters and working mode of the room where the robot is located, specifically including the following steps:

[0016] S21. The robot displays the three-dimensional semantic map of the task obtained in step S1 on the control terminal, and the user clicks on each entity in the room where the robot is located in sequence to determine each target object;

[0017] S22. The robot scans each target object, obtains each target object parameter, forms an array of each target object parameter, and stores it in the cache; the array of each target object parameter is stored in the cache in the form of a one-dimensional array, a[n] = {a1, a2,... a n}, where n is the number of target objects obtained in step S21, and a n is the parameter array of the nth target object;

[0018] S23. Determine the working mode of the robot, and the working mode is spraying or grinding;

[0019] S3. Deduce the points according to the target object parameters, specifically including the following steps:

[0020] S31. Obtain the parameter array of the target object a1 from the cache and obtain the working mode from step S23;

[0021] S32. Obtain the initial working pose and the end working pose in the three-dimensional coordinate system;

[0022] S33. Calculate the second initial working pose and the second end working pose in the robotic arm coordinate system through the inverse kinematics algorithm;

[0023] S34. Using the rotatable angle of the robotic arm set by the user as the state set and {-a, a, 0} as the action set, according to the preset reward and punishment mechanism, use the multi-agent reinforcement learning algorithm to generate the key point coordinates from the second initial working pose to the second end working pose;

[0024] Among them, when the working mode is spraying, a is the preset minimum spraying range of the robot; when the working mode is grinding, a is the preset minimum movement step of the robotic arm;

[0025] S4. Perform single-room point location planning to generate point location planning data for this room;

[0026] S5. Perform point location planning for all rooms under the decoration task according to the generated single-room planned point locations.

[0027] According to another aspect of the present invention, there is provided an indoor decoration robot site planning device based on a three-dimensional semantic map, including: a semantic map generation module, a parameter acquisition module, a point location derivation module, a single-room planning module, and a multi-room planning module. Each module is installed inside the robot, and data is transmitted between them through a wireless communication protocol.

[0028] The semantic map generation module is configured to generate a three-dimensional semantic map of the decoration task, which is specifically implemented in the following manner: collect indoor environment data through sensors built in the robot, and extract geographical images;

[0029] Further, the geographical image is a three-dimensional image of the room where the robot is located;

[0030] Perform feature extraction and classification on the collected geographical images to identify different types of entities.

[0031] Based on the types of each entity, determine the pixel values of each entity;

[0032] Further, extract the outlines of each entity object through the geographical image, perform semantic fusion on the pixel values of each entity and the corresponding outlines to obtain a quadruple SCR<D, V, N, R> for each entity, where D represents the entity object type, V represents the set of three-dimensional coordinate information of the outer edge of the outline, N represents the set of three-dimensional coordinate information of each pixel point in the outline layer, and R represents the pixel value;

[0033] Generate a three-dimensional semantic map through the quadruples of each entity;

[0034] Based on the decoration task, plan the paths of the robot to each door of this task through the SLAM path planning algorithm;

[0035] Drive the robot to move to each door, and repeat step S1 after moving to each door until the three-dimensional semantic map of the entire task is obtained;

[0036] The parameter acquisition module is configured to acquire the target object parameters and working modes of the room where the robot is located, which is specifically implemented in the following manner:

[0037] S21. The robot displays the three-dimensional semantic map of the task obtained in step S1 on the control terminal, and the user clicks on each entity in the room where the robot is located in turn to determine each target object;

[0038] S22. The robot scans each target object, obtains the parameters of each target object, forms an array of target object parameters, and stores it in the cache; the arrays of target object parameters are stored in the cache in the form of a one-dimensional array, a[n] = {a1, a2, …… a n}, where n is the number of target objects obtained in step S21, and a n is the parameter array of the nth target object;

[0039] S23. Determine the working mode of the robot, and the working mode is spraying or grinding;

[0040] The point position derivation module is configured to derive the point positions according to the target object parameters, and is specifically implemented in the following manner:

[0041] S31. Obtain the parameter array of the target object a1 from the cache and obtain the working mode from step S23;

[0042] S32. Obtain the initial working pose and the end working pose in the three-dimensional coordinate system;

[0043] S33. Calculate the second initial working pose and the second end working pose in the robotic arm coordinate system through the inverse kinematics algorithm;

[0044] S34. Using the rotatable angle of the robotic arm set by the user as the state set and {-a, a, 0} as the action set, according to the preset reward and punishment mechanism, use the multi-agent reinforcement learning algorithm to generate the key point coordinates from the second initial working pose to the second end working pose;

[0045] Among them, when the working mode is spraying, a is the preset minimum spraying range of the robot; when the working mode is grinding, a is the preset minimum movement step of the robotic arm;

[0046] The single-room planning module is configured to perform single-room point position planning and generate the point position planning data for this room;

[0047] The multi-room planning module is configured to perform point position planning for all rooms in this scenario according to the generated single-room planning point positions.

[0048] The beneficial effects of the present invention are:

[0049] (1) Derive the point positions of all rooms according to the target object parameters and the working mode, so that the measured workstations are applicable to different scenarios, have universality, and save time and labor costs. (2) The point positions planned according to the present invention have high accuracy. (3) When performing room point position planning, use the cache to store and read the target parameters, so that the planned point positions of multiple objects in the room can be obtained simultaneously, improving the planning efficiency. Description of the Drawings

[0050] Figure 1 Flow schematic diagram of an indoor decoration robot site planning method based on a three - dimensional semantic map according to the present invention;

[0051] Figure 2 Schematic diagram of each module of an indoor decoration robot site planning device based on a three - dimensional semantic map according to the present invention; Specific embodiments

[0052] The present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0053] Embodiment 1

[0054] As Figure 1 shown, the present invention proposes an indoor decoration robot site planning method based on a three - dimensional semantic map, which specifically includes the following steps:

[0055] S1. Generate a three - dimensional semantic map of the to - be - decorated task;

[0056] Further, the to - be - decorated task is the task received by the indoor decoration robot, including all rooms that need to be operated by the indoor decoration robot this time;

[0057] Collect indoor environment data through sensors built in the robot, and perform data processing and analysis to extract geographical images;

[0058] Further, the geographical image is a three - dimensional image of the room where the robot is located;

[0059] Further, the sensor is a binocular camera or a lidar, and the sensor can rotate 360 degrees to collect indoor environment data in all directions;

[0060] Extract features and classify the collected geographical images to identify different types of entities.

[0061] Specifically, use the collected geographical image as the input of a small - sample fine - grained image classification model based on Transformer to obtain different types of entities;

[0062] Further, the types of the entities are walls, corners, doors, windows, columns, ceiling slabs, and crossbeams;

[0063] Based on the types of each entity, determine the pixel values of each entity;

[0064] Furthermore, the background is black with a pixel value of RGB(0, 0, 0), the wall is green with a pixel value of RGB(0, 255, 0), the corner is purple with a pixel value of RGB(255, 0, 255), the door is yellow with a pixel value of RGB(255, 255, 0), the window is sky blue with a pixel value of RGB(0, 255, 255), the column is red with a pixel value of RGB(255, 0, 0), the ceiling slab is dark gray with a pixel value of RGB(195, 195, 195), and the crossbeam is light gray with a pixel value of RGB(127, 127, 127).

[0065] Furthermore, extract the contours of each entity object from the geographical image, perform semantic fusion on the pixel values of each entity with the corresponding contours, and obtain the quadruple SCR<D, V, N, R> of each entity, where D represents the entity object type, V represents the set of three-dimensional coordinate information of the outer edge of the contour, N represents the set of three-dimensional coordinate information of each pixel point in the contour layer, and R represents the pixel value;

[0066] The contour layer is the layer between the outer edge of the contour and the interior of the entity;

[0067] Generate a three-dimensional semantic map through the quadruples of each entity;

[0068] Furthermore, identify and mark the entities of type door in each quadruple;

[0069] Based on the decoration task to be performed, plan the path of the robot to each door of this task through the SLAM path planning algorithm;

[0070] Drive the robot to move to each door, and repeat step S1 after moving to each door until the three-dimensional semantic map of the entire task is obtained;

[0071] Judge the entrance door of the task according to the three-dimensional semantic map, and construct a three-dimensional coordinate system for this three-dimensional semantic map with the entrance door as the origin of the three-dimensional coordinate system;

[0072] Run the robot to the origin of the coordinates.

[0073] S2. Obtain the target object parameters and working methods of the room where the robot is located, which specifically include the following steps:

[0074] S21. The robot displays the three-dimensional semantic map of the task obtained in step S1 on the control terminal, and the user clicks on each entity in the room where the robot is located in turn to determine each target object;

[0075] S22. The robot scans each target object, obtains the parameters of each target object, forms an array of the parameters of each target object, and stores it in the cache. Among them, the parameter array includes: type, height, width, L(x1, y1, z1), R(x2, y2, z2);

[0076] The L(x1, y1, z1) is the coordinate of the left-bottommost point of the target object in the three-dimensional coordinate system and is the initial working point.

[0077] The R(x2, y2, z2) is the coordinate of the right-topmost point of the target object in the three-dimensional coordinate system and is the end working point.

[0078] Among them, x 1、 y 1、 z1 are respectively the values of the left-bottommost point of the target object on the x-axis, y-axis, and z-axis in the three-dimensional coordinate system.

[0079] Furthermore, each target object parameter array is stored in the cache in the form of a one-dimensional array, a[n] = {a1, a2,... a n}, where n is the number of target objects obtained in step S21, and a n = {type, height, width, L(x1, y1, z1), R(x2, y2, z2)};

[0080] The type is wall, corner, door, window, column, ceiling, beam, the same as in step S1;

[0081] S23. Determine the working mode of the robot, and the working mode is spraying and grinding;

[0082] Furthermore, the working mode is selected by the user at the control terminal.

[0083] S3. Deduce the positions according to the target object parameters, specifically including the following steps:

[0084] S31. Obtain the parameter array of the target object a1 from the cache and obtain the working mode from step S23;

[0085] S32. Obtain the initial working position and orientation and the end working position and orientation in the three-dimensional coordinate system;

[0086] Furthermore, the initial working position and orientation and the end working position and orientation in the three-dimensional coordinate system can be respectively expressed as (x1, y1, z1, α1, β1, γ1) and (x2, y2, z2, α2, β2, γ2). α1, β1, γ1 are respectively the preset maximum attitude angles of the robotic arm at the initial working point in three dimensions, and α2, β2, γ2 are respectively the preset maximum attitude angles of the robotic arm at the end working point in three dimensions;

[0087] S33. Calculate the second initial working position and orientation and the second end working position and orientation in the robotic arm coordinate system through the inverse kinematics algorithm;

[0088] S34. With the rotatable angle of the robotic arm set by the user as the state set and {-a, a, 0} as the action set, according to the preset reward and punishment mechanism, use the multi-agent reinforcement learning algorithm to generate the key point coordinates from the second initial working point pose to the second end working point pose;

[0089] Among them, the rotatable angle of the robotic arm set by the user is set according to the actual needs of the user. For example

[0090] {-20, -10, 0, 10, 20}, but not limited to this.

[0091] Among them, when the working mode is spraying, a is the preset minimum spraying range of the robot; when the working mode is grinding, a is the preset minimum movement step of the robotic arm;

[0092] The reward and punishment mechanism is as follows:

[0093] If the robotic arm reaches the second end pose and does not collide, set the reward to N1, mark it as the end of the task, and end the round;

[0094] Before the robotic arm reaches the second end pose, there is a penalty of N2 for each simulation movement;

[0095] Before the robotic arm reaches the second end pose, if the Euclidean distance between the pose in the joint coordinate system and the second end pose becomes smaller after a simulation movement, a reward of N3 is obtained;

[0096] If the robotic arm collides with an obstacle or itself during the movement process, a penalty of -N1 is obtained, and it is marked as the end of the task and the round is ended. The values of N1, N2, and N3 can be set according to actual needs; for example, N1 = 1, N2 = -0.01, N3 = 0.02, but not limited to this.

[0097] S4. Perform single-room point planning to generate the point planning data for this room, specifically including the following steps:

[0098] S41. Sequentially obtain a2,... a from the cache n , and at the same time generate the point positions of each target object according to the same method as in step S3;

[0099] S42. Obtain the planned point positions between each target object according to the SLAM algorithm;

[0100] S44. Merge the point positions of each target object and the planned point positions between the target objects to obtain the planned point positions of the room and store them in the cache;

[0101] S5. Perform point planning for all rooms under the to-be-decorated tasks according to the generated single-room planned point positions, specifically including the following steps:

[0102] S51. Click on the physical object door that can enter the next room at the control end, and drive the robot to that position;

[0103] S52. Repeat steps S2 - S4 to obtain the planned points of the next room and store them in the cache;

[0104] S53. Repeat steps S51 - S52 until the planning of all room points is completed;

[0105] S54. Obtain the coordinates of all physical object doors that the user clicks on and can enter the next room, retrieve all the planned room points from the cache, and merge the obtained door coordinates with all the planned room points to obtain the planned points of all rooms in this scenario.

[0106] Embodiment 2

[0107] As Figure 2 shown, the present invention proposes an indoor decoration robot site planning device based on a three - dimensional semantic map, including: a semantic map generation module, a parameter acquisition module, a point derivation module, a single - room planning module, and a multi - room planning module. Each module is installed inside the robot, and data is transmitted between them through a wireless communication protocol.

[0108] The semantic map generation module is configured to generate a three - dimensional semantic map of the decoration task to be performed;

[0109] Further, the decoration task to be performed is the task received by the indoor decoration robot, including all the rooms that need to be operated by the indoor decoration robot this time;

[0110] Collect indoor environment data through sensors built in the robot, and perform data processing and analysis to extract a geographical image;

[0111] Further, the geographical image is a three - dimensional image of the room where the robot is located;

[0112] Further, the sensor is a binocular camera or a lidar, and this sensor can rotate 360 degrees to collect indoor environment data in all directions;

[0113] Extract features and classify the collected geographical image to identify different types of entities.

[0114] Specifically, use the collected geographical image as the input of a small - sample fine - grained image classification model based on Transformer to obtain various different types of entities;

[0115] Further, the types of the entities are walls, corners, doors, windows, columns, ceiling panels, and cross - beams;

[0116] Determine the pixel values of each entity based on the type of each entity;

[0117] Further, the background is black with a pixel value of RGB(0, 0, 0), the wall is green with a pixel value of RGB(0, 255, 0), the corner is purple with a pixel value of RGB(255, 0, 255), the door is yellow with a pixel value of RGB(255, 255, 0), the window is sky blue with a pixel value of RGB(0, 255, 255), the column is red with a pixel value of RGB(255, 0, 0), the ceiling slab is dark gray with a pixel value of RGB(195, 195, 195), and the crossbeam is light gray with a pixel value of RGB(127, 127, 127).

[0118] Further, extract the contours of each entity object from the geographical image, perform semantic fusion on the pixel values of each entity with the corresponding contours, and obtain the quadruple SCR<D, V, N, R> of each entity, where D represents the entity object type, V represents the set of three-dimensional coordinate information of the outer edge of the contour, N represents the set of three-dimensional coordinate information of each pixel point in the contour layer, and R represents the pixel value;

[0119] The contour layer is the layer between the outer edge of the contour and the inside of the entity;

[0120] Generate a three-dimensional semantic map through the quadruples of each entity;

[0121] Further, identify and mark the entities of type door in each quadruple;

[0122] Based on the decoration task to be performed, plan the path of the robot to each door of this task through the SLAM path planning algorithm;

[0123] Drive the robot to move to each door, and repeat step S1 after moving to each door until the three-dimensional semantic map of the entire task is obtained;

[0124] Judge the entrance door of the task according to the three-dimensional semantic map, construct a three-dimensional coordinate system for this three-dimensional semantic map with the origin of the three-dimensional coordinate system as this entrance door, and then form the mapping relationship between each quadruple and its corresponding three-dimensional coordinates SCR<D, V, N, R> → R(x2, y2, z2), where R(x2, y2, z2) is the coordinate of the entity's top rightmost point in the two-dimensional coordinate system;

[0125] Among them, x2, y2, and z2 are the values of the entity's top rightmost point on the x-axis, y-axis, and z-axis of the three-dimensional coordinate system respectively;

[0126] Run the robot to the origin of the coordinates.

[0127] A parameter determination module, configured to obtain the target object parameters and working mode of the room where the robot is located, and is specifically implemented in the following manner:

[0128] S21. The robot displays the three-dimensional semantic map of the task obtained in step S1 on the control terminal, and the user clicks on each entity in the room where the robot is located in sequence to determine each target object;

[0129] S22. The robot scans each target object, obtains the parameters of each target object, forms an array of parameters for each target object, and stores it in the cache. Among them, the parameter array includes: type, height, width, L(x1, y1, z1), R(x2, y2, z2);

[0130] The L(x1, y1, z1) is the coordinate of the left-bottommost point of the target object in the three-dimensional coordinate system;

[0131] Among them, x 1、 y 1、 z1 are the values of the left-bottommost point of the target object on the x-axis, y-axis, and z-axis of the three-dimensional coordinate system respectively;

[0132] Furthermore, the parameter arrays of each target object are stored in the cache in the form of a one-dimensional array, a[n] = {a1, a2,... a n}, where n is the number of target objects obtained in step S21, and a n = {type, height, width, L(x1, y1, z1), R(x2, y2, z2)};

[0133] The type is wall, corner, door, window, column, ceiling, beam, which is the same as in step S1;

[0134] S23. Determine the working mode of the robot, and the working mode is spraying and grinding;

[0135] Furthermore, the working mode is selected by the user on the control terminal.

[0136] A point position derivation module, configured to derive point positions according to the target object parameters, and is specifically implemented in the following manner:

[0137] S31. Obtain the parameter array of the target object a1 from the cache and the working mode obtained from step S23;

[0138] S32. Obtain the initial working pose and the end working pose in the three-dimensional coordinate system;

[0139] Further, the initial working point pose and the end working point pose in the three-dimensional coordinate system can be respectively expressed as (x1, y1, z1, α1, β1, γ1) and (x2, y2, z2, α2, β2, γ2), where α1, β1, and γ1 are respectively the preset maximum pose angles of the manipulator at the initial working point in three dimensions, and α2, β2, and γ2 are respectively the preset maximum pose angles of the manipulator at the end working point in three dimensions;

[0140] S33. Calculate the second initial working point pose and the second end working point pose in the manipulator coordinate system through the inverse kinematics algorithm;

[0141] S34. Taking the rotatable angles of the manipulator set by the user as the state set and {-a, a, 0} as the action set, according to the preset reward and punishment mechanism, use the multi-agent reinforcement learning algorithm to generate the key point coordinates from the second initial working point pose to the second end working point pose;

[0142] Among them, the rotatable angles of the manipulator set by the user are set according to the actual needs of the user. For example

[0143] {-20, -10, 0, 10, 20}, but not limited to this.

[0144] Among them, when the working mode is spraying, a is the preset minimum spraying range of the robot; when the working mode is grinding, a is the preset minimum movement step of the manipulator;

[0145] The reward and punishment mechanism is as follows:

[0146] If the manipulator reaches the second end pose and does not collide, set the reward to N1, mark it as the end of the task, and end the round;

[0147] Before the manipulator reaches the second end pose, there is a penalty of N2 for each simulation movement;

[0148] Before the manipulator reaches the second end pose, if the pose in the joint coordinate system becomes closer to the second end pose after a simulation movement, a reward of N3 is obtained;

[0149] If the manipulator collides with an obstacle or itself during the movement process, a penalty of -N1 is obtained, mark it as the end of the task, and end the round. The values of N1, N2, and N3 can be set according to actual needs; for example, N1 = 1, N2 = -0.01, N3 = 0.02, but not limited to this.

[0150] The single-room planning module is configured to perform single-room point planning and generate the point planning data for this room, which is specifically implemented through the following methods:

[0151] S41. Sequentially obtain a2,... a from the cachen , and generate the point positions of each target object according to the same method as in step S3;

[0152] S42. Obtain the planned point positions between each target object according to the SLAM algorithm;

[0153] S44. Merge the point positions of each target object and the planned point positions between the target objects to obtain the planned point positions of the room, and store them in the cache.

[0154] The multi-room planning module is configured to perform point position planning for all rooms in this scenario according to the generated single-room planned point positions, and is specifically implemented in the following manner:

[0155] S51. Click on the physical object door that can enter the next room on the control terminal, and drive the robot to this position;

[0156] S52. Repeat steps S2 - S4 to obtain the planned point positions of the next room, and store them in the cache;

[0157] S53. Repeat steps S51 - S52 until the point position planning of all rooms is completed;

[0158] S54. Obtain the coordinates of all physical object doors that the user clicks on and can enter the next room, retrieve all room planned point positions from the cache, and merge the obtained door coordinates with all room planned point positions to obtain the planned point positions of all rooms in this scenario.

[0159] Embodiment 3

[0160] This application embodiment also provides an electronic device, which includes a memory and a processor. The memory is used to store a computer program, and the processor runs the computer program to enable the computer device to execute the indoor decoration and auxiliary construction robot site planning method based on a three-dimensional semantic map described in Embodiment 1.

[0161] This application embodiment also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is read and run by a processor, it executes the indoor decoration and auxiliary construction robot site planning method based on a three-dimensional semantic map described in Embodiment 1.

[0162] In several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the modules is only a logical function division. In actual implementation, there can be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces. The indirect couplings or communication connections of the devices or units can be in electrical, mechanical or other forms.

[0163] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0164] In addition, in each embodiment of the present invention, the functional units can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of a combination of hardware and software functional units.

[0165] The above-mentioned integrated units implemented in the form of software functional units can be stored in a computer-readable storage medium. The above-mentioned software functional units are stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a physical machine server, or a network cloud server, etc., and needs to install the Windows or Windows Server operating system) to execute some steps of the methods described in each embodiment of the present invention. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM), random access memories (RAM), magnetic disks or optical discs that can store program codes.

[0166] The above are only the preferred embodiments of the present invention, and do not impose any form of limitation on the present invention. Any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention still fall within the scope of the technical solution of the present invention.

Claims

1. An indoor decoration robot site planning method based on a three-dimensional semantic map, comprising the following steps: S1. Generate a three-dimensional semantic map of the to-be-decorated task, specifically including the following steps: Collect indoor environment data and extract geographical images; Extract features and classify the collected geographical images to identify different types of entities; Determine the pixel values of each entity based on the type of each entity; Extract the contours of each entity object through the geographical images, perform semantic fusion on the pixel values of each entity and the corresponding contours, and obtain a quadruple SCR<D, V, N, R> for each entity, where D represents the entity object type, V represents the set of three-dimensional coordinate information of the outer edge of the contour, N represents the set of three-dimensional coordinate information of each pixel point in the contour layer, and R represents the pixel value; Generate a three-dimensional semantic map through the quadruples of each entity; Based on the to-be-decorated task, plan the paths of the robot to each door of this task through the SLAM path planning algorithm; Drive the robot to move to each door, and repeat step S1 after moving to each door until the three-dimensional semantic map of the entire task is obtained; S2. Obtain the target object parameters and working modes of the room where the robot is located, specifically including the following steps: S21. The robot displays the three-dimensional semantic map of the task obtained in step S1 on the control terminal, and the user clicks on each entity in the room where the robot is located in turn to determine each target object; S22. The robot scans each target object, obtains the parameters of each target object, forms an array of parameters of each target object, and stores it in the cache; the arrays of parameters of each target object are stored in the cache in the form of a one-dimensional array, a[n] = {a1, a2, …… a n}, where n is the number of target objects obtained in step S21, and a n is the parameter array of the nth target object; S23. Determine the working mode of the robot, and the working mode is spraying or grinding; S3. Deduce the points based on the target object parameters, specifically including the following steps: S31. Obtain the parameter array of the target object a1 from the cache and the working mode obtained in step S23; S32. Obtain the initial working pose and the end working pose in the three-dimensional coordinate system; S33. Calculate the second initial working pose and the second end working pose in the robotic arm coordinate system through the inverse kinematics algorithm; S34. Using the rotatable angle of the robotic arm set by the user as the state set and {-a, a, 0} as the action set, generate the key point coordinates from the second initial working pose to the second end working pose according to the preset reward and punishment mechanism using the multi-agent reinforcement learning algorithm; Among them, when the working mode is spraying, a is the preset minimum spraying range of the robot; when the working mode is grinding, a is the preset minimum movement step of the robotic arm; S4. Perform single-room point planning to generate the point planning data for this room; S5. Perform point planning for all rooms under the to-be-decorated task according to the generated single-room planned points.

2. The method according to claim 1, wherein The parameter array includes type, height, width, L(x1, y1, z1), R(x2, y2, z2); The L(x1, y1, z1) is the coordinate of the left-bottommost point of the target object in the three-dimensional coordinate system and is the initial working point; The R(x2, y2, z2) is the coordinate of the right-topmost point of the target object in the three-dimensional coordinate system and is the end working point; Among them, x1, y1, z1, x2, y2, z2 are the values of the left-bottommost point and the right-topmost point of the target object on the x-axis, y-axis, and z-axis in the three-dimensional coordinate system respectively; Further, each target object parameter array is stored in the cache in the form of a one-dimensional array, a[n] = {a1, a2, …… a n}, where n is the number of target objects obtained in step S21, a n = {type, height, width, L(x1, y1, z1), R(x2, y2, z2)}; The types are wall surface, wall corner, door, window, column, ceiling board, and crossbeam, which are the same as those in step S1.

3. The method according to claim 2, wherein The reward and punishment mechanism is as follows: If the robotic arm reaches the second end pose and no collision occurs, the reward is set to N1, marked as the task completed, and the round ends. Before the robotic arm reaches the second end pose, there is a penalty of N2 for each simulation movement. Before the robotic arm reaches the second end pose, if the Euclidean distance between the pose in the joint coordinate system and the second end pose becomes smaller after a simulation movement, a reward of N3 is obtained. If the robotic arm collides with an obstacle or itself during the movement, a penalty of -N1 is obtained, marked as the task completed, and the round ends. N1 = 1, N2 = -0.01, N3 = 0.

02.

4. The method according to claim 3, characterized in that The single-room point position planning specifically includes the following steps: S41. Sequentially obtain a2, …… a from the cache n , and generate the position points of each target object in the same way as in step S3; S42. Obtain the planned point positions between each target object according to the SLAM algorithm. S44. Merge the point positions of each target object with the planned point positions between the target objects to obtain the planned point positions of the room and store them in the cache.

5. The method according to claim 4, wherein The point position planning for all rooms under the to-be-decorated task specifically includes the following steps: S51. Click on the physical object door that can enter the next room on the control terminal to drive the robot to the position of the door. S52. Repeat steps S2 - S4 to obtain the planned point positions of the next room and store them in the cache. S53. Repeat steps S51 - S52 until the point position planning for all rooms is completed. S54. Obtain the coordinates of all physical object doors that the user clicks on and can enter the next room, retrieve all the room planned point positions from the cache, and merge the obtained door coordinates with all the room planned point positions to obtain the planned point positions of all rooms under the to-be-decorated task.

6. The method according to claim 5, characterized in that: The initial working pose and the end working pose in the three-dimensional coordinate system are respectively represented as (x1, y1, z1, α1, β1, γ1) and (x2, y2, z2, α2, β2, γ2). α1, β1, γ1 are respectively the preset maximum pose angles of the three dimensions of the robotic arm at the initial working point, and α2, β2, γ2 are respectively the preset maximum pose angles of the three dimensions of the robotic arm at the end working point.

7. An indoor decoration robot site planning device based on a three-dimensional semantic map, comprising: Semantic map generation module, parameter acquisition module, point position derivation module, single-room planning module, multi-room planning module; The semantic map generation module is configured to generate a three-dimensional semantic map of the to-be-decorated task, which is specifically implemented in the following way: Collect indoor environment data and extract geographical images. Perform feature extraction and classification on the collected geographical images to identify different types of entities. Based on the types of each entity, determine the pixel values of each entity. Extract the outlines of each entity object through the geographical image, perform semantic fusion on the pixel values of each entity with the corresponding outlines to obtain the quadruple SCR<D, V, N, R> of each entity, where D represents the entity object type, V represents the set of three-dimensional coordinate information of the outer edge of the outline, N represents the set of three-dimensional coordinate information of each pixel point in the outline layer, and R represents the pixel value. Generate a three-dimensional semantic map through the quadruples of each entity. Based on the decoration task to be completed, plan the paths of the robot to each door of this task through the SLAM path planning algorithm; Drive the robot to move to each door, and repeat step S1 after moving to each door until a three-dimensional semantic map of the entire task is obtained; A parameter acquisition module, configured to acquire the target object parameters and working modes of the room where the robot is located, and is specifically implemented in the following ways: Acquire the target object parameters and working modes of the room where the robot is located, specifically including the following steps: S21. The robot displays the three-dimensional semantic map of the task obtained in step S1 on the control terminal, and the user clicks on each entity in the room where the robot is located in sequence to determine each target object; S22. The robot scans each target object, obtains the parameters of each target object, forms an array of parameters of each target object, and stores it in the cache; the arrays of parameters of each target object are stored in the cache in the form of a one-dimensional array, a[n] = {a1, a2,... a n}, where n is the number of target objects obtained in step S21, and a n is the parameter array of the nth target object; S23. Determine the working mode of the robot, and the working mode is spraying or grinding; A point position derivation module, configured to derive point positions according to the target object parameters, and is specifically implemented in the following ways: S31. Obtain the parameter array of target object a1 from the cache and the working mode obtained in step S23; S32. Obtain the initial working pose and the end working pose in the three-dimensional coordinate system; S33. Calculate the second initial working pose and the second end working pose in the robotic arm coordinate system through the inverse kinematics algorithm; S34. With the rotatable angle of the robotic arm set by the user as the state set and {-a, a, 0} as the action set, according to the preset reward and punishment mechanism, use the multi-agent reinforcement learning algorithm to generate the key point coordinates from the second initial working pose to the second end working pose; Among them, when the working mode is spraying, a is the preset minimum spraying range of the robot; when the working mode is grinding, a is the preset minimum movement step of the robotic arm; A single-room planning module, configured to perform single-room point position planning and generate the point position planning data for this room; A multi-room planning module, configured to perform point position planning for all rooms under the decoration task to be completed according to the generated single-room planned point positions.

8. An electronic device, including a memory and a processor, where the memory is used to store a computer program, and the processor runs the computer program to enable the computer device to execute the method for planning the station of an indoor decoration robot based on a three-dimensional semantic map according to any one of claims 1 to 6.

9. A computer-readable storage medium, in which a computer program is stored, and when the computer program is read and run by a processor, it executes the method for planning the station of an indoor decoration robot based on a three-dimensional semantic map according to any one of claims 1 to 6.

Images