Information reconstruction method, device and robot

Abstract

The application is suitable for the field of robot technology, and provides an information reconstruction method, device and robot, which comprises the following steps: acquiring an image; performing re-projection on the image according to a preset three-dimensional re-projection relationship to obtain a re-projected image; determining a pattern of a sub-region according to the re-projected image, the sub-region being a region where the robot is located when the image is acquired; determining a pattern of a global region according to the pattern of the sub-region involved in the movement of the robot; acquiring an actual pattern, the actual pattern being a three-dimensional pattern corresponding to a region actually passed through by the robot in the movement; and determining reconstructed information according to the pattern of the global region and the actual pattern. Through the above method, the user can obtain more information.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of robots, and particularly relates to an information reconstruction method and device, a robot, and a computer readable storage medium. BACKGROUND

[0002] At present, robots (such as sweeping robots) as a kind of high intelligent machine have gradually entered people's daily life. The robots often contain multiple sensors, such as collision sensors, gyroscopes, radars, inertial measurement units (IMU) wheel odometry and cameras, etc. These sensors are usually responsible for a single environmental perception. The sweeping robot can realize complex functions by fusing the data of these sensors, such as realizing cleaning, obstacle avoidance and positioning functions.

[0003] The most important thing in the robot is positioning and mapping (SLAM). The SLAM problem can be described as follows: whether there is a way to let the robot gradually depict a map completely consistent with the environment while moving, such as constructing a corresponding three-dimensional map through three-dimensional reconstruction, by putting a robot into an unknown environment at an unknown position.

[0004] At present, in the three-dimensional reconstruction of the existing robot (such as a sweeping robot), only the entire three-dimensional information of a house can be obtained, or the three-dimensional information of furniture in the house can be obtained in combination with recognition. SUMMARY

[0005] The information reconstruction method, device and robot provided in the embodiments of the present application can solve the problem that the user learns less information from the three-dimensional reconstruction of the robot.

[0006] In a first aspect, the embodiments of the present application provide an information reconstruction method applied to a robot, comprising:

[0007] obtaining an image;

[0008] reprojecting the image according to a preset three-dimensional reprojecting relationship to obtain a reprojected image;

[0009] determining a pattern of a sub-region according to the reprojected image, the sub-region being a region where the robot is located when the image is obtained;

[0010] determining a pattern of a global region according to the pattern of the sub-region involved in the moving process of the robot;

[0011] obtaining an actual pattern, the actual pattern being a three-dimensional pattern corresponding to a region actually passed through by the robot in the moving process;

[0012] determine the reconstructed information according to the global area pattern and the actual pattern.

[0013] In a second aspect, an information reconstruction apparatus is provided, which is applied to a robot and includes:

[0014] an image acquisition module, configured to acquire an image;

[0015] a post-reprojection image determination module, configured to perform re-projection on the image according to a preset three-dimensional re-projection relationship to obtain a post-reprojection image;

[0016] a sub-area pattern determination module, configured to determine a sub-area pattern according to the post-reprojection image, the sub-area being an area where the robot is located when the image is acquired;

[0017] a global area pattern determination module, configured to determine a global area pattern according to the sub-area pattern involved in the movement of the robot;

[0018] an actual pattern acquisition module, configured to acquire an actual pattern, the actual pattern being a three-dimensional pattern corresponding to an area actually passed through by the robot in the movement;

[0019] a reconstructed information determination module, configured to determine reconstructed information according to the global area pattern and the actual pattern.

[0020] In a third aspect, a robot is provided, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the method of the first aspect when executing the computer program.

[0021] In a fourth aspect, a computer readable storage medium is provided, which stores a computer program, and the computer program is executable on a processor to implement the method of the first aspect.

[0022] In a fifth aspect, a computer program product is provided, which, when executed on a robot, enables the robot to perform the method of the first aspect.

[0023] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0024] Since the image after re-projection is obtained by re-projecting the acquired image according to the preset three-dimensional re-projection relationship, the view angle corresponding to the image after re-projection is more in line with the view angle requirement of the user, so that the view angle corresponding to the pattern of the sub-region and the pattern of the global region determined according to the image after re-projection is more in line with the view angle requirement of the user, and then the reconstructed information determined by relying on the pattern of the global region and the actual pattern is more accurate. At the same time, since the actual pattern is a three-dimensional pattern corresponding to the region actually passed by the robot during movement, the reconstructed information can include pattern information and three-dimensional information at the same time, and in addition, since the pattern of the global region includes the pattern of the obstacle and the actual pattern does not include the pattern of the obstacle, the reconstructed information can also be more accurate. BRIEF DESCRIPTION OF DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows.

[0026] Figure 1 is a flowchart of an information reconstruction method provided by an embodiment of the present application;

[0027] Figure 2 is a schematic diagram of an image acquired by a robot provided by an embodiment of the present application;

[0028] Figure 3 is a schematic diagram of a re-projected image provided by an embodiment of the present application Figure 2

[0029] is a schematic diagram of an image obtained by cropping the re-projected image provided by an embodiment of the present application; Figure 4 Figure 3 is a structural schematic diagram of an information reconstruction device provided by another embodiment of the present application;

[0030] Figure 5 is a structural schematic diagram of a robot provided by an embodiment of the present application.

[0031] DETAILED DESCRIPTION Figure 6 In the following description, specific details such as specific system structures, techniques, etc. are presented in order to thoroughly understand the embodiments of the present application. However, it should be clear to those skilled in the art that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits and methods are omitted to avoid unnecessary details that hinder the description of the present application.

[0032] In the following description, specific details such as specific system structures, techniques, etc. are presented in order to thoroughly understand the embodiments of the present application. However, it should be clear to those skilled in the art that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits and methods are omitted to avoid unnecessary details that hinder the description of the present application.

[0033] ​It should be understood that the word "comprising" when used in the specification and claims of this application indicates the existence of the stated features, integers, steps, operations, elements, and / or components but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0034] It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items, and that the term "at least one of' denotes one, or a plurality of, of the listed items.

[0035] Reference throughout this application to "one embodiment" or "some embodiments" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the application. Thus, the appearances of the phrases "in one embodiment" or "in some embodiments" in various places throughout this specification are not necessarily all referring to the same embodiment, unless otherwise specified.

[0036] Example One

[0037] Currently, in the three-dimensional reconstruction of existing robots (such as sweeping robots), only global three-dimensional information of a house can be obtained, but corresponding color information cannot be obtained, for example, three-dimensional information and color information of the ground cannot be obtained at the same time, so that the amount of information that a user can obtain is less.

[0038] In order to enable a user to obtain a larger amount of information, an information reconstruction method is provided in the embodiments of the application, in which, after a projection operation is performed on an obtained image, a reprojected image is obtained, then a pattern of a sub-region is determined according to the reprojected image, a pattern of a global region is determined according to the patterns of the sub-regions involved in the movement of a robot, finally, an actual pattern is obtained, and reconstructed information is determined according to the pattern of the global region and the actual pattern.

[0039] Since the pattern of the global region is determined according to the patterns of all the sub-regions, and the pattern of each sub-region is determined according to the reprojected image including color information of the image, the reconstructed information includes the color information of the image. At the same time, since the actual pattern is a three-dimensional pattern corresponding to a region actually passed through by the robot in the movement process, the reconstructed information also includes three-dimensional information corresponding to the color information, so that a user can obtain color information and three-dimensional information at the same time according to the reconstructed information, and since the pattern of the global region contains the pattern of an obstacle and the actual pattern does not contain the pattern of the obstacle, the reconstructed information is more accurate.

[0040] The information reconstruction method provided by the embodiment of the present application is described below with reference to the accompanying drawings.

[0041] Figure 1 A flowchart of an information reconstruction method provided by the embodiment of the present application is shown, which is applied to a robot, and is described in detail as follows.

[0042] In step S11, an image is acquired.

[0043] In this embodiment, the robot acquires an image obtained by a camera performing a shooting action, and the camera can perform the shooting action at a certain frequency.

[0044] In some embodiments, the camera described above can be a camera built in the robot or a camera externally connected to the robot.

[0045] In some embodiments, the image shot by the camera described above is a two-dimensional image with red (R), green (G) and blue (B) color information.

[0046] In step S12, the image is re-projected according to a preset three-dimensional re-projection relationship to obtain a re-projected image.

[0047] In this embodiment, the three-dimensional re-projection relationship can be a corresponding relationship between a real world coordinate (which is a three-dimensional coordinate) and a pixel of the image shot by the camera.

[0048] In step S13, a pattern of a sub-region is determined according to the re-projected image, and the sub-region is a region where the robot is located when the image is acquired.

[0049] In this embodiment, color information included in the re-projected image is identified to obtain a pattern corresponding to the color information, and the obtained pattern is the pattern of the region where the robot is located when the robot acquires the image shot by the camera.

[0050] In some embodiments, adjacent two sub-regions have an overlapping region to ensure that a continuous pattern can be obtained.

[0051] In step S14, a pattern of a global region is determined according to the patterns of the sub-regions involved in the movement of the robot.

[0052] In this embodiment, the global region is a region obtained by merging at least two sub-regions.

[0053] In some embodiments, the moving process described above can be a process of one movement of the robot. For example, assuming that the starting time of one movement of the robot is 10:00 and the ending time is 11:00, the global region described above can be obtained by merging the sub-regions involved in the movement of the robot between 10:00 and 11:00.

[0054] In some embodiments, the robot can obtain the patterns of different sub-regions during the movement. However, if the sub-regions corresponding to the patterns at different times overlap, image matching is needed. The image matching generally includes three steps of feature point extraction, feature point matching and image fusion. Specifically, the scale-invariant feature transform (SIFT) features of the image are extracted, the extracted SIFT features are matched, such as using the Fast_Library_for_Approximate_Nearest_Neighbors (FLANN) matching method, and the images after feature point matching are fused, such as using the weighted fusion method.

[0055] It should be noted that, since the judgment of whether the pixels on the image really correspond to the region in the specified direction (such as whether they really correspond to the region in the downward direction, i.e. the region where the floor is located) is not performed during the re-projection, the pattern of the global region obtained contains part of the pattern that is not the floor, such as the pattern of the obstacle on the floor, so the pattern of the global region described above is a rough global floor pattern.

[0056] Step S15, obtaining an actual pattern, the actual pattern being a three-dimensional pattern corresponding to the region actually passed through by the robot during the movement.

[0057] Since the robot performs obstacle avoidance during the movement, the region actually passed through by the robot can be different from the region where the robot is located when the image is obtained.

[0058] In this embodiment, the three-dimensional pattern corresponding to the region actually passed through by the robot during the movement can be obtained according to the SLAM algorithm.

[0059] Step S16, determining the reconstructed information according to the pattern of the global region and the actual pattern.

[0060] In this embodiment, since the pattern of the global region can contain the pattern of the obstacle and the actual pattern does not contain the pattern of the obstacle, more accurate reconstructed information can be determined according to the pattern of the global region and the actual pattern.

[0061] In the embodiment of the present application, since the re-projected image is obtained by re-projecting the acquired image according to the preset three-dimensional re-projection relationship, the view angle corresponding to the re-projected image is more in line with the view angle requirement of the user, so that the view angle corresponding to the pattern of the sub-region and the pattern of the global region determined according to the re-projected image is more in line with the view angle requirement of the user, and then the reconstructed information determined by relying on the pattern of the global region and the actual pattern is more accurate. At the same time, since the actual pattern is a three-dimensional pattern corresponding to the region actually passed by the robot during movement, the reconstructed information can include pattern information and three-dimensional information at the same time, and in addition, since the pattern of the global region includes the pattern of the obstacle and the actual pattern does not include the pattern of the obstacle, the reconstructed information can also be more accurate.

[0062] In some embodiments, the re-projected image includes color information of the image before re-projection.

[0063] In some embodiments, the step S11 includes:

[0064] The image is acquired by a camera installed on the robot.

[0065] Correspondingly, before the step S12, it includes:

[0066] The preset three-dimensional re-projection relationship is determined according to the information of the camera, the preset geometric transformation information, and the pixel information of the image.

[0067] Correspondingly, the step S12 includes:

[0068] The coordinates of the corresponding pixels are determined according to the preset three-dimensional re-projection relationship and the preset real-world coordinates, and when the coordinates of the pixels corresponding to the real-world coordinates are the same as the coordinates of the pixels of the image, the color information corresponding to the real-world coordinates is set to be the same as the color information corresponding to the coordinates of the pixels of the image, so as to obtain the re-projected image.

[0069] In the embodiment, the information of the camera includes the focal length of the camera, the preset geometric transformation information includes a rotation matrix and a translation vector, the pixel information includes the physical size of the pixels in the x-axis and y-axis directions, and the coordinates of the origin of the image coordinate system.

[0070] In some embodiments, the preset three-dimensional re-projection relationship can be in the following form:

[0071]

[0072] wherein R is a rotation matrix (e.g., a 3x3 rotation matrix), t is a translation vector (e.g., a 3x1 translation vector), t is 0 when the coordinate system of the three-dimensional real world and the image camera have the same origin, f is the focal length of the camera, dx and dy are the physical sizes of the pixels in the x-axis and y-axis directions, respectively, and (u0, v0) is the coordinate of the origin of the image coordinate system. Each parameter in the above preset three-dimensional re-projection relationship can be calculated by a camera calibration algorithm, such as Zhang Zhengyou calibration method.

[0073] In this embodiment, after determining the preset three-dimensional re-projection relationship, the re-projected image can be determined according to the following formula:

[0074]

[0075] wherein (x w , y w , z w ) is the real world coordinate, (u, v) is the coordinate of the pixel on the image, and z c is a normalization coefficient for the last item of (u, v, 1) to be 1. According to the above formula, as long as the real world coordinate is given, the pixel coordinate and z c can be obtained. Considering that the height of the camera mounted on the robot will not change, y w can be set as a fixed value, for example, y w = h. At this time, as long as the values of (x w , z w ) are changed, the (u, v) values corresponding to any (x w , y w , z w ) and the color information (such as color value) at the (u, v) position can be obtained. In some embodiments, considering that the camera is mounted on the robot, the value range of x w can be set as the range corresponding to the width of the robot, and the value range of z w can be set as the clarity in the depth direction related to the image obtained by the camera, that is, the value range of z w is determined according to the clarity in the depth direction. Since the corresponding (u, v) values are calculated according to the value ranges of x w and z w , that is, the (u, v) values corresponding to any (x w , y w , z w ) are not calculated, unnecessary calculation amount can be reduced.

[0076] (x w , z w) as the coordinate value of the re-projected image, the color information on the coordinate value is the color information of the corresponding (u, v) position.

[0077] Taking the image of the floor photographed by the camera as an example, the original image (the image obtained by the robot) and the re-projected image are respectively as shown in Figure 2 and Figure 3 According to Figure 2 and Figure 3 It can be known that the viewing angle of the re-projected image is more consistent with the viewing angle of the user.

[0078] It should be noted that the embodiment of the present application determines the corresponding pixel coordinates by changing the real world coordinates, because the three-dimensional coordinates (real world coordinates) corresponding to the pixels in the image are non-uniform, that is, there are more pixels close to the camera and fewer pixels far from the camera, therefore, if the corresponding real world coordinates are determined by changing the coordinates of the pixels in the image, the re-projected image obtained will have many gaps, and the method of the embodiment of the present application will not have the above problem.

[0079] In some embodiments, the above step S13 comprises:

[0080] A1, determining the pose of the robot at the current position.

[0081] In the embodiment, since the robot itself needs to solve the SLAM problem, the robot is usually configured with sensors that can perceive the different positions of the robot on the map, such as odometer, laser radar, etc., and the pose of the robot at the current position can be obtained by integrating the data of these sensors, for example, the offset information of the robot is obtained according to the position of the laser radar in the map, and then the rotation direction of the robot is obtained by the odometer, and finally the pose of the robot at the current position is determined according to the offset information and the rotation direction. That is, since the pose of the robot at the current position can be determined according to the SLAM algorithm in the embodiment, it is not necessary to add a new algorithm to the robot to calculate the pose of the robot, thereby reducing the consumption of resources of the robot.

[0082] A2, determining the pattern of the sub-region according to the re-projected image and the pose of the robot at the current position.

[0083] Since each pixel point in the re-projected image corresponds to a point on the real world region (such as the floor), the corresponding position of each pixel point in the world coordinate system in the re-projected image can be determined according to the pose of the robot at the current position, thereby constructing the pattern of the sub-region of the robot at the current position. Specifically, according to the above formula (1), each coordinate in the sub-region pattern actually has its corresponding real world coordinate (three-dimensional coordinate), and each three-dimensional coordinate (x, y, z) corresponds to a pixel in the re-projected image.w ,y w ,z w ) also correspond to each (u, v) on the image. Therefore, when determining the pattern of the sub-region: 1) first convert the three-dimensional coordinates (x w ,y w ,z w ) to the coordinate system of the robot used: for example, assuming that the robot is a sweeper, convert (x w ,y w ,z w ) to the coordinate system of the sweeper by the following formula: (x' w = x w + x shift , y' w , z' w = z w + z shift ), where (x shift , z shift ) represents the offset of the coordinate origin of the coordinate system of the sweeper from the coordinate origin of the real world coordinate system; 2) use the pose (i.e. pose data) to translate and rotate the converted three-dimensional coordinates, the calculation formula is as follows:

[0084] x" w = cosθ·x' w - sinθ·z' w + T x

[0085] z" w = sinθ·x' w - cosθ·z' w + T x

[0086] where (T x , T z ) and θ are the translation value and rotation value of the current position relative to the origin of the sweeper contained in the pose data. At this time, the color value of (x" w , z" w ) of the sub-region corresponds to the color value of the position (u, v) on the image.

[0087] In A1 and A2 above, since the pattern of the sub-region is determined in combination with the position of the robot at the current position, i.e. in combination with the offset information and the rotation direction of the robot, the determined pattern is more accurate.

[0088] In some embodiments, considering that the camera has different resolutions, and when the resolutions are different, the clarity of the images obtained by the camera is also different, therefore, in order to be able to obtain a pattern that meets the requirement of clarity, the following is included before step A2 above:

[0089] Crop the area in the re-projected image that does not meet the requirement of definition to obtain a cropped image.

[0090] wherein, Figure 4 An image obtained after cropping the image is shown. Figure 3

[0091] Correspondingly, the step A2 specifically includes:

[0092] According to the cropped image and the pose of the robot at the current position, determine the pattern of the sub-region.

[0093] In this embodiment, since the area in the image that does not meet the requirement of definition is cropped, the definition is higher when the pattern of the sub-region is determined according to the cropped image subsequently, thereby improving the good experience of the user.

[0094] In some embodiments, the area that does not meet the requirement of definition can be determined according to any of the following ways:

[0095] (1) Perform definition detection on the re-projected image, and then determine the cropped area according to the result of the definition detection. The definition detection here includes one or more of the following: gradient detection, edge detection, variance calculation, etc.

[0096] (2) According to the resolution of the camera, determine the area to be cropped in combination with an empirical value to speed up the process of obtaining the cropped image.

[0097] In some embodiments, the step S16 includes:

[0098] Perform an AND operation on the pattern of the global region and the actual pattern to obtain the reconstructed information.

[0099] In this embodiment, since the pattern of the global region may include information of the obstacle, and the actual pattern does not include information of the obstacle, after performing the AND operation on the pattern of the global region and the actual pattern, it can be ensured that the reconstructed information no longer contains information of the obstacle, thereby improving the accuracy of the obtained reconstructed information.

[0100] In some embodiments, when the lens of the camera is directed at the floor, the pattern is the pattern on the floor, and at this time, after the step S16, the method can further include:

[0101] B1, identifying the material of the floor according to the pattern to obtain the material of the floor.

[0102] B2, controlling the robot to perform a corresponding action according to the material of the floor.

[0103] ​In this embodiment, the material of the floor can be identified according to the thickness of the lines of the pattern on the floor, the concave-convex feeling, whether there are hairs, etc. For example, when the floor is a wooden board, the corresponding pattern will have the grain of wood.

[0104] In the above B1 and B2, since the robot performs the corresponding action in combination with the material control of the floor, the accuracy of the robot performing the action can be improved. For example, assuming that the robot moves by track, when the robot identifies that the material of the floor is a blanket, the robot will control the robot to leave the floor where the blanket is, so that the hair products of the blanket will not be rolled into the track of the robot, and further cause the robot to malfunction.

[0105] In some embodiments, the above-mentioned robot is a sweeping robot, and the above-mentioned step B2 comprises:

[0106] B21, selecting a corresponding cleaning strategy according to the material of the above-mentioned floor.

[0107] B22, controlling the above-mentioned robot to perform the corresponding cleaning action according to the selected above-mentioned cleaning strategy.

[0108] In this embodiment, it is considered that the corresponding cleaning strategy of the floor of different materials is usually different: for example, wooden floor cannot be cleaned with a large amount of water, for example, ceramic tile floor does not have too much requirement for the amount of water for cleaning the floor, for example, the floor covered with a blanket cannot be directly cleaned by mopping, etc. Therefore, the above-mentioned B21 and B22 select the corresponding cleaning strategy according to the material of the floor, and further can more accurately control the robot to perform the corresponding cleaning action.

[0109] It should be understood that the size of the serial number of each step in the above-mentioned embodiment does not mean the order of execution, the execution order of each process should be determined according to its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0110] Example Two

[0111] According to the information reconstruction method described in the above embodiment, Figure 5 The structural block diagram of the information reconstruction device provided by the embodiments of the present application is shown, only the part related to the embodiments of the present application is shown for the convenience of description.

[0112] Referring to Figure 5 The information reconstruction device 5 comprises: an image acquisition module 51, a re-projected image determination module 52, a sub-region pattern determination module 53, a global region pattern determination module 54, an actual pattern acquisition module 55, a reconstructed information determination module 56, wherein:

[0113] The image acquisition module 51 is used for acquiring an image.

[0114] The image after re-projection determining module 52 is configured to re-project the image according to a preset three-dimensional re-projection relationship to obtain an image after re-projection.

[0115] In some embodiments, the image after re-projection includes color information of the image before re-projection.

[0116] The pattern of sub-region determining module 53 is configured to determine a pattern of a sub-region according to the image after re-projection, the sub-region being a region where the robot is located when the image is obtained.

[0117] In this embodiment, the color information included in the image after re-projection is identified to obtain a pattern corresponding to the color information, and the obtained pattern is the pattern of the region where the robot is located when the robot obtains the image captured by the camera.

[0118] In some embodiments, the two adjacent sub-regions have an overlapping region to ensure that a continuous pattern can be obtained.

[0119] The pattern of global region determining module 54 is configured to determine a pattern of a global region according to the patterns of the sub-regions involved in the movement of the robot.

[0120] In this embodiment, the global region is obtained by merging at least two sub-regions.

[0121] In some embodiments, the movement process can be a process belonging to the same movement of the robot. For example, assuming that the starting time of a movement of the robot is 10:00 and the ending time is 11:00, the global region is obtained by merging the sub-regions involved in the movement of the robot between 10:00 and 11:00.

[0122] In some embodiments, the robot obtains the patterns of different sub-regions during the movement. However, if the sub-regions corresponding to the patterns at different times overlap, image matching is required.

[0123] The actual pattern obtaining module 55 is configured to obtain an actual pattern, the actual pattern being a three-dimensional pattern corresponding to a region actually passed through by the robot during the movement.

[0124] In this embodiment, the three-dimensional pattern corresponding to the region actually passed through by the robot during the continuous movement can be obtained according to the SLAM algorithm.

[0125] The reconstructed information determining module 56 is configured to determine reconstructed information according to the pattern of the global region and the actual pattern.

[0126] In the embodiment of the present application, since the reprojected image is obtained by reprojecting the acquired image according to the preset three-dimensional reprojecting relationship, the view angle corresponding to the reprojected image is more in line with the view angle requirement of the user, so that the view angle corresponding to the pattern of the sub-region and the pattern of the global region determined according to the reprojected image is more in line with the view angle requirement of the user, and then the reconstructed information determined by relying on the pattern of the global region and the actual pattern is more accurate. At the same time, since the actual pattern is a three-dimensional pattern corresponding to the region actually passed by the robot during movement, the reconstructed information can include pattern information and three-dimensional information at the same time, and in addition, since the pattern of the global region includes the pattern of the obstacle and the actual pattern does not include the pattern of the obstacle, the reconstructed information can also be more accurate.

[0127] In some embodiments, the image acquisition module 51 is specifically configured to:

[0128] The image is acquired by a camera installed on the robot.

[0129] Correspondingly, the information reconstruction device 5 further includes:

[0130] A preset three-dimensional reprojecting relationship determination module is configured to determine the preset three-dimensional reprojecting relationship according to the information of the camera, preset geometric transformation information, and pixel information of the image.

[0131] Correspondingly, the reprojected image determination module 52 is specifically configured to:

[0132] According to the preset three-dimensional reprojecting relationship and preset real-world coordinates, the coordinates of the corresponding pixels are determined, and when the coordinates of the pixels corresponding to the real-world coordinates are the same as the coordinates of the pixels of the image, the color information corresponding to the real-world coordinates is set to be the same as the color information corresponding to the coordinates of the pixels of the image, so as to obtain the reprojected image.

[0133] In the embodiment, the information of the camera includes the focal length of the camera, the preset geometric transformation information includes a rotation matrix and a translation vector, the pixel information includes the physical size of the pixels in the x-axis and y-axis directions, and the coordinates of the origin of the image coordinate system.

[0134] In some embodiments, the sub-region pattern determination module 53 includes:

[0135] A pose determination unit is configured to determine the pose of the robot at the current position.

[0136] A sub-region pattern determination unit is configured to determine the pattern of the sub-region according to the reprojected image and the pose of the robot at the current position.

[0137] In some embodiments, the pattern determination unit of the sub-region is specifically configured to:

[0138] convert the three-dimensional coordinates corresponding to the coordinates of each pixel in the re-projected image into coordinates in the coordinate system of the robot to obtain converted three-dimensional coordinates;

[0139] perform corresponding operations on the converted three-dimensional coordinates according to the pose of the robot at the current position to obtain the pattern of the sub-region.

[0140] In some embodiments, the information reconstruction device 5 further comprises:

[0141] an image cropping module configured to crop a region in the re-projected image that does not meet the requirement of clarity to obtain a cropped image.

[0142] Correspondingly, the pattern determination unit of the sub-region is specifically configured to:

[0143] determine the pattern of the sub-region according to the cropped image and the pose of the robot at the current position.

[0144] In some embodiments, the region that does not meet the requirement of clarity can be determined in any of the following ways:

[0145] (1) performing clarity detection on the re-projected image, and then determining the cropped region according to the result of the clarity detection. Here, the clarity detection includes one or more of the following: gradient detection, edge detection, variance calculation, etc.

[0146] (2) determining the region to be cropped according to the resolution of the camera combined with an empirical value to speed up the obtaining of the cropped image.

[0147] In some embodiments, the reconstructed information determination module 56 is specifically configured to:

[0148] perform an AND operation on the pattern of the global region and the actual pattern to obtain the reconstructed information.

[0149] In some embodiments, the pattern is a pattern on a floor, and the information reconstruction device 5 further comprises:

[0150] a material identification module configured to identify the material of the floor according to the pattern to obtain the material of the floor.

[0151] an action control module configured to control the robot to perform a corresponding action according to the material of the floor.

[0152] In some embodiments, the robot described above is a sweeping robot, and the motion control module is specifically used for:

[0153] Select the corresponding cleaning strategy based on the material of the floor; control the robot to perform the corresponding cleaning action based on the selected cleaning strategy.

[0154] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0155] Example Three

[0156] Figure 6 This is a schematic diagram of the structure of a robot provided in one embodiment of this application. Figure 6 As shown, the robot 6 in this embodiment includes: at least one processor 60 ( Figure 6 The diagram shows only one processor, a memory 61, and a computer program 62 stored in the memory 61 and executable on the at least one processor 60, which, when executing the computer program 62, implements the steps in any of the above method embodiments.

[0157] The robot 6 may be a robotic vacuum cleaner or other types of robot. This robot may include, but is not limited to, a processor 60 and a memory 61. Those skilled in the art will understand that... Figure 6 This is merely an example of robot 6 and does not constitute a limitation on robot 6. It may include more or fewer parts than shown in the figure, or combine certain parts, or different parts, such as input / output devices, network access devices, etc.

[0158] The processor 60 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0159] The memory 61 can be an internal storage unit of the robot 6, such as a hard disk or a memory of the robot 6 in some embodiments. The memory 61 can also be an external storage device of the robot 6, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the robot 6 in some other embodiments. Further, the memory 61 can include both the internal storage unit and the external storage device of the robot 6. The memory 61 is used to store an operating system, an application program, a boot loader, data, and other programs, such as program codes of the computer program, etc. The memory 61 can also be used to temporarily store data that has been output or is to be output.

[0160] It can be clearly understood by those skilled in the art that, for the convenience and brevity of description, only the division of the above functional units and modules is taken as an example for illustration, and in actual application, the above functions can be completed by different functional units and modules according to needs, that is, the internal structure of the apparatus is divided into different functional units or modules to complete all or part of the above-described functions. Each functional unit and module in the embodiment can be integrated in one processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit, and the integrated unit can be realized in the form of hardware or in the form of a software functional unit. In addition, the specific names of the functional units and modules are only for the convenience of mutual distinction, and do not serve to limit the protection scope of the present application. The specific working process of the units and modules in the system can refer to the corresponding process in the foregoing method embodiments, which will not be described here.

[0161] The embodiment of the present application further provides a network device, which comprises at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, and the processor implements the steps in any of the method embodiments described above when executing the computer program.

[0162] The embodiment of the present application further provides a computer readable storage medium, which stores a computer program, and the computer program is executable on a processor to implement the steps in any of the method embodiments described above.

[0163] The embodiment of the present application provides a computer program product, which, when executed on a robot, enables the robot to implement the steps in any of the method embodiments described above.

[0164] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the present application can implement all or part of the processes in the above-mentioned embodiment methods through a computer program to instruct relevant hardware to complete, and the computer program can be stored in a computer readable storage medium. When the computer program is executed by a processor, the steps of each method embodiment described above can be implemented. The computer program includes computer program code, which can be in the form of source code, object code, executable files or some intermediate forms. The computer readable medium can at least include any entity or device capable of carrying the computer program code to the photographing device / terminal equipment, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium. For example, U disk, mobile hard disk, magnetic disk or optical disk, etc. In some jurisdictions, according to legislation and patent practice, the computer readable medium can not be an electrical carrier signal and a telecommunication signal.

[0165] In the above embodiments, the description of each embodiment has its own focus, and the parts not described or recorded in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0166] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in connection with the embodiments disclosed herein can be realized by electronic hardware, or a combination of computer software and electronic hardware. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0167] In the embodiments provided by the present application, it should be understood that the disclosed apparatus / network device and method can be implemented in other ways. For example, the above-described apparatus / network device embodiments are merely schematic, for example, the division of the modules or units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the shown or discussed each other can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0168] The units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, may be located in one place, or may also be distributed to multiple network units. Part or all of the units can be selected to achieve the purpose of the embodiment scheme according to actual needs.

[0169] The above embodiments are only used to illustrate the technical solutions of the present application, but not limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can still be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A method of information reconstruction, characterized by, The application is applied to a robot, comprising: acquiring an image; reprojecting the image according to a preset three-dimensional reprojecting relationship to obtain a reprojected image, the three-dimensional reprojecting relationship being a corresponding relationship between a real world coordinate and a pixel of the image, the three-dimensional reprojecting relationship being determined by changing the real world coordinate to determine the corresponding pixel coordinate; the reprojecting the image according to the preset three-dimensional reprojecting relationship to obtain the reprojected image comprising: determining the corresponding pixel coordinate according to the preset three-dimensional reprojecting relationship and a preset real world coordinate, and setting the color information corresponding to the real world coordinate to be the same as the color information corresponding to the pixel coordinate of the image when the pixel coordinate corresponding to the real world coordinate is the same as the pixel coordinate of the image, thereby obtaining the reprojected image; determining a pattern of a sub-region according to the reprojected image, the sub-region being a region where the robot is located when the image is acquired; determining a pattern of a global region according to the pattern of the sub-region involved in the moving process of the robot; acquiring an actual pattern, the actual pattern being a three-dimensional pattern corresponding to a region actually passed through by the robot in the moving process; determining reconstructed information according to the pattern of the global region and the actual pattern.

2. The information reconstruction method of claim 1, wherein, The reprojected image comprises color information of the image before reprojecting.

3. The information reconstruction method of claim 2, wherein, The acquiring the image comprises: acquiring the image by a camera installed on the robot; before the reprojecting the image according to the preset three-dimensional reprojecting relationship to obtain the reprojected image, comprising: determining the preset three-dimensional reprojecting relationship according to information of the camera, preset geometric transformation information and pixel information of the image.

4. The information reconstruction method of claim 1, wherein, The determining the pattern of the sub-region according to the reprojected image comprises: determining a pose of the robot at a current position; determining the pattern of the sub-region according to the reprojected image and the pose of the robot at the current position.

5. The information reconstruction method of claim 4, wherein, The determining the pattern of the sub-region according to the reprojected image and the pose of the robot at the current position comprises: converting three-dimensional coordinates corresponding to each pixel coordinate in the reprojected image to coordinates in a coordinate system of the robot to obtain converted three-dimensional coordinates; performing corresponding operations on the converted three-dimensional coordinates according to the pose of the robot at the current position to obtain the pattern of the sub-region.

6. The information reconstruction method of claim 4, wherein, Before the determining the pattern of the sub-region according to the reprojected image and the pose of the robot at the current position, comprising: cropping a region in the reprojected image whose definition does not meet a requirement to obtain a cropped image; The determining the pattern of the sub-region according to the reprojected image and the pose of the robot at the current position comprises: determining the pattern of the sub-region according to the cropped image and the pose of the robot at the current position.

7. The information reconstruction method of claim 1, wherein, The determining the reconstructed information according to the pattern of the global region and the actual pattern comprises: The global area pattern and the actual pattern are subjected to a with operation, and a result of the with operation is the reconstructed information.

8. The information reconstruction method according to any one of claims 1 to 7, characterized by, The global area pattern includes a pattern on a floor, and after the reconstructed information is determined according to the global area pattern, the method further includes: material identification of the floor is performed according to the pattern on the floor, and a material of the floor is obtained; the robot is controlled to perform a corresponding action according to the material of the floor.

9. The information reconstruction method of claim 8, wherein, The robot is a sweeping robot, and the robot is controlled to perform a corresponding action according to the material of the floor, including: a corresponding cleaning strategy is selected according to the material of the floor; the robot is controlled to perform a corresponding cleaning action according to the selected cleaning strategy.

10. An information reconstructing apparatus characterized by comprising: The application and the robot include: an image acquisition module, configured to acquire an image; an image after re-projection determination module, configured to perform re-projection on the image according to a preset three-dimensional re-projection relationship to obtain an image after re-projection, the three-dimensional re-projection relationship being a corresponding relationship between a real world coordinate and a pixel of the image, and the three-dimensional re-projection relationship being used to determine a corresponding pixel coordinate by changing the real world coordinate; the image after re-projection determination module is specifically configured to: determine a corresponding pixel coordinate according to the preset three-dimensional re-projection relationship and a preset real world coordinate, and when the pixel coordinate corresponding to the real world coordinate is the same as the pixel coordinate of the image, set color information corresponding to the real world coordinate to be the same as color information corresponding to the pixel coordinate of the image, so as to obtain the image after re-projection; a pattern of a sub-area determination module, configured to determine a pattern of a sub-area according to the image after re-projection, the sub-area being an area where the robot is located when the image is acquired; a global area pattern determination module, configured to determine a global area pattern according to the pattern of the sub-area involved in a movement process of the robot; an actual pattern acquisition module, configured to acquire an actual pattern, the actual pattern being a three-dimensional pattern corresponding to an area actually passed through by the robot in the movement process; a reconstructed information determination module, configured to determine reconstructed information according to the global area pattern and the actual pattern.

11. A robot comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the method in any one of claims 1 to 9.

12. A computer-readable storage medium, the computer-readable storage medium storing a computer program, characterized in that, The computer program is executed by the processor to implement the method in any one of claims 1 to 9.

Images