Three-dimensional scene modeling method and device, electronic equipment and storage medium

By using multi-angle image acquisition equipment in three-dimensional scene modeling, the three-dimensional regional model is optimized, and the problems of high labor costs and incomplete acquisition in the existing technology are solved, and three-dimensional modeling with high accuracy and low cost are achieved.

CN118918267APending Publication Date: 2024-11-08BEIJING GALBOT AI CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411182117.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-27
Publication Date
2024-11-08

Smart Images

  • Figure CN118918267A_ABST
    Figure CN118918267A_ABST
Patent Text Reader

Abstract

The embodiment of the invention provides a three-dimensional scene modeling method and device, electronic equipment and a storage medium, and relates to the technical field of image processing. The method comprises the following steps: in a process of moving in a to-be-reconstructed area, performing multi-angle shooting on the to-be-reconstructed area through first image acquisition equipment of a robot to obtain a plurality of three-dimensional images; wherein the plurality of three-dimensional images are used for performing three-dimensional modeling on the to-be-reconstructed region to obtain a three-dimensional region model. And determining a target object mapped by a to-be-optimized object model included in the three-dimensional area model in the to-be-reconstructed area, and performing multi-angle shooting on the target object through a second image acquisition device of the robot to obtain a plurality of two-dimensional images. Wherein the shooting angle of the second image acquisition device on the target object is controlled by adjusting the pose of the robot, the plurality of two-dimensional images are used for optimizing the object model to be optimized, and an optimized three-dimensional area model is obtained. The human cost consumption of three-dimensional scene modeling is reduced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a three-dimensional scene modeling method, device, electronic device and storage medium. Background Art

[0002] 3D scene modeling refers to building a 3D digital model in a computer that is proportional to the real 3D scene. This model can help people or electronic devices more intuitively understand and perceive the structure and layout of the 3D scene, as well as the objects contained in the 3D scene, so as to facilitate the rationality analysis and optimization of the 3D scene design, as well as the unified management of the objects in the 3D scene.

[0003] At present, the conventional method of 3D scene modeling is to place multiple cameras at fixed positions in the 3D scene, so that each camera captures the 3D scene at a fixed position. Then, the 3D scene is modeled based on the images captured by the cameras. However, this method relies on manual placement of cameras, so the labor cost is high. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a 3D scene modeling method, device, electronic device and storage medium to reduce the labor cost of 3D scene modeling. The specific technical solution is as follows:

[0005] In a first aspect, an embodiment of the present application provides a three-dimensional scene modeling method, the method comprising:

[0006] In the process of moving in the area to be reconstructed, the area to be reconstructed is photographed from multiple angles by a first image acquisition device of the robot to obtain a plurality of three-dimensional images, and the plurality of three-dimensional images are used to perform three-dimensional modeling on the area to be reconstructed to obtain a three-dimensional area model of the area to be reconstructed;

[0007] Determine a target object mapped by the object model to be optimized included in the three-dimensional region model within the region to be reconstructed;

[0008] The target object is photographed at multiple angles by the second image acquisition device of the robot to obtain multiple two-dimensional images. The shooting angle of the target object by the second image acquisition device is controlled by adjusting the posture of the robot. The multiple two-dimensional images are used to optimize the model of the object to be optimized to obtain an optimized three-dimensional area model.

[0009] Optionally, during the movement in the area to be reconstructed, the area to be reconstructed is photographed from multiple angles by a first image acquisition device of the robot to obtain multiple three-dimensional images, including:

[0010] In the area to be reconstructed, the robot is controlled to move continuously with a preset distance as a moving step length;

[0011] When the object is included in the field of view of the first image acquisition device, pausing the movement;

[0012] Using the first image acquisition device to capture each area to be captured, respectively, to obtain a plurality of three-dimensional images, wherein the area to be captured is a sub-area included in the current field of view of the first image acquisition device;

[0013] Return to the step of controlling the robot to move continuously with a preset distance as a moving step length until the robot reaches a preset end point in the area to be reconstructed or the robot has traversed the area to be reconstructed.

[0014] Optionally, before returning to the step of controlling the robot to move continuously with a preset distance as a moving step, the method further includes:

[0015] Performing an adjustment operation on an item to be adjusted of the first image acquisition device, wherein the item to be adjusted includes: height and / or pitch angle, and the adjustment operation includes: an increase operation or a decrease operation;

[0016] Using the first image acquisition device to photograph each area to be photographed, to obtain a plurality of three-dimensional images;

[0017] In a case where the value of the item to be adjusted is not the first value, returning to the step of performing the adjustment operation on the item to be adjusted of the first image acquisition device;

[0018] When the value of the item to be adjusted is the first value, adjusting the value of the item to be adjusted of the first image acquisition device to a second value;

[0019] Wherein, if the adjustment operation includes an increase operation, the first value is the maximum value of the item to be adjusted, and the second value is the minimum value of the item to be adjusted; if the adjustment operation includes a decrease operation, the first value is the minimum value of the item to be adjusted, and the second value is the maximum value of the item to be adjusted;

[0020] The step of returning to the step of controlling the robot to move continuously with a preset distance as a moving step length comprises:

[0021] When the value of the item to be adjusted is the second value, return to the step of controlling the robot to move continuously with a preset distance as a moving step length.

[0022] Optionally, before returning to the step of controlling the robot to move continuously with a preset distance as a moving step, the method further includes:

[0023] In the case where an object to be photographed is identified from each three-dimensional image captured by the first image acquisition device after the current pause in movement, an adjustment operation is performed on an item to be adjusted of the first image acquisition device, wherein a specified boundary of the object to be photographed is not in any of the three-dimensional images, and the item to be adjusted includes a height and / or a pitch angle;

[0024] Using the first image acquisition device to photograph each area to be photographed, to obtain a plurality of three-dimensional images;

[0025] When the designated boundary of the object to be photographed is not in at least one three-dimensional image photographed by the first image acquisition device after the current adjustment operation, and the value of the item to be adjusted of the first image acquisition device is not the third value, returning to the step of performing the adjustment operation on the item to be adjusted of the first image acquisition device;

[0026] If the designated boundary of the object to be photographed is not in at least one three-dimensional image captured by the first image acquisition device after the current adjustment operation, the value of the item to be adjusted of the first image acquisition device is a third value, and the designated boundary includes a left boundary or a right boundary, move the preset distance and return to the step of pausing the movement;

[0027] The step of returning to the step of controlling the robot to move continuously with a preset distance as a moving step length comprises:

[0028] When the object to be photographed is not identified from the three-dimensional images, or the designated boundary of the object to be photographed is in at least one three-dimensional image taken by the first image acquisition device after the current adjustment operation, returning to the step of controlling the robot to move continuously with a preset distance as the moving step length; or

[0029] When the designated boundary of the object to be photographed is not in at least one three-dimensional image captured by the first image acquisition device after the current adjustment operation, the value of the item to be adjusted of the first image acquisition device is the third value, and the designated boundary does not include the left boundary or the right boundary, returning to the step of controlling the robot to move continuously with the preset distance as the moving step length;

[0030] Among them, if the specified boundary includes an upper boundary, the adjustment operation includes an increase operation; if the specified boundary includes a lower boundary, the adjustment operation includes a decrease operation; if the specified boundary includes a left boundary or a right boundary, the adjustment operation includes an increase operation or a decrease operation; if the adjustment operation includes an increase operation, the third value is the maximum value of the item to be adjusted; if the adjustment operation includes a decrease operation, the third value is the minimum value of the item to be adjusted.

[0031] Optionally, photographing each area to be photographed by the first image acquisition device to obtain a plurality of three-dimensional images includes:

[0032] For each area to be photographed, determine the density of objects included in the area to be photographed, and generate at least one shooting point in the area to be photographed according to the density, wherein the density of objects included in the area to be photographed is positively correlated with the number of shooting points included in the area to be photographed;

[0033] For each shooting point, the robot's posture is adjusted with the imaging center axis of the first image acquisition device passing through the shooting point as a target, and a three-dimensional image is captured by the first image acquisition device.

[0034] Optionally, there are multiple first image acquisition devices; when an object is included in the field of view of the first image acquisition device, pausing the movement includes:

[0035] When the object is included in the field of view of any first image acquisition device, pausing the movement;

[0036] The first image acquisition device is used to photograph each area to be photographed to obtain a plurality of three-dimensional images, including:

[0037] Selecting a target first image acquisition device based on the density of objects within the field of view of each first image acquisition device, wherein the density of objects within the field of view of the target first image acquisition device is higher than the density of objects within the field of view of other first image acquisition devices among the first image acquisition devices except the target first image acquisition device;

[0038] Each area to be photographed is photographed respectively by the target first image acquisition device to obtain multiple three-dimensional images, and the three-dimensional image of its own field of view is photographed by the other first image acquisition devices.

[0039] Optionally, controlling the robot to move continuously with a preset distance as a moving step length comprises:

[0040] Based on the preset route of the robot, the preset end point, at least one of the preset traversal path planning algorithms, and the current position of the robot, a moving direction is determined, and the robot continuously moves in the moving direction with a preset distance as a moving step in the area to be reconstructed.

[0041] Optionally, photographing the target object at multiple angles by a second image acquisition device of the robot to obtain multiple two-dimensional images includes:

[0042] For each preset shooting angle, the shooting angle of the target object by the second acquisition device is taken as the preset shooting angle, the posture of the robot is adjusted, and the target object is photographed by the second image acquisition device at the shooting angle to obtain a two-dimensional image.

[0043] Optionally, after photographing the target object at multiple angles by the second image acquisition device of the robot to obtain multiple two-dimensional images, the method further includes:

[0044] In the case where the target object is a complex object, the position and posture of the robot is adjusted with the distance between the third image acquisition device of the robot and the target object being less than a preset distance as a goal, wherein the complex object includes at least one of the following objects: an object whose size does not belong to a preset size range, an object that is blocked in the three-dimensional image, and an object whose height does not belong to a preset height range;

[0045] The target object is photographed by the third image acquisition device to obtain a two-dimensional image.

[0046] Optionally, the second image acquisition device is the first image acquisition device, and the first image acquisition device is located at the head of the robot; or, the second image acquisition device and the first image acquisition device are both located at the head of the robot, and face different directions respectively;

[0047] The third image acquisition device is located at the hand of the robot, and the posture adjustment range of the hand of the robot is different from the posture adjustment range of the head of the robot.

[0048] In a second aspect, an embodiment of the present application provides a three-dimensional scene modeling device, the device comprising:

[0049] An acquisition module is used to photograph the area to be reconstructed from multiple angles by using a first image acquisition device of the robot during movement in the area to be reconstructed, so as to obtain a plurality of three-dimensional images, wherein the plurality of three-dimensional images are used to perform three-dimensional modeling on the area to be reconstructed, so as to obtain a three-dimensional regional model of the area to be reconstructed;

[0050] A determination module, used to determine a target object mapped by the object model to be optimized included in the three-dimensional region model in the region to be reconstructed;

[0051] The optimization module is used to shoot the target object determined by the determination module at multiple angles through the second image acquisition device of the robot to obtain multiple two-dimensional images. The shooting angle of the target object by the second image acquisition device is controlled by adjusting the posture of the robot. The multiple two-dimensional images are used to optimize the model of the object to be optimized to obtain an optimized three-dimensional area model.

[0052] In a third aspect, an embodiment of the present application provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0053] Memory, used to store computer programs;

[0054] The processor is used to implement the three-dimensional scene modeling method described in any one of the first aspects above when executing the program stored in the memory.

[0055] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the three-dimensional scene modeling method described in any one of the first aspects above is implemented.

[0056] In a fifth aspect, an embodiment of the present application provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the three-dimensional scene modeling method described in any one of the first aspects above.

[0057] Beneficial effects of the embodiments of the present application:

[0058] The three-dimensional scene modeling method, device, electronic device and storage medium provided in the embodiments of the present application use the three-dimensional image obtained by shooting the area to be reconstructed from multiple angles by the first image acquisition device to perform three-dimensional modeling of the area to be reconstructed, and use the two-dimensional image obtained by shooting the target object in the area to be reconstructed from multiple angles by the second image acquisition device to optimize the object model to be optimized in the three-dimensional area model, thereby improving the accuracy of the three-dimensional area model. Moreover, when the first image acquisition device and the second image acquisition device are the same, since the image acquisition device is deployed on a robot, the adjustment of the robot's posture can directly affect the change of the posture of the image acquisition device. Therefore, by moving the robot in the area to be reconstructed and adjusting the posture such as stretching, squatting, etc., the shooting angle of the image acquisition device can be changed, thereby realizing multi-point and multi-posture image acquisition in the area to be reconstructed. Therefore, the embodiments of the present application do not require artificial deployment of cameras in multiple postures in the area to be reconstructed, which reduces the dependence of the three-dimensional modeling process on manual labor and reduces the consumption of labor costs. Alternatively, when the first image acquisition device and the second image acquisition device are different, in addition to being able to realize multi-point and multi-posture image acquisition and reduce labor costs, the embodiments of the present application have different focuses when performing image acquisition, that is, the first image acquisition device pays more attention to global features, while the second image acquisition device pays more attention to local features. For example, the first image acquisition device pays more attention to obtaining the position and size of each object in the area to be reconstructed, while the second image acquisition device pays more attention to the shape, surface color and texture of the target objects in the area to be reconstructed. Therefore, the image taken by the second image acquisition device can make up for the shortcomings of the image taken by the first image acquisition device. Therefore, by using the image taken by the second image acquisition device to optimize the three-dimensional area model obtained based on the image taken by the first image acquisition device, a more accurate three-dimensional area model can be obtained.

[0059] Of course, implementing any product or method of the present application does not necessarily require achieving all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0061] Figure 1 A flowchart of a three-dimensional scene modeling method provided in an embodiment of the present application;

[0062] Figure 2 A flowchart of another three-dimensional scene modeling method provided in an embodiment of the present application;

[0063] Figure 3 An exemplary schematic diagram of a field of view provided in an embodiment of the present application;

[0064] Figure 4 An exemplary schematic diagram of another field of view provided in an embodiment of the present application;

[0065] Figure 5 An exemplary schematic diagram of a shooting point within a field of view provided by an embodiment of the present application;

[0066] Figure 6 A flowchart of a method for further capturing a three-dimensional image provided by an embodiment of the present application;

[0067] Figure 7 A flowchart of another method for further capturing a three-dimensional image provided by an embodiment of the present application;

[0068] Figure 8 A schematic diagram of the structure of a three-dimensional scene modeling device provided in an embodiment of the present application;

[0069] Fig. 9 A schematic diagram of the structure of a robot provided in an embodiment of the present application. DETAILED DESCRIPTION

[0070] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field based on the present application belong to the scope of protection of the present application.

[0071] In order to reduce the consumption of manpower costs for three-dimensional scene modeling, an embodiment of the present application provides a three-dimensional scene modeling method, which is applied to an electronic device. The electronic device can be a robot, or the electronic device can be deployed on a robot. In the subsequent embodiments, the electronic device is a robot as an example for explanation. Among them, the robot is equipped with an image acquisition device, and the image acquisition device can be used to capture images of the environment in which it is located. The basic capabilities of the robot provided by the present application include at least mobility. In addition, the basic capabilities may also include the ability to stretch vertically upward, squat downward, and at least one of the abilities of lateral rotation, leaning forward, leaning back, and tilting.

[0072] In different scenarios, the robot can use its own basic capabilities and cooperate with one or more of its own robotic arms to perform specific operations required for specific work tasks in the scenario. Among them, the end effector of the robot's robotic arm is adjustable and replaceable. That is, the robot can adjust at least one of the posture, shape and relative position relationship between the components of the end effector of the robotic arm. Moreover, the connection between the robot's robotic arm and the end effector is a detachable connection, such as a threaded connection or a snap connection, so that in different scenarios, the robotic arm can be connected to different end effectors to support the execution of specific operations required for specific work tasks in the scenario.

[0073] For example, in a commodity selling scenario, the end effector of the robot's mechanical arm is a two-finger claw effector, and the robot can adjust the distance between the two fingers included in the end effector to perform the operation of grabbing or placing commodities.

[0074] For another example, in a product processing scenario, the end effector of the robot's robotic arm is an air suction effector. The robot can control the entry or exhaust of air in the suction cup included in the end effector, thereby adjusting the shape of the suction cup, and use the suction cup to perform product suction or placement operations.

[0075] In addition to basic capabilities, the above robots can also have human-computer interaction capabilities. For example, data transmission and reception capabilities such as voice reception and voice playback, as well as artificial intelligence capabilities such as voice recognition and natural language processing. This enables robots to complete various work tasks such as information query, personalized recommendations, and entertainment and leisure in scenarios with human-computer interaction needs, using their own human-computer interaction capabilities.

[0076] It should be noted that the capabilities of the robot in the embodiments of the present application are not limited to this.

[0077] The embodiment of the present application executes a three-dimensional scene modeling method by a robot that can perform work tasks, and can obtain a three-dimensional model of the robot's daily work scene without manually placing a camera in the scene, thereby reducing the labor cost consumption of three-dimensional scene modeling.

[0078] like Figure 1 As shown, the three-dimensional scene modeling method provided in the embodiment of the present application includes the following steps:

[0079] S101. During the movement in the area to be reconstructed, the first image acquisition device of the robot shoots the area to be reconstructed from multiple angles to obtain a plurality of three-dimensional images; wherein the plurality of three-dimensional images are used to perform three-dimensional modeling of the area to be reconstructed to obtain a three-dimensional area model of the area to be reconstructed.

[0080] In the embodiment of the present application, the area where the three-dimensional scene reconstruction is required is referred to as the area to be reconstructed. For example, the area to be reconstructed can be an indoor or outdoor scene, such as a warehouse, a factory, a pharmacy, a supermarket, or an office area, etc., where three-dimensional scene modeling is required. For example, a robot used for subsequent image collection can work in this scene, or other robots other than the image collection robot can work in this scene.

[0081] The first image acquisition device of the robot can capture the environment in which the robot is located to obtain a three-dimensional image. For example, the three-dimensional image can be: a depth image, a point cloud model, or a three-dimensional scanned image. Taking the three-dimensional image as a depth image as an example, the first image acquisition device can be a depth camera, such as a binocular depth camera, a structured light depth camera, or a time of flight (ToF) depth camera.

[0082] Optionally, after obtaining multiple three-dimensional images, the robot can directly perform three-dimensional modeling on the area to be reconstructed based on the multiple three-dimensional images when it has the three-dimensional modeling capability. Alternatively, when the robot does not have the three-dimensional modeling capability, or in order to reduce the occupation of the robot's computing resources, the robot can send multiple three-dimensional images to other devices, so that the other devices can perform three-dimensional modeling on the area to be reconstructed based on the received multiple three-dimensional images. For example, the other devices can be: servers, desktop computers, laptops, or other robots other than the robot that have three-dimensional modeling capabilities and communication capabilities. Among them, the other robots can be robots in an idle state, or robots with sufficient computing resources, etc., which are not limited here.

[0083] Taking the three-dimensional image as a depth image as an example, each pixel in the depth image has a depth value, and the depth value is used to reflect the distance between the three-dimensional point mapped by the pixel in the environment and the first image acquisition device. The depth image also has position information, and the position information is used to indicate the position of the first image acquisition device in the world coordinate system when shooting the depth image. When performing three-dimensional modeling, for each depth image, according to the depth value of each pixel included in the depth image, the position of the three-dimensional point mapped by the pixel in the area to be reconstructed in the coordinate system of the first image acquisition device can be determined, and combined with the position information of the depth image, the position of the three-dimensional point mapped by the pixel in the area to be reconstructed in the world coordinate system can be determined. After that, the point cloud of each three-dimensional point in the world coordinate system is segmented or merged, so as to divide the three-dimensional points belonging to the same plane or the same object. For example, the three-dimensional points belonging to the same plane include: three-dimensional points on the same wall, three-dimensional points on the ground, and three-dimensional points on the ceiling of the room, etc.; the three-dimensional points belonging to the same object include: three-dimensional points on the same shelf and three-dimensional points on the same product, etc. Then, the shape of the plane is reconstructed using a fitting algorithm for each group of three-dimensional points belonging to the same plane, thereby obtaining a spatial frame model included in the three-dimensional region model. And the shape of the object is reconstructed using a fitting algorithm for each group of three-dimensional points belonging to the same object, thereby obtaining an object model included in the three-dimensional region model.

[0084] This three-dimensional modeling process is only an example, and the three-dimensional modeling process that can be used in the embodiments of the present application is not limited to this. For example, since there is generally some overlap in the three-dimensional images taken by the first image acquisition device at adjacent times, the three-dimensional images can be spliced ​​according to the overlapping parts between the three-dimensional images. Then, the positions of the three-dimensional points mapped by the pixels included in the spliced ​​image in the area to be reconstructed in the world coordinate system are determined, and the three-dimensional regional model is constructed based on this.

[0085] Exemplarily, the robot can move in a circle in the area to be reconstructed in a circular route. During this process, the robot can shoot the area to be reconstructed through the first image acquisition device at a preset posture every specified time interval to obtain a three-dimensional image. Among them, the specified time interval can be pre-set and configured on the robot by humans according to relevant parameters, or the specified time interval can be determined by the robot periodically or non-periodically based on relevant parameters, and of course it can also be determined in real time if there are sufficient computing resources. For example, the relevant parameters include: the movement speed of the robot, the spatial size of the area to be reconstructed, and the field of view of the robot's image acquisition device. Or the robot can determine the specified time interval according to the density of objects included in the field of view of the first image acquisition device. For example, the initial value of the specified time interval is pre-configured, and when the robot executes S101 later, if it is detected that the density of objects included in the field of view of the first image acquisition device is greater than the first preset density, the specified time interval is reduced by a preset millisecond; if it is detected that the density is less than the second preset density, the specified time interval is increased by a preset millisecond, wherein the first preset density is greater than or equal to the second preset density. In the case where the first preset density is greater than the second preset density, if it is detected that the density of objects included in the field of view of the first image acquisition device is less than or equal to the first preset density, and greater than or equal to the second preset density, it is determined not to adjust the specified time interval. Alternatively, in the case where the first preset density is equal to the second preset density, if it is detected that the density of objects included in the field of view of the first image acquisition device is equal to the first preset density, it is determined not to adjust the specified time interval. Alternatively, other methods can be used to perform multi-angle shooting of the area to be reconstructed by the first image acquisition device. For specific methods, please refer to the following description of Figure 2 Related description.

[0086] S102: Determine a target object mapped by the object model to be optimized included in the three-dimensional region model in the region to be reconstructed.

[0087] Optionally, each object model in the three-dimensional region model can be an object model to be optimized. Or an object model with an incomplete structure in the three-dimensional region model can be an object model to be optimized, so as to reduce the number of object models to be optimized and improve the optimization speed. The specific method of determining the object model to be optimized can refer to the relevant description of the specific implementation method of S102 below.

[0088] The three-dimensional area model includes the position of each object model in the world coordinate system. For example, the three-dimensional area model includes the position of the marking point of each object model in the world coordinate system, wherein the marking point can be one or more points on the surface or inside of the object such as the center point, the center of gravity point, the centroid point, the edge point, etc. Therefore, the robot can obtain the position of the object model to be optimized in the world coordinate system, and use the object at this position in the area to be reconstructed as the target object. The specific method of the robot determining the target object can refer to the relevant description of the specific implementation method of S102 below.

[0089] S103. Use the second image acquisition device of the robot to shoot the target object at multiple angles to obtain multiple two-dimensional images; wherein the shooting angle of the target object by the second image acquisition device is controlled by adjusting the posture of the robot, and the multiple two-dimensional images are used to optimize the model of the object to be optimized to obtain an optimized three-dimensional area model.

[0090] The two-dimensional image may be a color image or a grayscale image, etc. For example, the color image may be a Red Green Blue (RGB) image, a Hue Saturation Brightness (HSB) image, or a Luminance Chrominance Chroma (YUV) image, etc.

[0091] Optionally, the second image acquisition device may be different from the first image acquisition device, for example, the second image acquisition device may be an RGB camera or an industrial camera, etc. Alternatively, due to the limitation of the robot's body size, a first image acquisition device with two-dimensional image and three-dimensional image acquisition capabilities may be configured in the robot, in which case the second image acquisition device is the first image acquisition device, thereby reducing the number of image acquisition devices that need to be configured for the robot. It can be seen that in the embodiment of the present application, the first image acquisition device and the second image acquisition device may be the same or different, so that the three-dimensional scene modeling solution provided in the embodiment of the present application can be applied to more types of robots, thereby expanding the scope of application of the three-dimensional scene modeling solution provided in the embodiment of the present application.

[0092] When optimizing the object model to be optimized, a 3D Gaussian modeling algorithm can be used. Among them, the optimization process of the object model to be optimized includes: image enhancement of each two-dimensional image, for example, image enhancement includes: one or more processes such as image filtering, image denoising and color enhancement. Then, key points are detected from each two-dimensional image after image enhancement, for example, key points include: the intersection of at least two edges of the target object, that is, corner points, and other edge points on the edge of the target object. Then, the key points extracted from each two-dimensional image are matched with the object model to be optimized, so as to establish a corresponding relationship between the two-dimensional image and the object model to be optimized. Then, a set of 3D Gaussian distributions is initialized based on the object model to be optimized, wherein each 3D Gaussian distribution represents a part of the surface of the object model to be optimized. Then, based on the corresponding relationship between each two-dimensional image and the object model to be optimized, the position, covariance matrix and color parameters of each 3D Gaussian distribution are adjusted by a stochastic gradient descent (SGD) algorithm. Then, based on each optimized 3D Gaussian distribution, the geometric structure, surface color, pattern, texture and other information of the object model to be optimized in the three-dimensional region model are adjusted to achieve the optimization of the object model to be optimized, thereby obtaining the optimized three-dimensional region model. The above optimization process of the object model to be optimized is only an example, and the optimization process that can be used in the embodiments of the present application is not limited to this.

[0093] Optionally, after obtaining the 3D region model in S101, S102 and S103 may be executed. Alternatively, in the process of shooting multiple 3D images in S101, 3D modeling may be performed in real time based on the shot 3D images, so as to obtain 3D sub-region models corresponding to each of the sub-regions included in the area to be reconstructed, and S102 and S103 may be executed to determine the target object in the sub-region, and optimize the object model to be optimized corresponding to the target object in the sub-region.

[0094] In the embodiment of the present application, the three-dimensional image obtained by shooting the area to be reconstructed at multiple angles using the first image acquisition device is used to perform three-dimensional modeling of the area to be reconstructed, and the two-dimensional image obtained by shooting the target object in the area to be reconstructed at multiple angles using the second image acquisition device is used to optimize the object model to be optimized in the three-dimensional area model, thereby improving the accuracy of the three-dimensional area model. Moreover, when the first image acquisition device and the second image acquisition device are the same, since the image acquisition device is deployed on a robot, the adjustment of the robot's posture can directly affect the change of the posture of the image acquisition device. Therefore, by moving the robot in the area to be reconstructed and adjusting the posture such as stretching, squatting, etc., the shooting angle of the image acquisition device can be changed, thereby realizing multi-point and multi-posture image acquisition in the area to be reconstructed. Therefore, the embodiment of the present application does not require artificial deployment of cameras in multiple postures in the area to be reconstructed, which reduces the dependence of the three-dimensional modeling process on manual labor and reduces the consumption of labor costs. Alternatively, when the first image acquisition device and the second image acquisition device are different, in addition to being able to realize multi-point and multi-posture image acquisition and reduce labor costs, the embodiments of the present application have different focuses when performing image acquisition, that is, the first image acquisition device pays more attention to global features, while the second image acquisition device pays more attention to local features. For example, the first image acquisition device pays more attention to obtaining the position and size of each object in the area to be reconstructed, while the second image acquisition device pays more attention to the shape, surface color and texture of the target objects in the area to be reconstructed. Therefore, the image taken by the second image acquisition device can make up for the shortcomings of the image taken by the first image acquisition device. Therefore, by using the image taken by the second image acquisition device to optimize the three-dimensional area model obtained based on the image taken by the first image acquisition device, a more accurate three-dimensional area model can be obtained.

[0095] The following is a detailed description of the three-dimensional scene modeling method provided in the embodiment of the present application:

[0096] See also Figure 2 In the process of moving in the area to be reconstructed in S101, the first image acquisition device of the robot takes multi-angle photos of the area to be reconstructed to obtain multiple three-dimensional images, including the following steps:

[0097] S1011. In the area to be reconstructed, the robot is controlled to move continuously with a preset distance as a moving step length.

[0098] Optionally, the robot may execute S101 after receiving the collection task. The robot may receive a collection task input manually; or, the collection task may be pre-configured in the robot and triggered at a fixed time, in which case the robot may obtain the collection task stored in itself when the task triggering time arrives; or, the robot may receive a collection task sent by other devices.

[0099] Optionally, the acquisition task may include a moving route within the area to be reconstructed, so that the robot can start from a preset starting point of the moving route within the area to be reconstructed, and move continuously along the moving route with a preset distance as a moving step length until it reaches a preset end point of the moving route.

[0100] Alternatively, the acquisition task may include a preset starting point and a preset end point for moving within the area to be reconstructed, so that the robot can plan a moving route within the area to be reconstructed based on the starting point and the end point, and move continuously along the moving route with a preset distance as the moving step until it reaches the preset end point.

[0101] Alternatively, the acquisition task can be used to instruct the robot to move autonomously in the area to be reconstructed, so that the robot can determine the moving route in the area to be reconstructed based on a preset traversal path planning algorithm, and continuously move in the area to be reconstructed according to the determined moving route with a preset distance as the moving step length until the area to be reconstructed has been traversed. For example, the preset traversal path planning algorithm can be: A* algorithm, Voronoi graph algorithm or depth-first search (DFS) algorithm, etc.

[0102] The preset distance may be a fixed distance preset manually. For example, the preset distance may be preset manually based on the size of the area to be reconstructed, the density of objects placed in the area to be reconstructed, and / or the moving speed of the robot, and configured on the robot. For example, the preset distance is 1 meter.

[0103] Alternatively, the default value of the preset distance can be pre-configured manually, and then the preset distance can be adjusted in real time according to the density of objects in the three-dimensional image captured by the first image acquisition device during the movement of the robot in the area to be reconstructed, so that the preset distance is more suitable for the environment in which the robot is located. For example, the default value of the preset distance is set manually in advance and configured on the robot according to the spatial size of the area to be reconstructed, the density of objects placed in the area to be reconstructed, and / or the moving speed of the robot. Then, during the continuous movement of the robot in the area to be reconstructed, the density of objects in the three-dimensional image captured by the first image acquisition device is periodically determined. If the density of objects in the three-dimensional image captured by the first image acquisition device is greater than the first preset density, the preset distance is reduced by a fixed value; if the density of objects in the three-dimensional image captured by the first image acquisition device is less than the second preset density, the preset distance is increased by a fixed value. Among them, the first preset density is greater than or equal to the second preset density. In the case where the first preset density is greater than the second preset density, if the density of objects in the three-dimensional image captured by the first image acquisition device is less than or equal to the first preset density, and greater than or equal to the second preset density, it is determined that the preset distance is not adjusted. Alternatively, when the first preset density is equal to the second preset density, if the density of the objects in the three-dimensional image captured by the first image acquisition device is equal to the first preset density, it is determined that the preset distance is not adjusted. That is, when the objects in the environment are dense, the robot can reduce the value of the preset distance, thereby increasing the number of times the robot determines whether the object is included in the field of view of the first image acquisition device, and can increase the frequency of pausing to move and shooting, thereby increasing the richness of information contained in the captured three-dimensional image. On the contrary, when the objects in the environment are sparse, the robot can usually accurately perform three-dimensional modeling by shooting a small number of three-dimensional images of the environment, so the value of the preset distance can be increased, thereby reducing the frequency of the robot pausing to move and shooting, making the robot's movement process smoother, and also reducing the number of times the robot determines whether the object is included in the field of view of the first image acquisition device, saving the robot's computing resources. Alternatively, when the objects in the environment are neither sparse nor dense, the robot can not adjust the value of the preset distance, saving the robot's computing resources.

[0104] Among them, the moving step length represents the distance that the robot moves each time. In the embodiment of the present application, the preset distance is used as the moving step length, that is, the robot moves the preset distance each time. For example, in the process of the robot moving the first preset distance from the starting point of the area to be reconstructed, if the field of view of the first image acquisition device does not include the object, then the robot continues to move the second preset distance after the first preset distance is completed; if the field of view of the first image acquisition device includes the object, S1012 is executed when the first preset distance is completed, that is, the movement is paused and shooting is performed, and then the second preset distance is continued, and so on. Moreover, the robot can judge whether the field of view of the first image acquisition device includes the object during the movement, instead of pausing the movement after each movement of the preset distance and then judging whether the field of view of the first image acquisition device includes the object, thereby alleviating the jamming of the robot during the continuous movement, and realizing continuous movement when the field of view of the first image acquisition device does not include the object, and pausing the movement and shooting when the field of view of the first image acquisition device includes the object.

[0105] S1012: When the object is included in the field of view of the first image acquisition device, suspend movement.

[0106] The robot can obtain a preview image captured in real time by a first image acquisition device, and determine whether the preview image contains a foreground area through a foreground recognition model, wherein the foreground area is the area occupied by the object. If so, it is determined that the object is included in the field of view of the first image acquisition device; otherwise, it is determined that the object is not included in the field of view of the first image acquisition device. Alternatively, the robot can cluster each pixel according to the depth value of each pixel in the preview image to obtain the area occupied by each cluster in the preview image. If there is any cluster whose area occupied by the preview image belongs to a preset area range, it is determined that the object is included in the field of view of the first image acquisition device; otherwise, it is determined that the object is not included in the field of view of the first image acquisition device. Alternatively, it is also possible to determine whether the object is included in the field of view of the first image acquisition device by other means, and the embodiments of the present application do not specifically limit this.

[0107] S1013: photograph each area to be photographed respectively by using a first image acquisition device to obtain multiple three-dimensional images.

[0108] The area to be photographed is a sub-area included in the current field of view of the first image acquisition device. Figure 3 As shown, Figure 3 The solid line frame represents the field of view, which includes three sub-areas A, B, and C, each of which is an area to be photographed. Figure 4 As shown, Figure 4The solid line frame represents the field of view, which includes four sub-areas A, B, C, and D, each of which is an area to be photographed. Alternatively, the sub-areas included in the field of view may be divided in other ways, which are not specifically limited in the embodiments of the present application.

[0109] The robot can use the first image acquisition device to take multiple three-dimensional images of each area to be photographed at different shooting angles, so that the content contained in each three-dimensional image is not exactly the same, making the overall content of each three-dimensional image richer and more conducive to three-dimensional modeling.

[0110] For the specific method of capturing multiple three-dimensional images for each area to be captured, reference may be made to the relevant description of the specific implementation method of S1013 below.

[0111] S1014 , returning to S1011 to control the robot to move continuously with a preset distance as a moving step length until the robot reaches a preset end point in the area to be reconstructed or the robot has traversed the area to be reconstructed.

[0112] In the case where the above acquisition task includes a moving route, or the acquisition task includes a preset starting point and a preset end point, the robot can determine whether the current position is the preset end point, wherein the end point of the moving route is the preset end point. If not, return to S1011 to control the robot to move continuously with a preset distance as the moving step length; if so, stop moving further and determine that the three-dimensional image acquisition of the area to be reconstructed is completed.

[0113] In the case where the above acquisition task is used to instruct the robot to move autonomously in the area to be reconstructed, the robot can determine whether it has traversed the area to be reconstructed. If not, return to S1011 to control the robot to move continuously with a preset distance as the moving step; if so, stop moving further and determine that the three-dimensional image acquisition of the area to be reconstructed is completed.

[0114] Through the above method, when the field of view of the first image acquisition device includes an object, the robot can take multiple three-dimensional images of multiple areas to be photographed within the current field of view, thereby obtaining multi-angle three-dimensional images of the same scene. These three-dimensional images can contain richer information about the objects within the field of view, so as to facilitate more accurate three-dimensional modeling of these objects in the future.

[0115] In an embodiment of the present application, the above-mentioned S1011 controls the robot to move continuously with a preset distance as the moving step length, which can be implemented as follows: based on the robot's preset route, preset end point, and at least one of the preset traversal path planning algorithms, and the robot's current position, determine the moving direction, and move continuously in the moving direction with the preset distance as the moving step length within the area to be reconstructed.

[0116] That is, the robot's continuous movement modes include the following three:

[0117] The first movement mode: Based on the current position of the robot and the preset route, the movement direction is determined, and the robot continuously moves in the moving direction with a preset distance as the moving step in the area to be reconstructed.

[0118] Among them, the preset route can be carried in the above-mentioned collection task, so that the robot can extract the preset route from the collection task.

[0119] The second movement mode: Based on the current position of the robot and the preset end point, the movement direction is determined, and the robot continuously moves in the moving direction within the area to be reconstructed with a preset distance as the moving step.

[0120] The preset end point may be carried in the above-mentioned collection task, so that the robot can extract the preset end point from the collection task.

[0121] The third movement mode: Based on the current position of the robot, a preset traversal path planning algorithm is used to determine the movement direction, and the robot continuously moves in the moving direction within the area to be reconstructed with a preset distance as the moving step.

[0122] For example, the preset traversal path planning algorithm may be: A* algorithm, Voronoi diagram algorithm or DFS algorithm, etc.

[0123] Through the above method, the robot can move autonomously in the area to be reconstructed, thereby driving its own first image acquisition device to be located at different positions in the area to be reconstructed, so as to shoot the area to be reconstructed from multiple different angles, reduce the impact of mutual occlusion of objects in the area to be reconstructed on three-dimensional modeling, and improve the richness of information contained in each three-dimensional image taken of the area to be reconstructed, so as to perform three-dimensional modeling more accurately.

[0124] After the robot stops moving, the robot in S1013 uses the first image acquisition device to photograph each area to be photographed respectively to obtain multiple three-dimensional images, which may include the following steps:

[0125] Step 1: for each area to be photographed, determine the density of objects included in the area to be photographed, and generate at least one shooting point in the area to be photographed according to the density, wherein the density of objects included in the area to be photographed is positively correlated with the number of shooting points included in the area to be photographed.

[0126] Optionally, the robot can obtain a preview image captured in real time by the first image acquisition device, determine the foreground area in the preview image through a foreground recognition model, and use the proportion of the foreground area in the preview image as the density of objects included in the area to be photographed. Alternatively, the robot can cluster each pixel according to the depth value of each pixel in the preview image to obtain the area occupied by each cluster in the preview image. Then, from each cluster, a target cluster whose occupied area falls within a preset area range is selected, and the proportion of the area of ​​the target cluster in the preview image is used as the density of objects included in the area to be photographed, or the number of target clusters is used as the density of objects included in the area to be photographed. Alternatively, the density of objects in the area to be photographed can be determined by other means, which is not specifically limited in the embodiments of the present application.

[0127] Afterwards, the robot can obtain the preset correspondence between each density range and the number of shooting points, wherein the order of the density range is positively correlated with the number of shooting points, and the order of the density range is determined in ascending order. Then, for each area to be photographed included in the field of view, the robot can determine from each density range the target density range to which the density of the object included in the area to be photographed belongs, and then obtain the target number of shooting points corresponding to the target density range according to the preset correspondence between each density range and the number of shooting points, and then generate the target number of shooting points in the area to be photographed.

[0128] For example, the preset corresponding relationship between each density range and the number of shooting points is shown in Table 1.

[0129] Table 1

[0130] Density range Number of shooting points [0,30%) 3 [30%,60%) 6 [60%,100%] 9

[0131] Assuming that the density of objects included in a to-be-shot area is 50%, the target density range to which the density belongs is determined to be [30%, 60%), and the number of target shooting points corresponding to [30%, 60%) is 6, that is, 6 shooting points need to be generated in the to-be-shot area.

[0132] Afterwards, when the robot generates shooting points in the area to be shot, it can generate multiple shooting points in the area to be shot in a uniform or random distribution. Alternatively, multiple shooting points can be generated in the area to be shot in other ways, which are not specifically limited in the embodiments of the present application. For example, Figure 5 The solid line frame in the figure represents the field of view, which includes four areas to be photographed, A, B, C, and D. The robot can evenly generate four shooting points in the area to be photographed A, where Figure 5 Each black dot in the graph represents a shooting point.

[0133] Step 2: For each shooting point, the robot's posture is adjusted with the imaging center axis of the first image acquisition device passing through the shooting point as the target, and a three-dimensional image is captured by the first image acquisition device.

[0134] Step 2: Adjusting the posture of the robot includes: adjusting the position parameters and posture parameters of the first part of the robot, where the first part is the installation part of the first image acquisition device on the robot. The position parameters (x, y, z) represent the longitude, latitude and altitude of the first part of the robot in the world coordinate system, and the posture parameters (H, P, R) represent the heading angle (Heading), pitch angle (Pitch) and roll angle (Roll) of the first part of the robot.

[0135] Taking the first image acquisition device located at the head of the robot as an example, the robot can adjust the height of the robot head by controlling the extension or folding of the lower limb area, thereby adjusting the height of the first image acquisition device. And the robot can adjust the angle of the head to rotate upward, downward, left or right, thereby adjusting the posture parameters of the first image acquisition device.

[0136] Optionally, the robot can adjust its own posture only with the imaging center axis of the first image acquisition device passing through the shooting point as the goal; or, the robot can adjust its own posture with the imaging center axis of the first image acquisition device passing through the shooting point and the shooting angle of the first image acquisition device to the shooting point being a preset shooting angle as the goal. Or the goal of the robot adjusting its own posture can also include other conditions, which are not specifically limited in the embodiments of the present application. After adjusting the posture, the robot captures a three-dimensional image through the first image acquisition device.

[0137] Through the above method, the disclosed embodiment can generate multiple shooting points in each area to be photographed, and the number of shooting points is positively correlated with the density of objects included in the area to be photographed. Since the higher the density of objects included in the area to be photographed, the more objects it usually contains and the richer the details, more shooting points are generated in the area to be photographed, and more three-dimensional images of the area to be photographed can be captured, so that the information of the area to be photographed contained in each three-dimensional image is richer and more complete, so as to more accurately perform three-dimensional modeling of the objects in the area to be photographed later. Conversely, the lower the density of objects included in the area to be photographed, the fewer objects it usually contains and the fewer details, so fewer shooting points are generated in the area to be photographed, which can reduce the number of shots of the area to be photographed and improve shooting efficiency.

[0138] In the embodiment of the present application, the first image acquisition device of the robot may be one or more. When there are multiple first image acquisition devices, the multiple first image acquisition devices are used to take pictures from different directions. For example, the multiple first image acquisition devices are all configured on the head of the robot and face different directions respectively.

[0139] In this case, the method in which the S1012 robot pauses moving when an object is included in the field of view of the first image acquisition device can be implemented as follows: pausing moving when an object is included in the field of view of any first image acquisition device.

[0140] Correspondingly, the above-mentioned S1013 robot uses the first image acquisition device to photograph each area to be photographed respectively to obtain multiple three-dimensional images. This can be implemented as follows: based on the density of objects within the field of view of each first image acquisition device, a target first image acquisition device is selected, and then each area to be photographed is photographed respectively by the target first image acquisition device to obtain multiple three-dimensional images, and three-dimensional images of its own field of view are photographed by other first image acquisition devices.

[0141] Among them, the method of determining the density of objects within the field of view of each first image acquisition device, and the method of the target first image acquisition device taking multiple three-dimensional images of each area to be photographed, can all be referred to the relevant description of the specific implementation method of S1013 above, and will not be repeated here.

[0142] In the embodiment of the present application, the density of objects within the field of view of the target first image acquisition device is higher than the density of objects within the field of view of other first image acquisition devices except the target first image acquisition device among the first image acquisition devices. The target first image acquisition device may be one or more. When there is one target first image acquisition device, the first image acquisition device with the largest density of objects within the field of view may be selected as the target first image acquisition device. When there are multiple target first image acquisition devices, multiple first image acquisition devices may be selected as the target first image acquisition devices in descending order of the density of objects within the field of view. The specific number of target first image acquisition devices may be set according to actual needs.

[0143] In the process of using the target first image acquisition device to respectively photograph each area to be photographed, before each time the target first image acquisition device takes a three-dimensional image, the robot will adjust its own posture to control the imaging center axis of the target first image acquisition device to pass through a shooting point in a region to be photographed, and at this time, the postures of other first image acquisition devices will also change accordingly. Therefore, when there is only one target first image acquisition device, each area to be photographed can be photographed by the target first image acquisition device, and a three-dimensional image can be taken by other first image acquisition devices each time the robot changes its posture. Alternatively, when there are multiple target first image acquisition devices, each area to be photographed can be photographed by each target first image acquisition device, and each time the robot changes its posture, a three-dimensional image can be taken by other first image acquisition devices except the target first image acquisition device.

[0144] Since the density of objects included in the field of view of the first image acquisition device can reflect the richness of information included in the field of view of the first image acquisition device, the embodiment of the present disclosure can select a first image acquisition device with richer information in its field of view, and perform multi-angle shooting of each area to be photographed in its field of view, thereby obtaining richer information in the field of view, so as to more accurately perform three-dimensional modeling of objects in the area to be photographed later. Other first image acquisition devices can also shoot accordingly, thereby obtaining three-dimensional images of the area to be reconstructed taken from other angles, further improving the richness of information contained in each three-dimensional image as a whole, and improving the accuracy of subsequent three-dimensional modeling.

[0145] In the embodiment of the present disclosure, before the robot returns to the step of controlling the robot to move continuously with a preset distance as the moving step length in the above S1014, the robot can also adjust the posture of the first image acquisition device and further shoot a three-dimensional image. The further shooting method includes the following two methods:

[0146] See also Figure 6 , the first method of further shooting includes the following steps:

[0147] S601: Perform an adjustment operation on an item to be adjusted of a first image acquisition device.

[0148] The items to be adjusted include: height and / or pitch angle, and the adjustment operation includes: increase operation or decrease operation.

[0149] Optionally, each time the robot adjusts the item to be adjusted of the first image acquisition device, it may randomly select the height, or the pitch angle, or the height and the pitch angle as the item to be adjusted this time. Alternatively, the robot may select the height and / or the pitch angle as the item to be adjusted this time according to a preset adjustment rule. For example, the preset adjustment rule is: simultaneously select the height and the pitch angle as the item to be adjusted this time. For another example, the preset adjustment rule is: when the height of the first image acquisition device does not reach the preset maximum height, select the height as the item to be adjusted this time; when the height of the first image acquisition device reaches the preset maximum height, select the pitch angle as the item to be adjusted this time.

[0150] When determining the items to be adjusted, the robot can also consider obstacles in its environment. For example, the preset adjustment rule is: when the distance between the robot's head and the obstacle above the head is greater than or equal to the preset distance, the height and pitch angle are selected as the items to be adjusted this time; when the distance between the robot's head and the obstacle above the head is less than the preset distance, only the pitch angle is selected as the item to be adjusted this time, so as to avoid collision between the robot's head and the obstacle above the head.

[0151] S602: Use a first image acquisition device to photograph each area to be photographed, to obtain a plurality of three-dimensional images.

[0152] Among them, the method of photographing each area to be photographed respectively by the first image acquisition device can refer to the relevant description of the specific implementation method of S1013 above, and will not be repeated here.

[0153] S603: Determine whether the value of the item to be adjusted is the first value. If the value of the item to be adjusted is not the first value, return to S601 to perform an adjustment operation on the item to be adjusted of the first image acquisition device. If the value of the item to be adjusted is the first value, execute S604.

[0154] S604: Adjust the value of the item to be adjusted of the first image acquisition device to a second value.

[0155] Wherein, if the adjustment operation includes an increase operation, the first value is the maximum value of the item to be adjusted, and the second value is the minimum value of the item to be adjusted. That is, the robot can increase the height and / or pitch angle of the first image acquisition device, wherein the amplitude of each adjustment can be preset, for example, the adjustment amplitude of the height is 10 cm, and the adjustment amplitude of the pitch angle is 15 degrees. Then, each area to be photographed within the current field of view of the robot is photographed by the first image acquisition device to obtain multiple three-dimensional images. Then, it is determined whether the height of the first image acquisition device reaches the preset maximum height, and whether the pitch angle reaches the preset maximum pitch angle. Wherein, the preset maximum height is the maximum height that the first image acquisition device can reach, and the preset maximum pitch angle is the maximum pitch angle that the first image acquisition device can reach. For example, when the first image acquisition device is located at the head of the robot, the preset maximum height is the height at which the first image acquisition device is located when the robot is fully extended longitudinally, and the preset maximum pitch angle is the angle between the imaging center axis of the first image acquisition device and the horizontal plane when the robot head is at the maximum angle of upward rotation. If the height of the first image acquisition device does not reach the preset maximum height or the pitch angle does not reach the preset maximum pitch angle, return to S601 to continue adjusting. If the height of the first image acquisition device reaches the preset maximum height and the pitch angle reaches the preset maximum pitch angle, the height of the first image acquisition device is reduced to the preset minimum height, and the pitch angle is reduced to the preset minimum pitch angle. Among them, the preset minimum height is the lowest height that the first image acquisition device can reach, and the preset minimum pitch angle is the minimum pitch angle that the first image acquisition device can reach. For example, when the first image acquisition device is located at the head of the robot, the preset minimum height is the height at which the first image acquisition device is located when the robot is completely bent or folded longitudinally, and the preset minimum pitch angle is the angle between the imaging center axis of the first image acquisition device and the horizontal plane when the robot head is at the maximum downward rotation angle.

[0156] On the contrary, if the adjustment operation includes a reduction operation, the first value is the minimum value of the item to be adjusted, and the second value is the maximum value of the item to be adjusted. That is, the robot can reduce the height and / or pitch angle of the first image acquisition device, wherein the magnitude of each reduction can be preset. Afterwards, each area to be photographed within its current field of view is photographed by the first image acquisition device to obtain multiple three-dimensional images. Then, it is determined whether the height of the first image acquisition device reaches the preset minimum height, and whether the pitch angle reaches the preset minimum pitch angle. If the height of the first image acquisition device does not reach the preset minimum height or the pitch angle does not reach the preset minimum pitch angle, return to S601 to continue adjusting. If the height of the first image acquisition device reaches the preset minimum height and the pitch angle reaches the preset minimum pitch angle, the height of the first image acquisition device is raised to the preset maximum height, and the pitch angle is increased to the preset maximum pitch angle.

[0157] In passing Figure 6When the method shown further shoots the area to be reconstructed, the above S1014 returns to the step of controlling the robot to move continuously with a preset distance as the moving step, which can be implemented as: when the value of the item to be adjusted is the second value, returns to the step of controlling the robot to move continuously with a preset distance as the moving step.

[0158] If the adjustment operation includes an increase operation, the first value is the maximum value of the item to be adjusted, and the second value is the minimum value of the item to be adjusted. In this case, the robot can return to the step of controlling the robot to move continuously with a preset distance as the moving step when the height of the first image acquisition device is a preset minimum height and the pitch angle is a preset minimum pitch angle.

[0159] On the contrary, if the adjustment operation includes a reduction operation, the first value is the minimum value of the item to be adjusted, and the second value is the maximum value of the item to be adjusted. In this case, the robot can return to the step of controlling the robot to move continuously with a preset distance as the moving step when the height of the first image acquisition device is a preset maximum height and the pitch angle is a preset maximum pitch angle.

[0160] Through the above method, the robot can adjust the height and pitch angle of the first image acquisition device. For scenes where the area to be reconstructed is a workshop or factory building with a high floor height, the objects placed in the area are generally high. Therefore, raising the height of the first image acquisition device and / or increasing the pitch angle can enable the first image acquisition device to capture higher objects more completely. For scenes where the area to be reconstructed is a basement or residential building with a low floor height, the objects placed in the area are generally low. Therefore, lowering the height and / or pitch angle of the first image acquisition device can enable the first image acquisition device to capture objects at a lower height more completely. In addition, in an embodiment of the present application, the robot can autonomously adjust the posture of the first image acquisition device without manually adjusting the posture of the three-dimensional image acquisition device, thereby improving the efficiency of capturing three-dimensional images of the area to be reconstructed.

[0161] See also Figure 7 Before returning to the step of controlling the robot to continuously move with a preset distance as the moving step length in S1014, the robot adjusts the posture of the first image acquisition device and further captures the three-dimensional image, the second method includes the following steps:

[0162] S701: Determine whether the object to be photographed is recognized in each three-dimensional image captured by the first image acquisition device after the current pause. If yes, execute S702; if no, execute S1014.

[0163] The designated boundary of the object to be photographed is not in any of the three-dimensional images photographed by the first image acquisition device after the current pause in movement.

[0164] S702: Perform an adjustment operation on an item to be adjusted of the first image acquisition device, wherein the item to be adjusted includes height and / or pitch angle.

[0165] The robot can match the same object in each three-dimensional image captured by the first image acquisition device after the current pause, so as to obtain the position of the same object in different three-dimensional images. And further, for each object, it is determined whether the specified boundary of the object is in at least one three-dimensional image captured by the first image acquisition device after the current pause. If so, it is determined that the object is not the object to be photographed, otherwise it is determined that the object is the object to be photographed.

[0166] The specified boundary may be an upper boundary, a lower boundary, a left boundary or a right boundary. For the upper boundary, the robot may determine whether the upper boundary of the object in the three-dimensional image overlaps with the upper boundary of the three-dimensional image. If so, it is determined that the upper boundary of the object is not in the three-dimensional image; otherwise, it is determined that the upper boundary of the object is in the three-dimensional image. Similarly, the robot may determine whether the lower boundary of the object in the three-dimensional image overlaps with the lower boundary of the three-dimensional image. If so, it is determined that the lower boundary of the object is not in the three-dimensional image, otherwise, it is determined that the lower boundary of the object to be photographed is in the three-dimensional image; and determine whether the left boundary of the object in the three-dimensional image overlaps with the left boundary of the three-dimensional image, if so, it is determined that the left boundary of the object is not in the three-dimensional image, otherwise, it is determined that the left boundary of the object to be photographed is in the three-dimensional image; determine whether the right boundary of the object in the three-dimensional image overlaps with the right boundary of the three-dimensional image, if so, it is determined that the right boundary of the object is not in the three-dimensional image, otherwise, it is determined that the right boundary of the object to be photographed is in the three-dimensional image. Alternatively, other methods may be used to determine whether the upper / lower / left / right boundaries of the object are in the three-dimensional image, which is not specifically limited in the embodiments of the present application.

[0167] If the specified boundary includes an upper boundary, the adjustment operation includes an increase operation, and the third value is the maximum value of the item to be adjusted. That is, if the object to be photographed whose upper boundary is not in any of the three-dimensional images is identified from the three-dimensional images taken by the first image acquisition device after the current pause, it means that the position of the object to be photographed is relatively high or the object itself is relatively high, so the robot can increase the height and / or pitch angle of the first image acquisition device and execute S703 to continue photographing the three-dimensional image, wherein the amplitude of each adjustment can be preset.

[0168] If the specified boundary includes the lower boundary, the adjustment operation includes a reduction operation, and the third value is the minimum value of the item to be adjusted. That is, if the object to be photographed whose lower boundary is not in any of the three-dimensional images is identified from the three-dimensional images taken by the first image acquisition device after the current pause, it means that the position of the object to be photographed is low or the object itself is short, so the robot can reduce the height and / or pitch angle of the first image acquisition device and execute S703 to continue photographing the three-dimensional image, wherein the amplitude of each adjustment can be preset.

[0169] If the designated boundary includes a left boundary or a right boundary, the adjustment operation includes an increase operation or a decrease operation. If the adjustment operation includes an increase operation, the third value is the maximum value of the item to be adjusted, and if the adjustment operation includes a decrease operation, the third value is the minimum value of the item to be adjusted. That is, if the left boundary or the right boundary of the object to be photographed is not in any of the three-dimensional images from the three-dimensional images taken by the first image acquisition device after the current pause movement, it means that the shooting angle of the first image acquisition device when taking each three-dimensional image after the current pause movement cannot cover the left boundary or the right boundary of the object to be photographed, so the robot can adjust the height and / or pitch angle of the first image acquisition device, and execute S703 to continue shooting the three-dimensional image, wherein the amplitude of each adjustment can be preset. Wherein, in the case where the designated boundary includes a left boundary or a right boundary, each time the adjustment operation is performed, an increase operation or a decrease operation can be randomly selected, or an increase operation or a decrease operation can be selected according to a preset selection rule. For example, the preset selection rule is: if the height of the first image acquisition device does not reach the preset maximum height, and the pitch angle of the first image acquisition device does not reach the preset maximum pitch angle, then select the increase operation; if the height of the first image acquisition device reaches the preset maximum height, or the pitch angle of the first image acquisition device reaches the preset maximum pitch angle, then select the decrease operation.

[0170] Among them, the way in which the robot adjusts the height and / or pitch angle of the first image acquisition device can refer to the relevant description in the above S604, which will not be repeated here.

[0171] S703: photograph each area to be photographed by using a first image acquisition device to obtain multiple three-dimensional images.

[0172] Among them, the way in which the robot captures a three-dimensional image of each area to be photographed through the first image acquisition device can be referred to the relevant description in the above S1013, which will not be repeated here.

[0173] S704: Determine whether the designated boundary of the object to be photographed is not in at least one three-dimensional image photographed by the first image acquisition device after the current adjustment operation. If yes, execute S705; if no, execute S1014.

[0174] S705: Determine whether the value of the item to be adjusted of the first image acquisition device is not the third value. If so, return to S702 to perform an adjustment operation on the item to be adjusted of the first image acquisition device. If not, execute S706.

[0175] S706: Determine whether the specified boundary includes the left boundary or the right boundary. If yes, execute S707; if no, execute S1014.

[0176] S707, move the preset distance, and return to S1012 to pause the movement.

[0177] When the designated boundary of the object to be photographed is not in at least one three-dimensional image photographed by the first image acquisition device after this adjustment operation, the value of the item to be adjusted of the first image acquisition device is the third value, and the designated boundary includes the left boundary or the right boundary, that is, when the left boundary or the right boundary of the object to be photographed cannot be photographed by adjusting the item to be adjusted of the first image acquisition device after the current pause, the robot can continue to move the preset distance. Since the left boundary or the right boundary of the object to be photographed may be included in the field of view of the first image acquisition device after the preset distance is moved, and when the field of view only includes the left boundary or the right boundary of the object to be photographed but does not include other objects, the robot may determine that there is no object in the field of view. In order to avoid the problem of missing the left / right boundary of the object to be photographed caused by this situation, the robot can pause the movement after moving the preset distance, regardless of whether the object is included in the field of view of the first image acquisition device, and execute S703 to further photograph each area to be photographed respectively through the first image acquisition device, thereby increasing the possibility of photographing the left / right boundary of the object to be photographed after the movement.

[0178] In passing Figure 7 In the case where the method shown further shoots the area to be reconstructed, the above S1014 returns to the step of controlling the robot to move continuously with a preset distance as the moving step length, which can be implemented as follows: when the object to be photographed is not identified from each three-dimensional image, or the specified boundary of the object to be photographed is in at least one three-dimensional image taken by the first image acquisition device after this adjustment operation, return to the step of controlling the robot to move continuously with a preset distance as the moving step length.

[0179] Alternatively, when the designated boundary of the object to be photographed is not in at least one three-dimensional image taken by the first image acquisition device after this adjustment operation, the value of the item to be adjusted of the first image acquisition device is the third value, and the designated boundary does not include the left boundary or the right boundary, return to the step of controlling the robot to move continuously with a preset distance as the moving step length.

[0180] In the embodiment of the present application, if the object to be photographed is not identified in the three-dimensional images taken by the first image acquisition device after the current pause in movement, it means that the objects photographed after the current pause in movement have been completely photographed, so the robot can execute S1014 to continue moving.

[0181] Alternatively, if the designated boundary of the object to be photographed is in at least one three-dimensional image taken by the first image acquisition device after this adjustment operation, it means that the designated boundary of the object to be photographed has been photographed after this adjustment, that is, the object to be photographed has been completely photographed, so the robot can execute S1014 to continue moving.

[0182] Alternatively, if the designated boundary of the object to be photographed is not in at least one three-dimensional image taken by the first image acquisition device after this adjustment operation, the value of the item to be adjusted of the first image acquisition device is the third value, and the designated boundary does not include the left boundary or the right boundary, it means that the designated boundary of the object to be photographed cannot be obtained by further photographing by adjusting the item to be adjusted, so the robot can execute S1014 to continue moving.

[0183] Through the above method, the robot can flexibly adjust the height and / or pitch angle of the first image capturing device according to the boundary conditions of the objects contained in the captured three-dimensional image, so as to further capture the objects to be captured that have not been fully captured in a targeted manner. While improving the efficiency of three-dimensional image capturing, it also improves the completeness of object capturing, thereby improving the accuracy of subsequent three-dimensional modeling.

[0184] After completing the three-dimensional image acquisition in the above S101, the robot can send the acquired three-dimensional images to other devices, so that other devices can perform three-dimensional modeling on the area to be reconstructed based on multiple three-dimensional images to obtain a three-dimensional area model of the area to be reconstructed.

[0185] When other devices perform three-dimensional modeling, the three-dimensional image and / or two-dimensional image of the area to be reconstructed taken by the fourth image acquisition device installed or placed in the area to be reconstructed, and the three-dimensional image acquired by the first image acquisition device can be combined to perform three-dimensional modeling of the area to be reconstructed. In combination with the image of the area to be reconstructed taken by the fourth image acquisition device, the influence of the visual blind spot of the first image acquisition device on the three-dimensional modeling can be reduced, thereby improving the accuracy of the three-dimensional modeling.

[0186] Alternatively, other devices may first use the three-dimensional image and / or two-dimensional image taken by the fourth image acquisition device on the area to be reconstructed to perform preliminary three-dimensional modeling of the area to be reconstructed, and obtain a preliminary three-dimensional regional model of the area to be reconstructed, at which time the rough spatial parameters of the area to be reconstructed can be obtained. Afterwards, other devices use the three-dimensional images sent by the first image acquisition device to optimize the preliminary three-dimensional regional model, thereby improving the speed and efficiency of three-dimensional reconstruction.

[0187] When three-dimensional modeling is performed by other devices, the method in which the above-mentioned S102 robot determines the target object mapped by the object model to be optimized included in the three-dimensional area model in the area to be reconstructed can be implemented as follows: receiving the target position sent by other devices, and taking the object located at the target position in the area to be reconstructed as the target object, wherein the target position is the position where the object model to be optimized included in the three-dimensional area model is mapped in the area to be reconstructed.

[0188] After three-dimensional modeling, the overall volume, area and other parameters of the area to be reconstructed, as well as the position, volume and size and other parameters of each object included in the area to be reconstructed can be obtained based on the three-dimensional area model. After three-dimensional modeling, other devices can determine the object model to be optimized from each object model included in the three-dimensional area model, and send the position of the object model to be optimized mapped in the area to be reconstructed to the robot. Among them, other devices can use the object model selected by the user as the object model to be optimized, or other devices can use each object model included in the three-dimensional area model as an object model to be optimized, or other devices can determine the object model to be optimized by other means, and the embodiments of the present disclosure do not specifically limit this.

[0189] In another scenario, when the robot has the ability of three-dimensional modeling, the robot can directly perform three-dimensional modeling on the area to be reconstructed based on the multiple three-dimensional images taken by the first image acquisition device to obtain a three-dimensional regional model of the area to be reconstructed. Afterwards, the robot determines the object model to be optimized from the three-dimensional regional model, and determines the target position of the object model to be optimized in the area to be reconstructed, and uses the object located at the target position in the area to be reconstructed as the target object.

[0190] After the target object is determined in the above S102, the robot can further photograph the target object, that is, the above S103 robot photographs the target object at multiple angles through the second image acquisition device of the robot to obtain multiple two-dimensional images. The method can be implemented as follows: for each preset shooting angle, the shooting angle of the target object by the second acquisition device is taken as the preset shooting angle, the robot's posture is adjusted, and the target object is photographed at the shooting angle through the second image acquisition device to obtain a two-dimensional image.

[0191] At this time, adjusting the position and posture of the robot includes: adjusting the position parameters and posture parameters of the second part of the robot. The second part is the installation part of the second image acquisition device on the robot. The preset shooting angle can be carried in the above-mentioned acquisition task, or the preset shooting angle can be pre-configured on the robot.

[0192] Optionally, the robot can adjust its own posture only with the shooting angle of the second acquisition device to the target object as the preset shooting angle, or with the shooting angle of the second acquisition device to the target object as the preset shooting angle, and the distance between the second acquisition device and the target object as the preset distance. Or the robot's goal of adjusting its own posture can also include other conditions, which are not specifically limited in the embodiments of the present application. After adjusting the posture, the robot captures a two-dimensional image through the second image acquisition device.

[0193] For example, the preset shooting angles include at least one of the following: a shooting angle toward the center point of the area to be reconstructed, a shooting angle away from the center point of the area to be reconstructed, a shooting angle toward the edge area, a shooting angle away from the edge area, a shooting angle toward the east, a shooting angle toward the west, a shooting angle toward the south, a shooting angle toward the north, a shooting angle toward the top, and a shooting angle toward the bottom, etc.

[0194] Through the above method, the robot can use the second image acquisition device to further shoot multi-angle two-dimensional images of the target object in the reconstruction area. Since the two-dimensional image can be a color image, it contains rich texture and color information on the surface of the target object, such as text and patterns on the surface of the object, so the texture and color of the surface of the object model to be optimized can be further supplemented based on the two-dimensional image. In addition, shooting two-dimensional images from multiple angles can further optimize the structure of the object model to be optimized, make up for the impact of the visual blind spot of the first image acquisition device on three-dimensional modeling, and improve the accuracy of three-dimensional modeling.

[0195] In the embodiment of the present application, after the robot shoots the target object at multiple angles through the second image acquisition device of the robot in the above S103 to obtain multiple two-dimensional images, the robot can also determine whether it is necessary to further shoot the target object, including the following steps:

[0196] Step 1: When the target object is a complex object, the robot's posture is adjusted with the goal that the distance between the robot's third image acquisition device and the target object is less than a preset distance.

[0197] The complex object includes at least one of the following objects: an object whose size does not fall within a preset size range, an object that is blocked in a three-dimensional image, and an object whose height does not fall within a preset height range.

[0198] Optionally, the robot can determine whether the target object is a complex object based on information such as the size, shape, and position of the target object. Alternatively, when the robot performs three-dimensional modeling, the robot can use the object model manually marked in the three-dimensional area model as a complex object model, and determine that the object mapped by the complex object model is a complex object. Alternatively, when other devices perform three-dimensional modeling, the other devices can use the object model manually marked in the three-dimensional area model as a complex object model, and set a complex label for the position mapped by the complex object model in the area to be reconstructed. The robot regards the object at the position with the complex label sent by other devices as a complex object.

[0199] Wherein, step 1 of adjusting the posture of the robot includes: adjusting the position parameters and posture parameters of a third part of the robot, the third part being the installation part of the third image acquisition device on the robot.

[0200] Optionally, the robot can adjust the robot's posture only based on the distance between the third image acquisition device and the target object being less than a preset distance, or based on the distance between the third image acquisition device and the target object being less than a preset distance, and the shooting angle of the third image acquisition device to the target object being a preset shooting angle. Alternatively, the robot's goal of adjusting its own posture may also include other conditions, which are not specifically limited in the embodiments of the present application.

[0201] For example, for large objects such as sofas, dining tables and shelves, after multi-angle shooting in S103, complete surface detail information can usually be obtained. However, for small and complex objects such as medicines, figurines or food displayed on shelves, after multi-angle shooting in S103, complete surface detail information may still not be obtained. Therefore, step 2 can be continued to further shoot the complex objects at close range to obtain more complete surface detail information of the complex objects.

[0202] Step 2: photograph the target object using a third image acquisition device to obtain a two-dimensional image.

[0203] The third image acquisition device is located at the hand of the robot. The second image acquisition device may be the first image acquisition device, which is located at the head of the robot, thereby reducing the number of image acquisition devices required to be configured on the head of the robot; or, the second image acquisition device and the first image acquisition device are both located at the head of the robot, and face different directions respectively.

[0204] Among them, the posture adjustment range of the robot's hand is different from the posture adjustment range of the robot's head. The posture adjustment range may include: horizontal adjustment range, vertical adjustment range and posture adjustment range, etc. For example, the vertical adjustment range of the robot's head is [0.5 meters, 1.2 meters], that is, the center point of the robot's head can be at a minimum of 0.5 meters from the ground and a maximum of 1.2 meters from the ground; the vertical adjustment range of the robot's hand is [0.3 meters, 1.5 meters], that is, the center point of the robot's hand can be at a minimum of 0.3 meters from the ground and a maximum of 1.5 meters from the ground.

[0205] Since there may be occlusions between objects and the position adjustment range of the robot head is limited, even if the second image acquisition device is used to shoot the target object from multiple angles, some details on the surface of the target object may not be captured. The position adjustment range of the third image acquisition device in the robot hand is larger. For example, the hand can be raised or lowered, so the range of adjustment in the vertical direction is wider relative to the head, and the hand can be extended in all directions, so the range of adjustment in the horizontal direction is wider relative to the head. For example, in a warehouse scenario, the third image acquisition device in the robot hand can extend into the shelf to capture two-dimensional images of objects placed inside the shelf. Therefore, the embodiment of the present application uses a third image acquisition device configured in the robot hand, which can further capture richer details on the surface of the target object, thereby improving the accuracy of three-dimensional modeling. In addition, the robot's operations such as carrying objects generally rely on the end effector of the robot arm, that is, the actuator of the robot hand. The use of a third image acquisition device also configured in the robot hand to acquire images of objects can match the images acquired by the third image acquisition device when the end effector performs operations to a higher degree. Therefore, three-dimensional modeling using images acquired by the third image acquisition device is more suitable for subsequent robot operations on objects through actuators based on three-dimensional regional models. For example, the robot can subsequently obtain the adjacent relationship between objects based on the three-dimensional regional model, and use this to choose a more appropriate way to grasp the object, thereby improving the success rate of grasping.

[0206] Through the above method, the robot can use the third image acquisition device to further take close-up photos of complex objects, thereby obtaining richer detailed information on the surface of the complex object and improving the accuracy of three-dimensional modeling of the complex object. When the target object is not a complex object, the second image acquisition device is usually used to take multi-angle photos of the target object, and complete information on the surface of the target object can be obtained. Therefore, it can be determined not to take further photos of the target object, thereby improving the efficiency of two-dimensional image acquisition.

[0207] In the following, in combination with actual application scenarios, the three-dimensional scene modeling method provided in the embodiment of the present application is illustrated by taking the first image acquisition device as a depth camera, the second image acquisition device and the third image acquisition device as RGB cameras as examples:

[0208] The robot receives a collection task sent by a server, wherein the collection task includes a moving route in a pharmacy. The robot starts to move continuously with a preset distance as a moving step length from the starting point of the moving route, and in the process of moving along the moving route, determines whether the field of view of the depth camera installed on the head includes an object. If the object is included, the movement is paused, and each area to be photographed included in the current field of view is photographed by the depth camera to obtain multiple three-dimensional images. The height and pitch angle of the depth camera are increased, and each area to be photographed in the current field of view is continued to be photographed by the depth camera to obtain multiple three-dimensional images. If the height of the depth camera is not the preset maximum height or the pitch angle is not the preset maximum pitch angle, the step of increasing the height and pitch angle of the depth camera is returned until the height of the depth camera is the preset maximum height and the pitch angle is the preset maximum pitch angle, the height of the depth camera is adjusted to the preset minimum height and the pitch angle is the preset minimum pitch angle, so as to realize the interval raising of the shooting angle of the depth camera in the pharmacy and collect three-dimensional images. After that, the robot returns to the step of continuously moving with a preset distance as a moving step length until the robot reaches the preset end point of the moving route.

[0209] The robot sends the collected depth image to the server, so that the server performs three-dimensional modeling of the pharmacy based on the received depth image to obtain a three-dimensional pharmacy model, and sends the position of each drug model and shelf model included in the three-dimensional pharmacy model in the world coordinate system to the robot.

[0210] For each received position, the robot takes the object at that position in the pharmacy as the target object, and uses the RGB camera on the head to shoot the target object from multiple angles to obtain multiple RGB images. Each RGB image includes information such as the color, pattern, and text of the drug packaging, or includes information such as the color of the shelf surface, pasted text instructions, and etched shelf identification numbers. When the target object is a complex object, the robot's position is adjusted so that the distance between the RGB camera of the robot's hand and the target object is less than the preset distance, and the RGB image is obtained by shooting the target object through the RGB camera of the hand, thereby achieving close-up shooting of the target object. Afterwards, the robot sends the RGB images taken of each target object to the server.

[0211] For each target object, the server optimizes the object model mapped by the target object in the three-dimensional drugstore model by using the RGB image taken of the target object, thereby obtaining an optimized three-dimensional drugstore model.

[0212] Based on the same inventive concept, corresponding to the above method embodiment, the present application embodiment also provides a three-dimensional scene modeling device, such as Figure 8 As shown, the device includes: a collection module 801, a determination module 802 and an optimization module 803;

[0213] The acquisition module 801 is used to take multi-angle photos of the area to be reconstructed by using the first image acquisition device of the robot during the process of moving in the area to be reconstructed, so as to obtain multiple three-dimensional images. The multiple three-dimensional images are used to perform three-dimensional modeling of the area to be reconstructed, so as to obtain a three-dimensional area model of the area to be reconstructed;

[0214] A determination module 802 is used to determine a target object mapped by the object model to be optimized included in the three-dimensional region model in the region to be reconstructed;

[0215] The optimization module 803 is used to shoot the target object determined by the determination module 802 at multiple angles through the second image acquisition device of the robot to obtain multiple two-dimensional images. The shooting angle of the target object by the second image acquisition device is controlled by adjusting the posture of the robot. The multiple two-dimensional images are used to optimize the model of the object to be optimized to obtain an optimized three-dimensional area model.

[0216] Optionally, the acquisition module 801 is specifically used for:

[0217] In the area to be reconstructed, the robot is controlled to move continuously with a preset distance as the moving step length;

[0218] When the object is included in the field of view of the first image acquisition device, pausing the movement;

[0219] Using a first image acquisition device to capture each area to be captured, respectively, to obtain a plurality of three-dimensional images, wherein the area to be captured is a sub-area included in the current field of view of the first image acquisition device;

[0220] Return to the step of controlling the robot to move continuously with a preset distance as the moving step length until the robot reaches a preset end point in the area to be reconstructed or the robot has traversed the area to be reconstructed.

[0221] Optionally, the device further comprises: an adjustment module, which is used to perform an adjustment operation on the to-be-adjusted item of the first image acquisition device before returning to the step of controlling the robot to move continuously with a preset distance as the moving step length, wherein the to-be-adjusted item comprises: height and / or pitch angle, and the adjustment operation comprises: an increase operation or a decrease operation;

[0222] The acquisition module 801 is further used to respectively photograph each area to be photographed by using a first image acquisition device to obtain multiple three-dimensional images;

[0223] The adjustment module is further used to return to the step of performing an adjustment operation on the item to be adjusted of the first image acquisition device when the value of the item to be adjusted is not the first value;

[0224] The adjustment module is further used to adjust the value of the to-be-adjusted item of the first image acquisition device to a second value when the value of the to-be-adjusted item is the first value;

[0225] Wherein, if the adjustment operation includes an increase operation, the first value is the maximum value of the item to be adjusted, and the second value is the minimum value of the item to be adjusted; if the adjustment operation includes a decrease operation, the first value is the minimum value of the item to be adjusted, and the second value is the maximum value of the item to be adjusted;

[0226] The adjustment module is specifically used to: when the value of the item to be adjusted is the second value, return to the step of controlling the robot to move continuously with a preset distance as the moving step length.

[0227] Optionally, the adjustment module is further used for, before returning to the step of controlling the robot to move continuously with a preset distance as the moving step length, in a case where the object to be photographed is identified from each three-dimensional image captured by the first image acquisition device after the current pause in movement, performing an adjustment operation on the item to be adjusted of the first image acquisition device, wherein the specified boundary of the object to be photographed is not in any of the three-dimensional images, and the item to be adjusted includes the height and / or the pitch angle;

[0228] The acquisition module 801 is further used to photograph each area to be photographed by using a first image acquisition device to obtain multiple three-dimensional images;

[0229] A calling module, configured to call the acquisition module 801 to perform the step of performing the adjustment operation on the item to be adjusted of the first image acquisition device when the designated boundary of the object to be captured is not in at least one three-dimensional image captured by the first image acquisition device after the current adjustment operation and the value of the item to be adjusted of the first image acquisition device is not the third value;

[0230] The calling module is further configured to move the preset distance and call the acquisition module 801 to execute the step of pausing the movement when the designated boundary of the object to be photographed is not in at least one three-dimensional image photographed by the first image acquisition device after the current adjustment operation, the value of the item to be adjusted of the first image acquisition device is the third value, and the designated boundary includes the left boundary or the right boundary;

[0231] The acquisition module 801 is specifically used for:

[0232] When the object to be photographed is not identified from each three-dimensional image, or the designated boundary of the object to be photographed is in at least one three-dimensional image taken by the first image acquisition device after the current adjustment operation, returning to the step of controlling the robot to move continuously with a preset distance as the moving step length; or

[0233] When the designated boundary of the object to be photographed is not in at least one three-dimensional image captured by the first image acquisition device after the current adjustment operation, the value of the item to be adjusted of the first image acquisition device is the third value, and the designated boundary does not include the left boundary or the right boundary, returning to the step of controlling the robot to move continuously with the preset distance as the moving step length;

[0234] Among them, if the specified boundary includes an upper boundary, the adjustment operation includes an increase operation; if the specified boundary includes a lower boundary, the adjustment operation includes a decrease operation; if the specified boundary includes a left boundary or a right boundary, the adjustment operation includes an increase operation or a decrease operation; if the adjustment operation includes an increase operation, the third value is the maximum value of the item to be adjusted; if the adjustment operation includes a decrease operation, the third value is the minimum value of the item to be adjusted.

[0235] Optionally, the acquisition module 801 is specifically used to: for each area to be photographed, determine the density of objects included in the area to be photographed, and generate at least one shooting point in the area to be photographed according to the density, and the density of objects included in the area to be photographed is positively correlated with the number of shooting points included in the area to be photographed; for each shooting point, adjust the robot's posture with the imaging center axis of the first image acquisition device passing through the shooting point as the target, and capture a three-dimensional image through the first image acquisition device.

[0236] Optionally, there are multiple first image acquisition devices; the acquisition module 801 is specifically used to:

[0237] When the object is included in the field of view of any first image acquisition device, pausing the movement;

[0238] The acquisition module 801 is specifically used to: select a target first image acquisition device based on the density of objects in the field of view of each first image acquisition device, the density of objects in the field of view of the target first image acquisition device is higher than the density of objects in the field of view of other first image acquisition devices except the target first image acquisition device among the first image acquisition devices; photograph each area to be photographed by the target first image acquisition device to obtain multiple three-dimensional images, and photograph three-dimensional images of its own field of view by other first image acquisition devices.

[0239] Optionally, the acquisition module 801 is specifically used to: determine the moving direction based on the robot's preset route, preset end point, at least one of the preset traversal path planning algorithms, and the robot's current position, and continuously move in the moving direction within the area to be reconstructed with a preset distance as the moving step.

[0240] Optionally, the optimization module 803 is specifically used to: for each preset shooting angle, take the shooting angle of the target object of the second acquisition device as the preset shooting angle as the target, adjust the posture of the robot, and obtain a two-dimensional image by shooting the target object at the shooting angle through the second image acquisition device.

[0241] Optionally, the device further includes: an adjustment module, which is used to, after shooting the target object at multiple angles through the second image acquisition device of the robot to obtain multiple two-dimensional images, if the target object is a complex object, adjust the robot's posture with the distance between the third image acquisition device of the robot and the target object being less than a preset distance as a goal, wherein the complex object includes at least one of the following objects: an object whose size does not fall within a preset size range, an object that is blocked in a three-dimensional image, and an object whose height does not fall within a preset height range;

[0242] The shooting module is used to shoot the target object through a third image acquisition device to obtain a two-dimensional image.

[0243] Optionally, the second image acquisition device is the first image acquisition device, and the first image acquisition device is located on the head of the robot; or, the second image acquisition device and the first image acquisition device are both located on the head of the robot, and face different directions respectively;

[0244] The third image acquisition device is located at the hand of the robot, and the position adjustment range of the hand of the robot is different from the position adjustment range of the head of the robot.

[0245] The present application also provides an electronic device, such as Fig. 9 As shown, it includes a processor 901, a communication interface 902, a memory 903 and a communication bus 904, wherein the processor 901, the communication interface 902, and the memory 903 communicate with each other through the communication bus 904, and the memory 903 is used to store computer programs; the processor 901 is used to implement the method performed by the above-mentioned electronic device in the above-mentioned method embodiment when executing the program stored in the memory 903.

[0246] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0247] The communication interface is used for communication between the above electronic device and other devices.

[0248] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0249] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0250] In another embodiment provided in the present application, a computer-readable storage medium is also provided, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the method for three-dimensional scene modeling described in any of the above embodiments is implemented.

[0251] In another embodiment provided in the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute the three-dimensional scene modeling method described in any one of the above embodiments.

[0252] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive Solid State Disk (SSD)), etc.

[0253] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0254] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0255] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. A three-dimensional scene modeling method, characterized in that: The method comprises: In the process of moving in the area to be reconstructed, the area to be reconstructed is photographed from multiple angles by a first image acquisition device of the robot to obtain a plurality of three-dimensional images, and the plurality of three-dimensional images are used to perform three-dimensional modeling on the area to be reconstructed to obtain a three-dimensional area model of the area to be reconstructed; Determine a target object mapped by the object model to be optimized included in the three-dimensional region model within the region to be reconstructed; The target object is photographed at multiple angles by the second image acquisition device of the robot to obtain multiple two-dimensional images. The shooting angle of the target object by the second image acquisition device is controlled by adjusting the posture of the robot. The multiple two-dimensional images are used to optimize the model of the object to be optimized to obtain an optimized three-dimensional area model.

2. The method according to claim 1, characterized in that During the movement in the area to be reconstructed, the first image acquisition device of the robot shoots the area to be reconstructed from multiple angles to obtain multiple three-dimensional images, including: In the area to be reconstructed, the robot is controlled to move continuously with a preset distance as a moving step length; When the object is included in the field of view of the first image acquisition device, pausing the movement; Using the first image acquisition device to capture each area to be captured, respectively, to obtain a plurality of three-dimensional images, wherein the area to be captured is a sub-area included in the current field of view of the first image acquisition device; Return to the step of controlling the robot to move continuously with a preset distance as a moving step length until the robot reaches a preset end point in the area to be reconstructed or the robot has traversed the area to be reconstructed.

3. The method according to claim 2, characterized in that Before returning to the step of controlling the robot to move continuously with a preset distance as a moving step length, the method further includes: Performing an adjustment operation on an item to be adjusted of the first image acquisition device, wherein the item to be adjusted includes: height and / or pitch angle, and the adjustment operation includes: an increase operation or a decrease operation; Using the first image acquisition device to photograph each area to be photographed, to obtain a plurality of three-dimensional images; In a case where the value of the item to be adjusted is not the first value, returning to the step of performing the adjustment operation on the item to be adjusted of the first image acquisition device; When the value of the item to be adjusted is the first value, adjusting the value of the item to be adjusted of the first image acquisition device to a second value; Wherein, if the adjustment operation includes an increase operation, the first value is the maximum value of the item to be adjusted, and the second value is the minimum value of the item to be adjusted; if the adjustment operation includes a decrease operation, the first value is the minimum value of the item to be adjusted, and the second value is the maximum value of the item to be adjusted; The step of returning to the step of controlling the robot to move continuously with a preset distance as a moving step length comprises: When the value of the item to be adjusted is the second value, return to the step of controlling the robot to move continuously with a preset distance as a moving step length.

4. The method according to claim 2, characterized in that: Before returning to the step of controlling the robot to move continuously with a preset distance as a moving step length, the method further includes: In the case where an object to be photographed is identified from each three-dimensional image captured by the first image acquisition device after the current pause in movement, an adjustment operation is performed on an item to be adjusted of the first image acquisition device, wherein a specified boundary of the object to be photographed is not in any of the three-dimensional images, and the item to be adjusted includes a height and / or a pitch angle; Using the first image acquisition device to photograph each area to be photographed, to obtain a plurality of three-dimensional images; When the designated boundary of the object to be photographed is not in at least one three-dimensional image photographed by the first image acquisition device after the current adjustment operation, and the value of the item to be adjusted of the first image acquisition device is not the third value, returning to the step of performing the adjustment operation on the item to be adjusted of the first image acquisition device; If the designated boundary of the object to be photographed is not in at least one three-dimensional image captured by the first image acquisition device after the current adjustment operation, the value of the item to be adjusted of the first image acquisition device is a third value, and the designated boundary includes a left boundary or a right boundary, move the preset distance and return to the step of pausing the movement; The step of returning to the step of controlling the robot to move continuously with a preset distance as a moving step length comprises: When the object to be photographed is not identified from the three-dimensional images, or the designated boundary of the object to be photographed is in at least one three-dimensional image taken by the first image acquisition device after the current adjustment operation, returning to the step of controlling the robot to move continuously with a preset distance as the moving step length; or When the designated boundary of the object to be photographed is not in at least one three-dimensional image captured by the first image acquisition device after the current adjustment operation, the value of the item to be adjusted of the first image acquisition device is the third value, and the designated boundary does not include the left boundary or the right boundary, returning to the step of controlling the robot to move continuously with the preset distance as the moving step length; Among them, if the specified boundary includes an upper boundary, the adjustment operation includes an increase operation; if the specified boundary includes a lower boundary, the adjustment operation includes a decrease operation; if the specified boundary includes a left boundary or a right boundary, the adjustment operation includes an increase operation or a decrease operation; if the adjustment operation includes an increase operation, the third value is the maximum value of the item to be adjusted; if the adjustment operation includes a decrease operation, the third value is the minimum value of the item to be adjusted.

5. The method according to any one of claims 2 to 4, characterized in that: The controlling the robot to move continuously with a preset distance as a moving step length comprises: Based on the preset route of the robot, the preset end point, at least one of the preset traversal path planning algorithms, and the current position of the robot, determine the moving direction, and continuously move in the to-be-reconstructed area in the moving direction with a preset distance as a moving step length; The first image acquisition device is used to photograph each area to be photographed to obtain a plurality of three-dimensional images, including: For each area to be photographed, determine the density of objects included in the area to be photographed, and generate at least one shooting point in the area to be photographed according to the density, wherein the density of objects included in the area to be photographed is positively correlated with the number of shooting points included in the area to be photographed; For each shooting point, the robot's posture is adjusted with the imaging center axis of the first image acquisition device passing through the shooting point as a target, and a three-dimensional image is captured by the first image acquisition device.

6. The method according to any one of claims 2 to 4, characterized in that: There are multiple first image acquisition devices; when the object is included in the field of view of the first image acquisition device, pausing the movement comprises: When the object is included in the field of view of any first image acquisition device, pausing the movement; The first image acquisition device is used to photograph each area to be photographed to obtain a plurality of three-dimensional images, including: Selecting a target first image acquisition device based on the density of objects within the field of view of each first image acquisition device, wherein the density of objects within the field of view of the target first image acquisition device is higher than the density of objects within the field of view of other first image acquisition devices among the first image acquisition devices except the target first image acquisition device; Each area to be photographed is photographed respectively by the target first image acquisition device to obtain multiple three-dimensional images, and the three-dimensional image of its own field of view is photographed by the other first image acquisition devices.

7. The method according to any one of claims 1 to 6, characterized in that: The second image acquisition device of the robot shoots the target object at multiple angles to obtain multiple two-dimensional images, including: For each preset shooting angle, taking the shooting angle of the target object by the second acquisition device as the preset shooting angle as the target, adjusting the posture of the robot, and photographing the target object at the shooting angle by the second image acquisition device to obtain a two-dimensional image; After the target object is photographed at multiple angles by the second image acquisition device of the robot to obtain multiple two-dimensional images, the method further includes: In the case where the target object is a complex object, the position and posture of the robot is adjusted with the distance between the third image acquisition device of the robot and the target object being less than a preset distance as a goal, wherein the complex object includes at least one of the following objects: an object whose size does not belong to a preset size range, an object that is blocked in the three-dimensional image, and an object whose height does not belong to a preset height range; photographing the target object by means of the third image acquisition device to obtain a two-dimensional image; The second image acquisition device is the first image acquisition device, and the first image acquisition device is located at the head of the robot; or the second image acquisition device and the first image acquisition device are both located at the head of the robot and face different directions respectively; The third image acquisition device is located at the hand of the robot, and the posture adjustment range of the hand of the robot is different from the posture adjustment range of the head of the robot.

8. A three-dimensional scene modeling device, characterized in that: The device comprises: An acquisition module is used to photograph the area to be reconstructed from multiple angles by using a first image acquisition device of the robot during movement in the area to be reconstructed, so as to obtain a plurality of three-dimensional images, wherein the plurality of three-dimensional images are used to perform three-dimensional modeling on the area to be reconstructed, so as to obtain a three-dimensional regional model of the area to be reconstructed; A determination module, used to determine a target object mapped by the object model to be optimized included in the three-dimensional region model in the region to be reconstructed; The optimization module is used to shoot the target object determined by the determination module at multiple angles through the second image acquisition device of the robot to obtain multiple two-dimensional images. The shooting angle of the target object by the second image acquisition device is controlled by adjusting the posture of the robot. The multiple two-dimensional images are used to optimize the model of the object to be optimized to obtain an optimized three-dimensional area model.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the method according to any one of claims 1 to 7 when executing a program stored in a memory.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.