Method, apparatus, electronic device, and readable storage medium for establishing a local map

Abstract

The present application provides a method, an apparatus, an electronic device, and a readable storage medium for establishing a local map. The method includes: obtaining a field-of-view image corresponding to a to-be-measured environment collected by a monocular camera, and a grid map corresponding to the to-be-measured environment; determining a passage area in the to-be-measured environment according to the field-of-view image, where the passage area includes an initial area and an extended area; obtaining initial grid information corresponding to the initial area in the grid map, and determining extended grid information corresponding to the extended area according to the initial grid information; updating the extended grid information to the grid map to establish a local map corresponding to the to-be-measured environment. The present application improves the accuracy of establishing a complete local map and also reduces the cost of establishing the local map.

Description

Technical Field

[0001] This application relates to the technical field of robotics, and particularly to a method, device, electronic device, and readable storage medium for establishing a local map. Background Art

[0002] A local map is a map centered on a robot that updates grid information of a grid map within a certain range around the robot according to the movement of the robot. The robot can perform tasks such as moving or avoiding obstacles with the help of the local map. To understand the environmental information around, a depth sensor is usually set on the robot, and the environmental information collected by the depth sensor is used to update the local map.

[0003] However, when establishing a local map using a depth sensor in the traditional method, the depth sensor is affected by environmental factors, such as lighting factors, the color and material of the object surface, etc., resulting in missing depth information collected and unable to accurately establish a complete local map. Summary of the Invention

[0004] This application aims to at least solve one of the technical problems in the related art to some extent. For this reason, an object of this application is to provide a method, device, electronic device, and readable storage medium for establishing a local map, which improves the accuracy of establishing a complete local map.

[0005] One aspect of this application provides a method for establishing a local map. The method includes: obtaining a vision image corresponding to a to-be-measured environment collected by a monocular camera, and a grid map corresponding to the to-be-measured environment; determining a passage area in the to-be-measured environment according to the vision image, where the passage area includes an initial area and an extended area; obtaining initial grid information corresponding to the initial area in the grid map, and determining extended grid information corresponding to the extended area according to the initial grid information; updating the extended grid information to the grid map to establish a local map corresponding to the to-be-measured environment.

[0006] Another aspect of this application provides a device for establishing a local map. The device includes: a collection module, configured to obtain a vision image corresponding to a to-be-measured environment collected by a monocular camera, and a grid map corresponding to the to-be-measured environment. A passage area determination module, configured to determine a passage area in the to-be-measured environment according to the vision image, where the passage area includes an initial area and an extended area. An extended grid information determination module, configured to obtain initial grid information corresponding to the initial area in the grid map, and determine extended grid information corresponding to the extended area according to the initial grid information. A local map establishment module, configured to update the extended grid information to the grid map to establish a local map corresponding to the to-be-measured environment.

[0007] Another aspect of the present application provides an electronic device, which may include a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the steps in the method for establishing a local map as described in any of the above embodiments are implemented.

[0008] Another aspect of the present application provides a computer-readable storage medium storing a computer program, which is adapted to be loaded by a processor to execute the steps in the method for establishing a local map as described in any of the above embodiments.

[0009] According to a method, device, electronic device, and readable storage medium for establishing a local map of the present application, using the vision image of a monocular camera, an extended area is extended from the initial area of the passage area, and combined with the initial grid information of the grid map, the extended grid information of the extended area is supplemented, improving the accuracy of establishing a complete local map and also reducing the cost of establishing the local map. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 is a schematic diagram of the hardware structure of a robot provided by an embodiment of the present application;

[0011] Figure 2 is a schematic diagram of the mechanical structure of a robot provided by an embodiment of the present application;

[0012] Figure 3 is a schematic flowchart of the method for establishing a local map provided by an embodiment of the present application;

[0013] Figure 4 is a schematic diagram of the detection range centered on the robot provided by an embodiment of the present application;

[0014] Figure 5 is a schematic diagram of the passage area provided by an embodiment of the present application;

[0015] Figure 6 is a schematic diagram of the extended grid of the hole type provided by an embodiment of the present application;

[0016] Figure 7 is a schematic diagram of the extended grid of the extension type provided by an embodiment of the present application;

[0017] Figure 8 is an application scenario diagram of the local map centered on the robot provided by an embodiment of the present application;

[0018] Figure 9 is a block diagram of an apparatus for establishing a local map provided by an embodiment of the present application;

[0019] Figure 10 It is a schematic structural diagram of an electronic device provided by an embodiment of the present application;

[0020] Figure 11 It is a schematic structural diagram of a computer-readable storage medium provided by an embodiment of the present application. Detailed implementation manners

[0021] To better understand the present application, more detailed descriptions of various aspects of the present application will be made with reference to the accompanying drawings. It should be understood that these detailed descriptions are only descriptions of exemplary embodiments of the present application and do not limit the scope of the present application in any way. Throughout the specification, the same reference numerals refer to the same elements. The expression "and / or" includes any and all combinations of one or more of the associated listed items.

[0022] It should be noted that in this specification, the expressions such as first, second, third, etc. are only used to separate one feature from another feature region and do not represent any limitation on the features, especially do not represent any order of precedence. Therefore, without departing from the teachings of the present application, the first document type discussed in the present application may also be referred to as the second document type, and the first document level may also be referred to as the second document level, and vice versa.

[0023] In the accompanying drawings, for the sake of convenience of illustration, the thickness, dimensions, and shapes of the components have been slightly adjusted. The accompanying drawings are only examples and are not drawn strictly to scale. As used herein, the terms "substantially", "about", and similar terms are used as terms indicating approximation, rather than terms indicating degree, and are intended to illustrate the inherent deviations in measured or calculated values that would be recognized by those of ordinary skill in the art.

[0024] It should also be understood that expressions such as "including", "including having", "having", "containing", and / or "containing having" are open-ended rather than closed-ended expressions in this specification, which means that there are the stated features, elements, and / or components, but do not exclude the existence of one or more other features, elements, components, and / or their combinations. In addition, when an expression such as "at least one of..." appears after the list of listed features, it modifies the entire list of features, rather than just an individual element in the list. In addition, when describing the embodiments of the present application, the use of "may" means "one or more embodiments of the present application". And the term "exemplary" is intended to refer to an example or illustration.

[0025] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. It should also be understood that, unless explicitly stated otherwise in this application, terms defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and should not be interpreted in an idealized or overly formal sense.

[0026] It should be noted that, without conflict, the embodiments in this application and the features in the embodiments can be combined with each other. Additionally, unless explicitly defined or in conflict with the context, the specific steps included in the methods described in this application do not have to be limited to the recited order and can be executed in any order or in parallel. The following will detail this application with reference to the drawings and in combination with the embodiments.

[0027] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0028] In the subsequent descriptions, the suffixes such as "module", "component", or "unit" used to represent components are only for the convenience of explaining the present invention and have no specific meaning in themselves. Therefore, "module", "component", or "unit" can be used interchangeably.

[0029] Please refer to Figure 1 , Figure 1 which is a schematic diagram of the hardware structure of a robot provided according to an embodiment of this application. In the Figure 1 illustrated embodiment, the robot 100 includes a mechanical unit 101, a communication unit 102, a sensing unit 103, an interface unit 104, a storage unit 105, a display unit 106, an input unit 107, a control module 110, and a power supply 111. The various components of the robot 100 can be connected in any manner, including wired or wireless connections, etc. Those skilled in the art can understand that Figure 1 the specific structure of the robot 100 shown in

[0030] Figure 2 does not constitute a limitation on the robot 100. The robot 100 can include more or fewer components than shown in the figure, and some components are not essential components of the robot 100 and can be omitted or combined with some components entirely within the scope of not changing the essence of the invention as needed. Figure 1 and Figure 2 to specifically introduce each component of the robot 100:

[0031] The mechanical unit 101 is the hardware of the robot 100. As Figure 1As shown, the mechanical unit 101 may include a drive board 1011, a motor 1012, and a mechanical structure 1013. As Figure 2 shown, the mechanical structure 1013 may include a fuselage main body 1014, extendable legs 1015, and feet 1016. In other embodiments, the mechanical structure 1013 may further include one or more of an extendable robotic arm (not shown in the figure), a rotatable head structure 1017, a wagging tail structure 1018, a load-carrying structure 1019, a saddle structure 1020, or a camera structure 1021, etc. It should be noted that each component module of the mechanical unit 101 can be one or multiple, and can be set according to specific circumstances. For example, the number of legs 1015 can be 4, and each leg 1015 can be configured with 3 motors 1012, corresponding to 12 motors 1012.

[0032] The communication unit 102 can be used for signal reception and transmission, and can also communicate with the network and other devices. For example, after receiving instruction information sent by a remote control or other robots 100 to move in a specific gait at a specific speed value in a specific direction, it is transmitted to the control module 110 for processing. The communication unit 102 includes a Wi-Fi module, a 4G module, a 5G module, a Bluetooth module, or an infrared module, etc.

[0033] The sensing unit 103 is used to obtain information data on the surrounding environment of the robot 100 and monitor parameter data of each component inside the robot 100, and send them to the control module 110. The sensing unit 103 includes a variety of sensors, such as sensors for obtaining the surrounding environment information: lidar (for remote object detection, distance determination, and / or speed value determination), millimeter-wave radar (for short-range object detection, distance determination, and / or speed value determination), cameras, infrared cameras, Global Navigation Satellite System (GNSS), etc. Such as sensors for monitoring each component inside the robot 100: Inertial Measurement Unit (IMU) (for measuring speed values, acceleration values, and angular velocity values), sole sensors (for monitoring the position of the sole contact point, sole posture, ground contact force magnitude and direction), temperature sensors (for detecting component temperature). As for other sensors that the robot 100 can also be configured with, such as load sensors, touch sensors, motor angle sensors, torque sensors, etc., they will not be elaborated here.

[0034] The interface unit 104 can be used to receive inputs (such as data information, power, etc.) from external devices and transmit the received inputs to one or more components within the robot 100, or can be used to output (such as data information, power, etc.) to external devices. The interface unit 104 may include a power port, a data port (such as a USB port), a memory card port, a port for connecting a device with an identification module, an audio input / output (I / O) port, a video I / O port, etc.

[0035] The storage unit 105 is used to store software programs and various data. The storage unit 105 may mainly include a program storage area and a data storage area. Among them, the program storage area can store an operating system program, a motion control program, application programs (such as a text editor), etc.; the data storage area can store data generated during the use of the robot 100 (such as various sensing data acquired by the sensing unit 103, log file data), etc. In addition, the storage unit 105 may include a high-speed random access memory and may also include a non-volatile memory, such as a disk memory, a flash memory, or other non-volatile solid-state memories.

[0036] The display unit 106 is used to display information input by the user or information provided to the user. The display unit 106 may include a display panel 1061, and the display panel 1061 can be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), etc.

[0037] The input unit 107 can be used to receive input numerical or character information. Specifically, the input unit 107 may include a touch panel 1071 and other input devices 1072. The touch panel 1071, also known as a touch screen, can collect the user's touch operations (such as the user's operations on or near the touch panel 1071 using a palm, a finger, or a suitable accessory), and drive the corresponding connected device according to a preset program. The touch panel 1071 may include two parts: a touch detection device 1073 and a touch controller 1074. Among them, the touch detection device 1073 detects the user's touch orientation and detects the signal brought by the touch operation, and transmits the signal to the touch controller 1074; the touch controller 1074 receives the touch information from the touch detection device 1073, converts it into contact coordinates, and then sends it to the control module 110, and can receive and execute the commands sent by the control module 110. In addition to the touch panel 1071, the input unit 107 may also include other input devices 1072. Specifically, the other input devices 1072 may include, but are not limited to, one or more of a remote control operation handle, etc., and are not specifically limited here.

[0038] Further, the touch panel 1071 can cover the display panel 1061. After the touch panel 1071 detects a touch operation on or near it, it transmits the operation to the control module 110 to determine the type of touch event. Subsequently, the control module 110 provides a corresponding visual output on the display panel 1061 according to the type of touch event. Although in Figure 1 the touch panel 1071 and the display panel 1061 are implemented as two separate components to separately implement the input and output functions, in some embodiments, the touch panel 1071 and the display panel 1061 can be integrated to implement the input and output functions, and specific details are not limited here.

[0039] The control module 110 is the control center of the robot 100. It uses various interfaces and circuits to connect all components of the entire robot 100. By running or executing software programs stored in the storage unit 105 and calling data stored in the storage unit 105, it can thus perform overall control of the robot 100.

[0040] The power supply 111 is used to supply power to each component. The power supply 111 can include a battery and a power control board. The power control board is used to control functions such as battery charging, discharging, and power consumption management. In Figure 1 the illustrated embodiment, the power supply 111 is electrically connected to the control module 110. In other embodiments, the power supply 111 can also be electrically connected to the sensing unit 103 (such as cameras, radars, speakers, etc.) and the motor 1012 respectively. It should be noted that each component can be connected to different power supplies 111 or powered by the same power supply 111.

[0041] Based on the above embodiments, specifically, in some embodiments, the robot 100 can be communicatively connected to a terminal device. When communicating between the terminal device and the robot 100, the terminal device can send command information to the robot 100. The robot 100 can receive the command information through the communication unit 102 and, when the command information is received, transmit the command information to the control module 110, so that the control module 110 can process the command information to obtain a target speed value. The terminal device includes but is not limited to: mobile phones with image capture functions, tablet computers, servers, personal computers, wearable smart devices, and other electrical devices.

[0042] The instruction information may be determined according to preset conditions. In one embodiment, the robot 100 may include a sensor unit 103, and the sensor unit 103 may generate instruction information according to the current environment in which the robot 100 is located. The control module 110 may determine whether the current speed value of the robot 100 meets the corresponding preset conditions according to the instruction information. If satisfied, the current speed value and the current gait of the robot 100 will be maintained; if not satisfied, the target speed value and the corresponding target gait will be determined according to the corresponding preset conditions, so that the robot 100 can be controlled to move with the target speed value and the corresponding target gait. The environmental sensor may include a temperature sensor, an air pressure sensor, a visual sensor, and a sound sensor. The instruction information may include temperature information, air pressure information, image information, and sound information. The communication method between the environmental sensor and the control module 110 may be wired communication or wireless communication. The wireless communication method includes, but is not limited to: wireless network, mobile communication network (3G, 4G, 5G, etc.), Bluetooth or infrared, etc.

[0043] Figure 3 It is a flowchart of a method for establishing a local map provided in one embodiment of the present application.

[0044] like Figure 3 As shown, one aspect of the present application provides a method for establishing a local map, which may include: step 301, obtaining a field of view image corresponding to the environment to be tested acquired by a monocular camera, and a grid map corresponding to the environment to be tested. Step 302, determining a passable area in the environment to be tested based on the field of view image, the passable area including an initial area and an extended area. Step 303, obtaining initial grid information corresponding to the initial area in the grid map, and determining extended grid information corresponding to the extended area based on the initial grid information. Step 304, updating the extended grid information to the grid map, and establishing a local map corresponding to the environment to be tested.

[0045] The following will be combined Figures 4 to 8 The specific implementation of each step in the above-mentioned local map establishment method is described in detail.

[0046] Step 301, obtaining a field of view image corresponding to the environment to be tested, which is captured by a monocular camera, and a grid map corresponding to the environment to be tested.

[0047] Among them, as a category branch of visual SLAM (Simultaneous Localization And Mapping), a monocular camera can complete SLAM with only one camera. The biggest advantage of a monocular camera is its wide detection range and low cost; its disadvantage is that it cannot accurately determine the elevation of each pixel point. In this application, a robot is used as the carrier of the monocular camera, and the monocular camera is used to detect the environment to be measured centered on the robot to obtain the visual image corresponding to the environment to be measured.

[0048] The environment to be measured is used to characterize the activity range of the robot. Specifically, the environment to be measured can be the area covered by the field of view of the monocular camera within the activity range of the robot; in other words, the environment to be measured is the acquisition object of the monocular camera.

[0049] The visual image is the acquisition result of the monocular camera for the detection environment, and it can reflect the pixel coordinates of each pixel point in the pixel coordinate system. However, since the monocular camera cannot accurately determine the elevation of each pixel point, the visual image cannot reflect the elevation of each pixel point.

[0050] The grid map is the result of rasterizing the detection image of the depth sensor and can reflect the grid information corresponding to each grid in the detection image, where the grid information includes the two-dimensional coordinates of each grid and the elevation of each grid. In other words, during the construction of the grid map, it is necessary to use the depth sensor to obtain the detection image and the elevation of each pixel point in its detection range within its detection range. However, the depth sensor has the following several disadvantages: the field of view angle of the depth sensor is limited, and there will be a certain detection blind area when it obtains information about the surrounding environment. For example, some close-range areas cannot be covered; the detection range of the depth sensor is also limited, and farther areas cannot be covered; the depth sensor is also easily affected by external environments such as light, the color or material of the object surface, resulting in the situation that there is no data in some areas of its detection image. It can be imagined that the coverage range of the grid map constructed based on the depth sensor is small, and there are also a certain number of "holes" without grid information. The field of view of the monocular camera is larger than the detection range of the depth sensor, and the cost is smaller and it is less susceptible to external environmental factors. Therefore, adding monocular cameras around the robot can overcome the above defects of the depth sensor.

[0051] Figure 4 It is a schematic diagram of the detection range centered on the robot provided by an embodiment of this application. As Figure 4 shown, at least four depth sensors and at least four monocular cameras are respectively distributed around the robot 200.

[0052] Specifically, a first depth sensor 2011 and a first monocular camera 2021 with the same detection direction are respectively arranged on the four sides of the robot 200, a second depth sensor 2012 and a second monocular camera 2022 with the same detection direction, a third depth sensor 2013 and a third monocular camera 2023 with the same detection direction, and a fourth depth sensor 2014 and a fourth monocular camera 2024 with the same detection direction. The field of view range D21 of the first monocular camera 2021 is greater than the detection range D11 of the first depth sensor 2011. The setting of the first monocular camera 2021 expands the detection range of obstacles in the direction where the first depth sensor is set for the robot 200. The field of view range D22 of the second monocular camera 2022 is greater than the detection range D12 of the second depth sensor 2012. The setting of the second monocular camera 2022 expands the detection range of obstacles in the direction where the second depth sensor is set for the robot 200. The field of view range D23 of the third monocular camera 2023 is greater than the detection range D13 of the third depth sensor 2013. The setting of the third monocular camera 2023 expands the detection range of obstacles in the direction where the third depth sensor is set for the robot 200. The field of view range D24 of the fourth monocular camera 2024 is greater than the detection range D14 of the fourth depth sensor 2014. The setting of the fourth monocular camera 2024 expands the detection range of obstacles in the direction where the fourth depth sensor is set for the robot 200. It should be noted that any number of depth sensors and corresponding monocular cameras can be set at any position of the robot 200 according to the volume of the robot 200, the detection ability of the depth sensor, the detection ability of the monocular camera, or the needs of the user, so that the visual field image of the monocular camera can expand and supplement the grid map.

[0053] Step 302, determine the passage area in the environment to be measured according to the visual field image, where the passage area includes an initial area and an extended area.

[0054] Among them, the passage area is the area corresponding to the set of connection lines between each grounding point in the grounding wire and the monocular camera, and the grounding wire is the intersection line between the obstacle and the ground in the visual field image. The method for determining the passage area is as follows: call the grounding wire recognition model to recognize the visual field image to obtain the image coordinate set corresponding to the grounding wire in the visual field image; convert each image coordinate in the image coordinate set to obtain the grounding wire position of the grounding wire in the environment to be measured; obtain the camera position where the monocular camera is located when the visual field image is collected; determine the passage area in the environment to be measured based on the camera position and the grounding wire position.

[0055] Figure 5 It is a schematic diagram of the passage area provided by an embodiment of the present application. As Figure 5As shown, taking the monocular camera set on one side of the robot as an example, the monocular camera can capture the vision image of the environment to be measured within its field of view by using the principle of rectilinear propagation of light. When there is an obstacle (such as a wall), the light will be blocked by the obstacle, and at this time, the monocular camera will obtain a vision image containing the obstacle.

[0056] Specifically, the vision image containing the obstacle is transmitted to the ground wire recognition model, and the intermediate neurons of the ground wire recognition model process and recognize the vision image, and then output the image coordinate set corresponding to the ground wire. Among them, the ground wire is composed of several grounding points, and the image coordinate set is the set of pixel coordinates of each grounding point.

[0057] Since the ground wire is the intersection line of the obstacle and the ground, the elevation h0 of the ground is the elevation of each grounding point of the ground wire, that is, z w = h0. Substituting the image coordinate set corresponding to the ground wire into the coordinate transformation formula, the three-dimensional coordinates of each grounding point in the world coordinate system can be obtained. Of course, the set of three-dimensional coordinates of each grounding point in the world coordinate system can represent the position of the ground wire in the environment to be measured. In addition, since the position information and attitude information of the robot in the robot coordinate system are used as the transformation medium when obtaining the coordinate transformation formula from the pixel coordinate system to the world coordinate system, and the sampling time of the vision image of the monocular camera is different from the sampling time of the position information and attitude information of the robot, therefore, before using the coordinate transformation formula to perform the transformation between the pixel coordinate system and the world coordinate system, it is also necessary to perform timestamp alignment to avoid errors.

[0058] After obtaining the three-dimensional coordinates of each grounding point in the world coordinate system, determine the camera position of the monocular camera that captures the vision image. Specifically, the camera position where it is located can be obtained through the positioning device of the monocular camera itself; it can also be approximately determined by the positioning device of the carrier (i.e., the robot) where the monocular camera is set.

[0059] According to the camera position of the monocular camera and the position of the grounding wire, the robot's passage area in the test environment can be determined. It can be imagined that the grounding wire is the intersection of the obstacle and the ground, and there is no obstacle on the connection line between each grounding point and the monocular camera. Therefore, the area corresponding to the set of connection lines between each grounding point and the monocular camera is also the passage area that the robot can reach in the setting direction of the monocular camera. Specifically, if there is only one grounding wire, that is, when the obstacle is a continuous whole, it is also possible to only connect the grounding points at the two end points of the grounding wire to the monocular camera, and finally define the area surrounded by the grounding wire, the monocular camera and the two connection lines as the passage area. Of course, the above-mentioned method of determining the passage area only takes a monocular camera as an example, and its passage area only corresponds to the setting direction of the monocular camera. If a robot is equipped with multiple monocular cameras, then the set of passage areas corresponding to each monocular camera is the entire passage area of ​​the robot in the current test environment.

[0060] The pass area includes an initial area and an extended area. The initial area corresponds to the area with grid information in the grid map; the extended area corresponds to the area without grid information in the grid map, that is, the area that cannot be detected by the depth sensor. When dividing the pass area into an initial area and an extended area, the following method can be used: perform area matching on the pass area and the grid map to obtain a matching result; determine the area in the pass area with a successful matching result as the initial area, and determine the area in the pass area with a failed matching result as the extended area.

[0061] Specifically, due to the limitations of the depth sensor, the range of the passage area determined by the monocular camera will inevitably be larger than the range of the corresponding area of ​​the grid map generated by the depth sensor in the test environment. Based on this, the passage area is matched with the grid map, and the area in the passage area that can match the grid information in the grid map is used as the initial area, and the area that cannot match the grid information in the grid map is used as the extended area. In other words, since the field of view of the monocular camera is larger than the detection range of the depth sensor, the area that belongs to the detection blind area of ​​the depth sensor but can be collected by the monocular camera is used as the extended area; and the area that can be detected by both the monocular camera and the depth sensor is the initial area. Of course, since the monocular camera is not easily disturbed by external factors such as light, ground color and material when collecting field of view images, the extended area also includes a supplementary part for the "hole".

[0062] Step 303, obtaining initial grid information corresponding to the initial area in the grid map, and determining extended grid information corresponding to the extended area according to the initial grid information.

[0063] Among them, the grids corresponding to the initial area in the grid map are defined as initial grids, and the initial grid information includes the two-dimensional coordinates and the initial grid elevation of each initial grid; the grids corresponding to the extended area in the grid map are defined as extended grids, and the extended grid information includes the two-dimensional coordinates and the extended grid elevation of each extended grid. It should be noted that since a monocular camera cannot determine the elevation of each pixel point in the passage area, the extended grids do not have extended grid information, and it is necessary to use the initial grid information to supplement the data of the extended grid information.

[0064] The method for obtaining the initial grid information is as follows: perform grid matching on the initial area and the grid map to obtain a plurality of initial grids corresponding to the initial area in the grid map; obtain the initial grid information corresponding to each initial grid in the grid map, and the initial grid information includes the initial grid elevation corresponding to the initial grid.

[0065] The method for determining the extended grid information corresponding to the extended area according to the initial grid information includes: determining a plurality of extended grids corresponding to the extended area in the grid map; determining at least one initial grid adjacent to the extended grid among the plurality of initial grids as the neighborhood grid corresponding to the extended grid; determining the extended grid information corresponding to the extended grid according to the initial grid information corresponding to the neighborhood grid, and the extended grid information includes the extended grid elevation.

[0066] Determine a plurality of extended grids corresponding to the extended area in the grid map. Specifically, map the extended area to the grid map, and determine a plurality of grids corresponding to the extended area in the grid map as the extended grids. At this time, the extended grids do not have extended grid information.

[0067] Determine at least one initial grid adjacent to the extended grid among the plurality of initial grids as the neighborhood grid corresponding to the extended grid. Specifically, obtain the grid type corresponding to each extended grid, where the grid type includes a hole type and an extension type; obtain the neighborhood range associated with the grid type, and the neighborhood range includes the neighborhood direction and the number of grids; among the plurality of initial grids, determine the initial grids matching the number of grids along the neighborhood direction as the neighborhood grids corresponding to the extended grids.

[0068] Figure 6 It is a schematic diagram of an extended grid of the hole type provided by an embodiment of the present application. As Figure 6 shown, the extended grid of the hole type is generated corresponding to the influence of external environmental factors on the depth sensor, so it is usually located between a plurality of initial grids, that is, surrounded by a plurality of initial grids.

[0069] Figure 7 It is a schematic diagram of an extended grid of the extension type provided by an embodiment of the present application. As Figure 7As shown, the extended grid of the extended type is generated in response to the problem of the small detection range of the depth sensor. Therefore, it is concentrated on one side of multiple initial grids and is an extension of the initial grids.

[0070] Under normal circumstances, according to the initial grid information and the two-dimensional coordinates of the extended grid, the position of the extended grid relative to each initial grid can be determined. Furthermore, the grid type of the extended grid can be determined. The grid type includes the hole type and the extended type.

[0071] For the extended grid of the hole type, the neighborhood direction can be the horizontal direction, the vertical direction, or the diagonal direction passing through the extended grid of the hole type, or any straight line direction can be used as the neighborhood direction according to requirements. The number of grids can be any number that can ensure the accuracy of the elevation of the extended grid. For example, with the extended grid of the hole type as the center, four neighborhood grids are selected on each side along the neighborhood direction. At this time, the number of grids is 8.

[0072] For the extended grid of the extended type, its neighborhood direction can be the horizontal direction, the vertical direction, the diagonal direction of the boundary grid, or the circumferential direction centered on the direct neighborhood grid. The number of grids can be any number that can ensure the accuracy of the elevation of the extended grid, and there is no limit here.

[0073] After determining the neighborhood direction and the number of grids, the initial grids that match the number of grids can be determined along the neighborhood direction among multiple initial grids as the neighborhood grids corresponding to the extended grid.

[0074] In some embodiments, after determining the neighborhood grids, the method for determining the extended grid information can be as follows: using the initial grid information to determine the two-dimensional coordinates of the extended grid; respectively constructing fitting lines corresponding to at least two neighborhood directions according to the initial grid information corresponding to the neighborhood grids; respectively calculating multiple fitting values representing the elevation of the extended grid by using the fitting lines corresponding to at least two neighborhood directions according to the two-dimensional coordinates of the extended grid; obtaining the average value of the multiple fitting values as the extended grid elevation of the extended grid; and integrating the two-dimensional coordinates and the extended grid elevation of the extended grid to determine the extended grid information.

[0075] The following is an elaboration on obtaining the extended grid information of two types of extended grids respectively by using the above method for determining the extended grid information.

[0076] For the extended grid of the hole type:

[0077] Using the initial grid information, determine the two-dimensional coordinates of the extended grid of the hole type in the grid map. For example, when the two-dimensional coordinates of multiple initial grids adjacent to the extended grid of the hole type are (m, n), (m+1, n-1), (m+1, n+1), and (m+2, n) respectively, and the grid corresponding to the coordinates (m+1, n) has no initial grid information, then this grid is the extended grid of the hole type, and (m+1, n) is the two-dimensional coordinates of the extended grid of this hole type.

[0078] After determining the two-dimensional coordinates of the extended grid of the hole type, use the obstacle recognition model to judge whether there are pits or obstacles in the extended grid of the hole type. Specifically, since the depth sensor is extremely susceptible to light, ground color, and material, "holes" with missing elevation in the detection area will be generated. For example, when the ground material reflects light, the depth sensor detects that the elevation at this position is infinite, and the elevation is invalid at this time; when the ground color is darker, the depth sensor misidentifies this position as a pit, and it detects that the elevation at this position is infinitely small, and the elevation is also invalid at this time. Of course, it does not rule out the situation where there are indeed pits or obstacles in the "hole". Based on this, map the extended grid of the hole type to the field of view image, determine the "hole" position of the extended grid of the hole type in the field of view image, and then use the obstacle recognition model to determine whether there are pits or obstacles at the extended grid of the hole type. If there are pits or obstacles at this place, there is no need to fill in the elevation of the extended grid, and directly mark this place as a pit or an obstacle to prompt the robot to detour. If there are no pits or obstacles at this place, it is considered that the depth sensor has limitations in collecting the elevation at this place, and the elevation of the extended grid needs to be filled. It should be noted that the obstacle recognition model is a model that uses a neural network to identify objects in an image. The field of view image is transmitted to the obstacle recognition model, and the intermediate neurons of the obstacle recognition model analyze and identify the "hole" position of the field of view image, and then output the recognition result of the "hole" position, including results such as pits, obstacles, no pits, or no obstacles. Since the obstacle recognition model is obtained through a large number of sample trainings, the results it outputs are credible.

[0079] According to the initial grid information corresponding to the neighborhood grid, at least two fitting lines corresponding to the neighborhood directions are constructed respectively. When filling the elevation of the extended grid of the hole type, taking the extended grid of the hole type as the center, a plurality of initial grids are respectively selected as neighborhood grids in at least two neighborhood directions of the extended grid of the hole type. For example, when taking the horizontal direction of the extended grid of the hole type as the neighborhood direction, taking the extended grid of the hole type as the center, 4 initial grids are respectively selected on the left and right of the extended grid of the hole type as horizontal neighborhood grids; when taking the vertical direction of the extended grid of the hole type as the neighborhood direction, taking the extended grid of the hole type as the center, 4 initial grids are respectively selected above and below the extended grid of the hole type as vertical neighborhood grids. The initial grid information of each neighborhood grid is respectively extracted, and according to the initial grid information of the neighborhood grids in each neighborhood direction, the fitting lines in each neighborhood direction are constructed.

[0080] Specifically, the line to be fitted can be expressed as: ax w +by w +cz w +d = 0, where a, b, c, and d are all linear parameters. According to different fitting situations, the linear parameters have different values. For example, when performing linear fitting in the vertical direction, the initial grid information of multiple vertical neighborhood grids is respectively substituted into the line to be fitted to determine the specific values of each linear parameter, and finally the first fitting line representing the fitting situation in the vertical direction is obtained: a1x w +b1y w +c1z w +d1 = 0 (1). In formula (1), a1, b1, c1, and d1 are all determined values of the linear parameters in the vertical direction. Similarly, when performing linear fitting in the horizontal direction, the initial grid information of multiple horizontal neighborhood grids selected in the horizontal direction is respectively substituted into the line to be fitted to determine the specific values of each linear parameter, and finally the second fitting line representing the fitting situation in the horizontal direction is obtained: a2x w +b2y w +c2z w +d2 = 0 (2). In formula (2), a2, b2, c2, and d2 are all determined values of the linear parameters in the horizontal direction.

[0081] According to the two-dimensional coordinates of the extended grid, multiple fitting values representing the elevation of the extended grid are respectively calculated by using the fitting lines corresponding to at least two neighborhood directions. For example, substituting the two-dimensional coordinates (x w , y w ) of the extended grid of the hole type into formula (1) and formula (2) respectively, the first fitting value and the second fitting value are respectively obtained, where the first fitting value is obtained from formula (1) and the second fitting value is obtained from formula (2).

[0082] Calculate the average of multiple fitted values as the extended grid elevation of the extended grid. After obtaining the first fitted value and the second fitted value, use the average of the two as the extended grid elevation of the extended grid of this hole type. The extended grid elevation obtained in this way has higher credibility.

[0083] Integrate the two-dimensional coordinates of the extended grid and the extended grid elevation to determine the extended grid information. In the above manner, traverse the extended grids of all hole types to obtain the extended grid elevations of the extended grids of each hole type. Finally, integrate the two-dimensional coordinates of the extended grid of the hole type and its extended grid elevation to obtain the extended grid information of the extended grid of the hole type.

[0084] For the extended grid of the extension type:

[0085] According to the initial grid information, determine the two-dimensional coordinates of the extended grid of the extension type in the grid map. For example, when the two-dimensional coordinates of the initial grid adjacent to the extended grid of the extension type are (m, n) respectively, and the extended grid of the extension type is directly above the adjacent initial grid, the two-dimensional coordinates of the extended grid of the extension type can be (m, n + 1), then the two-dimensional coordinates of the adjacent extended grid in the same column can be (m, n + 2), and so on. In the above manner, the two-dimensional coordinates of each extended grid of the extension type can be deduced.

[0086] Determine the neighborhood range including the neighborhood grid, neighborhood direction, and the number of neighborhood grids according to the requirements. Use the initial grid information corresponding to the neighborhood grid to construct at least two fitting lines corresponding to the neighborhood directions respectively. According to the two-dimensional coordinates of the extended grid of the extension type, calculate multiple fitted values representing the extended grid elevation of the extended grid of the extension type by using the fitting lines corresponding to at least two neighborhood directions respectively. Calculate the average of the multiple fitted values as the extended grid elevation of the extended grid of the extension type. In the above manner, traverse all the extended grids of the extension type to obtain the extended grid elevations of the extended grids of each extension type. Finally, integrate the two-dimensional coordinates of the extended grid of the extension type and its extended grid elevation to obtain the extended grid information of the extended grid of the extension type. For a more detailed implementation method, reference can be made to the process of obtaining the extended grid information of the extended grid of the hole type, which will not be elaborated here.

[0087] In some other embodiments, after determining the neighborhood grid, the method for determining the extended grid information can also be: based on the initial grid information, determine the two-dimensional coordinates of the extended grid; extract the neighborhood grid elevation from the initial grid information corresponding to the neighborhood grid, determine the average of the multiple neighborhood grid elevations as the extended grid elevation of the extended grid; integrate the two-dimensional coordinates of the extended grid and the extended grid elevation to determine the extended grid information corresponding to the extended grid.

[0088] The following is an elaboration on obtaining the extended grid information of two types of extended grids respectively by using the above-mentioned method for determining the extended grid information.

[0089] For the extended grid of the extension type:

[0090] Based on the initial grid information, determine the two-dimensional coordinates of the extended grid. For example, when the two-dimensional coordinates of the initial grid adjacent to the extended grid of the extension type are (m, n) respectively, and the extended grid of the extension type is directly above its adjacent initial grid, then the two-dimensional coordinates of the extended grid of the extension type can be (m, n + 1), and the two-dimensional coordinates of the adjacent extended grid in the same column can be (m, n + 2), and so on. Through the above method, the two-dimensional coordinates of each extended grid of the extension type can be deduced.

[0091] Extract the elevation of the neighborhood grid from the initial grid information corresponding to the neighborhood grid, and determine the average value of the elevations of multiple neighborhood grids as the extended grid elevation of the extended grid. Specifically, the grid directly adjacent to the initial grid among multiple extended grids of the extension type is used as the boundary grid, at least one initial grid adjacent to the boundary grid is used as the direct neighborhood grid, and other initial grids in the same neighborhood direction as the direct neighborhood grid are used as the indirect neighborhood grids. Calculate the average value of the extended grid elevations of multiple neighborhood grids as the extended grid elevation of the boundary grid. Then synchronously cover the extended grid elevation of the boundary grid to other extended grids of the extension type that have the same abscissa value or ordinate value as it, so that each extended grid of the extension type has an extended grid elevation.

[0092] For example Figure 7 As shown, in the extended grid of the extension type, the extended grid G1 is the boundary grid directly adjacent to the initial grid, and its extended grid elevation is unknown. The initial grid G0 is adjacent to the extended grid G1, so the initial grid G0 is defined as the direct neighborhood grid of the extended grid G1. The circumferential direction centered on the direct neighborhood grid is used as the neighborhood direction, and other initial grids adjacent to the initial grid G0 are determined as the indirect neighborhood grids of the extended grid G1. Among them, the direct neighborhood grid and the indirect neighborhood grid are collectively referred to as the neighborhood grid. In the case of no pits or obstacles, the elevations of adjacent grids are similar. Calculate the average value of the neighborhood grid elevation of the initial grid G0 and the neighborhood grid elevations of multiple indirect neighborhood grids, and use this as the extended grid elevation of the extended grid G1. In addition, among multiple extended grids of the extension type, the two-dimensional coordinates of the extended grid G1, the extended grid G2, and the extended grid G3 have the same column value or row value, and the extended grid G2 and the extended grid G3 are both in the robot's passing area. Therefore, the extended grid elevation of the extended grid G1 can be used to cover the extended grid elevations of the extended grid G2 and the extended grid G3.

[0093] Integrate the two-dimensional coordinates of the extended grid and the extended grid elevation to determine the extended grid information corresponding to the extended grid. Traverse the extended grids of each extension type in the same way, so as to complete the purpose of supplementing the elevation of the extended grids of the extension type using the initial grid information. After obtaining the two-dimensional coordinates of the extended grid of the extension type and the extended grid elevation, integrate the two-dimensional coordinates of the extended grid of the extension type with the extended grid elevation to generate the extended grid information of the extended grid of the extension type.

[0094] For the extended grid of the hole type:

[0095] Based on the initial grid information, determine the two-dimensional coordinates of the extended grid. For example, when the two-dimensional coordinates of multiple initial grids adjacent to the extended grid of the hole type are (m, n), (m + 1, n - 1), (m + 1, n + 1), (m + 2, n) respectively, and the grid corresponding to the coordinate (m + 1, n) has no initial grid information, then this grid is the extended grid of the hole type, and (m + 1, n) is the two-dimensional coordinate of the extended grid of the hole type.

[0096] Extract the neighborhood grid elevation from the initial grid information corresponding to the neighborhood grid, and determine the average value of the multiple neighborhood grid elevations as the extended grid elevation of the extended grid. Specifically, in the case of no pit or obstacle, the elevations of adjacent grids are similar. Therefore, after obtaining the two-dimensional coordinates of the extended grid, extract at least one initial grid adjacent to the extended grid of the hole type in the preset neighborhood direction as the neighborhood grid, and then calculate the average value of the neighborhood grid elevations of each neighborhood grid as the extended grid elevation of the extended grid of the hole type. Of course, the extended grid elevation of the extended grid of the hole type directly adjacent to the initial grid can cover other extended grids with the same abscissa value or ordinate value adjacent to it and having the same two-dimensional coordinates. Finally, integrate the two-dimensional coordinates of the extended grid of the hole type with the extended grid elevation to generate the extended grid information of the extended grid of the hole type.

[0097] Correspondingly update the extended grid information of the extended grid of the hole type and the extended grid information of the extended grid of the extension type to the grid map, and a local map centered on the robot can be established. Of course, as the position information and attitude information of the robot change, the initial grid elevation of the initial grid and the extended grid elevation of the extended grid can also be updated respectively, and the local map can be updated based on the update results of the initial grid elevation and the extended grid elevation.

[0098] The local map can be in 2D form or 2.5D form. When the local map is in 2D form, the elevations of each grid can be distinguished by colors or markings; when the local map is in 2.5D form, the elevations of each grid can be characterized by the heights of the three-dimensional columns, which are not restricted here.

[0099] Figure 8 is an application scenario diagram of a local map centered on a robot provided by an embodiment of the present application. Figure 8 As shown, in a 2D or 2.5D grid map, when the robot P is at position A, the display range of the local map is D A , when the robot P moves to position B according to the motion route, the display range of the local map is D B Obviously, as the robot moves, the display range of the local map changes accordingly to ensure that the robot understands the surrounding environment information and achieves the effect of accurate obstacle avoidance.

[0100] In some embodiments, in step 301, before obtaining the field of view image corresponding to the environment to be measured captured by the monocular camera and the grid map corresponding to the environment to be measured, it also includes: using a depth sensor to establish a grid map.

[0101] Specifically, the robot's built-in positioning system and inertial measurement unit are called to determine the robot's position information and posture information; the depth sensor is used to obtain the initial point cloud data within its detection range; based on the robot's position information and posture information, the initial point cloud data of the depth sensor is converted into three-dimensional coordinates in the world coordinate system; based on the three-dimensional coordinates of each pixel point within the detection range of the depth sensor, a grid map of the environment to be tested is constructed or updated.

[0102] Among them, the robot's built-in positioning system and inertial measurement unit are called to determine the robot's position information and attitude information. Specifically, the inertial measurement unit is a device for measuring the robot's three-axis attitude angle and acceleration, including three single-axis accelerometers and three single-axis gyroscopes. The accelerometer is used to detect the robot's independent three-axis acceleration signal in the robot coordinate system, and the gyroscope detects the robot's angular velocity signal relative to the world coordinate system, and measures the robot's angle and acceleration in three-dimensional space. The inertial measurement unit can be set at the robot's center of mass.

[0103] The depth sensor is used to obtain the initial point cloud data within its detection range. Specifically, the depth sensor collects a depth image within its detection range. The depth image includes a detection image with RGB three channels and depth data. The pixels of the detection image and the pixels of the depth data correspond one to one according to the position relationship. The detection image provides the pixel coordinates of each pixel within the detection range of the depth sensor in the pixel coordinate system, including row values ​​and column values. The depth data is used to characterize the distance from the highest point of each pixel within the detection range to the ground along the vertical direction, that is, the elevation of each pixel. The pixel coordinates of each pixel and the elevation corresponding to each pixel are integrated to obtain the initial point cloud data used to characterize the information of each pixel within the detection range of the depth sensor.

[0104] Convert the initial point cloud data of the depth sensor into three-dimensional coordinates in the world coordinate system according to the position information and attitude information of the robot. Specifically, use the current position information and attitude information of the robot to obtain the transformation relationship between the robot coordinate system and the world coordinate system. Among them, the transformation relationship between the robot coordinate system and the world coordinate system is given by the built-in odometer of the robot. According to the transformation relationship from the camera coordinate system to the robot coordinate system in the offline calibration result of the depth sensor and the transformation relationship between the robot coordinate system and the world coordinate system, the transformation relationship formula from the world coordinate system to the camera coordinate system can be obtained. When the origins of the image coordinate system and the camera coordinate system coincide, using the principle of similar triangles, the transformation relationship formula between the camera coordinate system and the pixel coordinate system can be obtained. Finally, combined with the calibration parameters of the depth sensor, according to the transformation relationship formula from the world coordinate system to the camera coordinate system and the transformation relationship formula between the camera coordinate system and the pixel coordinate system, using the camera coordinate system as an intermediary, the coordinate transformation formula between the world coordinate system and the pixel coordinate system can be obtained. It should be noted that since the sampling time of the initial point cloud data does not exactly correspond to the sampling time of the current position information and attitude information of the robot, in order to avoid errors, timestamp alignment is also required before using the coordinate transformation formula to obtain the three-dimensional coordinates of the initial point cloud data in the world coordinate system.

[0105] The transformation relationship between the camera coordinate system and the world coordinate system can be specifically expressed as:

[0106]

[0107] In formula (3), s is the scale factor, u is the column value in the pixel coordinates, v is the row value in the pixel coordinates, dX is the physical size of each pixel point on the X-axis of the image coordinate system, dY is the physical size of each pixel point on the Y-axis of the image coordinate system, f is the camera focal length, u0, v0, a x 、a y are all known internal parameters of the depth sensor, R is the rotation matrix, t is the translation matrix, x w is the abscissa value of the three-dimensional coordinates in the world coordinate system, y w is the ordinate value of the three-dimensional coordinates in the world coordinate system, z w is the vertical coordinate value (i.e., elevation) of the three-dimensional coordinates in the world coordinate system, M1 is the internal parameter matrix of the depth sensor, M2 is the external parameter matrix of the depth sensor, and M is the projection matrix of the depth sensor.

[0108] By combining the internal parameter matrix and the external parameter matrix of the depth sensor in formula (3), the coordinate transformation formula can be obtained:

[0109]

[0110] In formula (4), M is the projection matrix.

[0111] Using the coordinate transformation formula, the pixel coordinates and elevation of each pixel point collected by the depth sensor are converted into three-dimensional coordinates in the world coordinate system, where the three-dimensional coordinates include the two-dimensional coordinates and elevation of each pixel point within the detection range of the depth sensor.

[0112] According to the three-dimensional coordinates of each pixel point within the detection range of the depth sensor, a grid map of the environment to be measured is constructed or updated. Specifically, the detection image corresponding to the environment to be measured is divided into multiple grids, and each grid contains multiple pixel points. The elevations of the pixel points in each grid are numerically sorted, and the elevation value of the grid is taken as the maximum elevation value in each grid; the two-dimensional coordinates of the pixel point corresponding to the center of each grid are taken as the two-dimensional coordinates of the grid. Based on the above, the grid information including the two-dimensional coordinates and elevation of each grid is synchronized or updated to the map corresponding to the environment to be measured, and a grid map capable of representing the grid information of each grid is obtained. It should be noted that due to the depth sensor being susceptible to external environmental factors, there are a certain number of "holes" in the grid map.

[0113] In some embodiments, since the depth sensor is mounted around the robot, as the robot moves, the detection range of the depth sensor will also change. Based on this, the initial point cloud data collected by the depth sensor will be updated as its detection range changes, and the grid map corresponding to the initial point cloud data will also be updated accordingly to ensure that the grid map is always centered on the robot.

[0114] According to a method for establishing a local map of the present application, using the vision image of a monocular camera, an extended area is extended in the initial area of the passage area, and combined with the initial grid information of the grid map, the extended grid information of the extended area is supplemented, which improves the accuracy of establishing a complete local map and also reduces the cost of establishing the local map.

[0115] Figure 9 It is a block diagram of a device for establishing a local map provided by an embodiment of the present application.

[0116] As Figure 9As shown in the figure, another aspect of the present application provides a device 300 for establishing a local map, which may include: a collection module 310, a passage area determination module 320, an extended grid information determination module 330, and a local map establishment module 340. The collection module 310 is used to obtain the visual field image corresponding to the to-be-measured environment collected by the monocular camera, and the grid map corresponding to the to-be-measured environment. The passage area determination module 320 is used to determine the passage area in the to-be-measured environment according to the visual field image, and the passage area includes an initial area and an extended area. The extended grid information determination module 330 is used to obtain the initial grid information corresponding to the initial area in the grid map, and determine the extended grid information corresponding to the extended area according to the initial grid information. The local map establishment module 340 is used to update the extended grid information to the grid map and establish the local map corresponding to the to-be-measured environment.

[0117] The execution steps of the passage area determination module 320 may include: calling the grounding wire recognition model to recognize the visual field image to obtain the image coordinate set corresponding to the grounding wire in the visual field image; converting each image coordinate in the image coordinate set to obtain the grounding wire position of the grounding wire in the to-be-measured environment; obtaining the camera position where the monocular camera is located when collecting the visual field image; determining the passage area in the to-be-measured environment based on the camera position and the grounding wire position; performing area matching between the passage area and the grid map to obtain a matching result; determining the area with a successful matching result in the passage area as the initial area, and determining the area with a failed matching result in the passage area as the extended area.

[0118] The execution steps of the passage area determination module 320 may further include: performing grid matching between the initial area and the grid map to obtain a plurality of initial grids corresponding to the initial area in the grid map; obtaining the initial grid information corresponding to each initial grid in the grid map, and the initial grid information includes the initial grid elevation corresponding to the initial grid.

[0119] The execution steps of the passage area determination module 320 may further include: determining a plurality of extended grids corresponding to the extended area in the grid map; determining at least one initial grid adjacent to the extended grid among the plurality of initial grids as the neighborhood grid corresponding to the extended grid; determining the extended grid information corresponding to the extended grid according to the initial grid information corresponding to the neighborhood grid, and the extended grid information includes the extended grid elevation.

[0120] The execution steps of the passage area determination module 320 may further include: obtaining the grid type corresponding to each extended grid, where the grid type includes a hole type and an extension type; obtaining the neighborhood range associated with the grid type, and the neighborhood range includes a neighborhood direction and the number of grids; determining, among the plurality of initial grids, the initial grids matching the number of grids along the neighborhood direction as the neighborhood grids corresponding to the extended grid.

[0121] The execution steps of the passage area determination module 320 may further include: using the initial grid information to determine the two-dimensional coordinates of the extended grid; respectively constructing fitting lines corresponding to at least two neighborhood directions according to the initial grid information corresponding to the neighborhood grids; respectively calculating multiple fitting values representing the elevation of the extended grid by using the fitting lines corresponding to at least two neighborhood directions according to the two-dimensional coordinates of the extended grid; obtaining the average value of the multiple fitting values as the extended grid elevation of the extended grid; and integrating the two-dimensional coordinates and the extended grid elevation of the extended grid to determine the extended grid information.

[0122] The execution steps of the passage area determination module 320 may further include: based on the initial grid information, determining the two-dimensional coordinates of the extended grid; extracting the neighborhood grid elevations from the initial grid information corresponding to the neighborhood grids, and determining the average value of the multiple neighborhood grid elevations as the extended grid elevation of the extended grid; integrating the two-dimensional coordinates and the extended grid elevation of the extended grid to determine the extended grid information corresponding to the extended grid.

[0123] According to a local map establishment device of the present application, using the vision image of a monocular camera, an extended area is extended in the initial area of the passage area, and in combination with the initial grid information of the grid map, the extended grid information of the extended area is supplemented, which improves the accuracy of establishing a complete local map and also reduces the establishment cost of the local map.

[0124] Figure 10 It is a schematic structural diagram of an electronic device provided by an embodiment of the present application. As Figure 10 shown, according to another aspect of the present application, an electronic device 400 is further provided. The electronic device 400 may include one or more processors and one or more memories. Among them, computer-readable code is stored in the memory, and when the computer-readable code is run by one or more processors, it can execute the local map establishment method as described above.

[0125] The method or device according to the embodiment of the present application can also be implemented by means of Figure 10 the architecture of the electronic device shown. As Figure 10As shown, the electronic device 400 may include a bus 401, one or more CPUs 402, a read-only memory (ROM) 403, a random access memory (RAM) 404, a communication port 405 connected to a network, an input / output component 406, a hard disk 407, etc. The storage device in the electronic device 400, such as the ROM 403 or the hard disk 407, may store the method for establishing a local map provided in this application. The method for establishing a local map may, for example, include: obtaining a field-of-view image corresponding to the environment to be measured collected by a monocular camera, and a grid map corresponding to the environment to be measured. Determining a passage area in the environment to be measured according to the field-of-view image, where the passage area includes an initial area and an extended area. Obtaining initial grid information corresponding to the initial area in the grid map, and determining extended grid information corresponding to the extended area according to the initial grid information. Updating the extended grid information to the grid map to establish a local map corresponding to the environment to be measured. Further, the electronic device 400 may also include a user interface 408. Of course, Figure 10 the architecture shown is only exemplary, and when implementing different devices, one or more components in the electronic device shown may be omitted according to actual needs. Figure 10

[0126] Figure 11 This is a schematic structural diagram of a computer-readable storage medium provided by an embodiment of this application. As Figure 11 shown, it is a computer-readable storage medium 500 according to an embodiment of this application. Computer-readable instructions are stored on the computer-readable storage medium 500. When the computer-readable instructions are run by a processor, the method for establishing a local map according to the embodiment of this application described with reference to the above drawings can be executed. The storage medium 500 includes, but is not limited to, for example, volatile memory and / or non-volatile memory. Volatile memory may, for example, include random access memory (RAM) and cache memory, etc. Non-volatile memory may, for example, include read-only memory (ROM), hard disk, flash memory, etc.

[0127] In addition, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the present application provides a non-transitory machine-readable storage medium storing machine-readable instructions that can be run by a processor to execute instructions corresponding to the method steps provided by the present application. For example: obtaining a field-of-view image corresponding to a to-be-measured environment collected by a monocular camera, and a grid map corresponding to the to-be-measured environment. Determining a passage area in the to-be-measured environment according to the field-of-view image, where the passage area includes an initial area and an extended area. Obtaining initial grid information corresponding to the initial area in the grid map, and determining extended grid information corresponding to the extended area according to the initial grid information. Updating the extended grid information to the grid map to establish a local map corresponding to the to-be-measured environment. When this computer program is executed by a central processing unit (CPU), the above functions defined in the method of the present application are executed.

[0128] The method, apparatus, and device of the present application can be implemented in many ways. For example, the method, apparatus, and device of the present application can be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is only for illustration, and the steps of the method of the present application are not limited to the specific order described above, unless otherwise specifically stated. In addition, in some embodiments, the present application can also be implemented as a program recorded in a recording medium, and these programs include machine-readable instructions for implementing the method according to the present application. Therefore, the present application also covers a recording medium storing a program for executing the method according to the present application.

[0129] In addition, in the above technical solutions provided by the embodiments of the present application, parts that are the same as the corresponding technical solutions in the prior art in terms of implementation principles are not described in detail to avoid excessive elaboration.

[0130] The above description is only for the embodiments of the present application and the description of the technical principles applied. Those skilled in the art should understand that the protection scope involved in the present application is not limited to the technical solutions formed by the specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the technical concept. For example, technical solutions formed by mutually replacing the above features with (but not limited to) technical features having similar functions disclosed in the present application.

Claims

1. A method for establishing a local map, characterized in that The method includes: Obtaining a field of view image corresponding to the environment to be measured collected by a monocular camera, and a grid map corresponding to the environment to be measured; Determining a passage area in the environment to be measured according to the field of view image, where the passage area includes an initial area and an extended area; Obtaining initial grid information corresponding to the initial area in the grid map, and determining extended grid information corresponding to the extended area according to the initial grid information; Updating the extended grid information to the grid map, and establishing a local map corresponding to the environment to be measured; The determining the extended grid information corresponding to the extended area according to the initial grid information includes: Determining a plurality of extended grids corresponding to the extended area in the grid map; Determining at least one initial grid adjacent to the extended grid among the plurality of initial grids as a neighborhood grid corresponding to the extended grid; Determining the extended grid information corresponding to the extended grid according to the initial grid information corresponding to the neighborhood grid, where the extended grid information includes an extended grid elevation; The determining the extended grid information corresponding to the extended grid according to the initial grid information corresponding to the neighborhood grid, where the extended grid information includes an extended grid elevation, includes: Using the initial grid information to determine the two-dimensional coordinates of the extended grid; Respectively constructing at least two fitting lines corresponding to the neighborhood directions according to the initial grid information corresponding to the neighborhood grid; According to the two-dimensional coordinates of the extended grid, respectively calculating a plurality of fitting values representing the extended grid elevation by using at least two fitting lines corresponding to the neighborhood directions; Obtaining the average value of the plurality of fitting values as the extended grid elevation of the extended grid; and Integrating the two-dimensional coordinates of the extended grid and the extended grid elevation to determine the extended grid information.

2. The method for establishing a local map according to claim 1, characterized in that, The determining the passage area in the environment to be measured according to the field of view image includes: Invoking a ground wire recognition model to recognize the field of view image, and obtaining an image coordinate set corresponding to the ground wire in the field of view image; Converting each image coordinate of the image coordinate set to obtain the ground wire position of the ground wire in the environment to be measured; Obtaining the camera position of the monocular camera when the field of view image is collected; Determining the passage area in the environment to be measured based on the camera position and the ground wire position; Performing area matching between the passage area and the grid map to obtain a matching result; Determining the area with a successful matching result in the passage area as the initial area, and determining the area with a failed matching result in the passage area as the extended area.

3. The method for establishing a local map according to claim 1, characterized in that, The obtaining the initial grid information corresponding to the initial area in the grid map includes: Performing grid matching between the initial area and the grid map to obtain a plurality of initial grids corresponding to the initial area in the grid map; Obtaining the initial grid information corresponding to each of the initial grids in the grid map, where the initial grid information includes the initial grid elevation corresponding to the initial grid.

4. The method for establishing a local map according to claim 1, characterized in that, Determining at least one initial grid adjacent to the extended grid among the multiple initial grids as the neighborhood grid corresponding to the extended grid includes: Obtaining the grid type corresponding to each of the extended grids, where the grid type includes a hole type and an extension type; Obtaining the neighborhood range associated with the grid type, where the neighborhood range includes a neighborhood direction and the number of grids; Among the multiple initial grids, determining the initial grids that match the number of grids along the neighborhood direction as the neighborhood grids corresponding to the extended grid.

5. The method for establishing a local map according to claim 1, wherein Determining the extended grid information corresponding to the extended grid according to the initial grid information corresponding to the neighborhood grid, where the extended grid information includes the extended grid elevation, includes: Based on the initial grid information, determining the two-dimensional coordinates of the extended grid; Extracting the neighborhood grid elevations from the initial grid information corresponding to the neighborhood grid, and determining the average value of the multiple neighborhood grid elevations as the extended grid elevation of the extended grid; Integrating the two-dimensional coordinates of the extended grid and the extended grid elevation to determine the extended grid information corresponding to the extended grid.

6. An apparatus for establishing a local map, characterized in that, The apparatus includes: An acquisition module, configured to acquire a field of view image corresponding to a to-be-measured environment collected by a monocular camera, and a grid map corresponding to the to-be-measured environment; A passage area determination module, configured to determine a passage area in the to-be-measured environment according to the field of view image, where the passage area includes an initial area and an extended area; An extended grid information determination module, configured to obtain the initial grid information corresponding to the initial area in the grid map, and determine the extended grid information corresponding to the extended area according to the initial grid information; and A local map establishment module, configured to update the extended grid information to the grid map to establish a local map corresponding to the to-be-measured environment; Determining the extended grid information corresponding to the extended area according to the initial grid information includes: Determining a plurality of extended grids corresponding to the extended area in the grid map; Determining at least one initial grid adjacent to the extended grid among the multiple initial grids as the neighborhood grid corresponding to the extended grid; According to the initial grid information corresponding to the neighborhood grid, determining the extended grid information corresponding to the extended grid, where the extended grid information includes the extended grid elevation; Determining the extended grid information corresponding to the extended grid according to the initial grid information corresponding to the neighborhood grid, where the extended grid information includes the extended grid elevation, includes: Using the initial grid information to determine the two-dimensional coordinates of the extended grid; According to the initial grid information corresponding to the neighborhood grid, respectively constructing fitting lines corresponding to at least two of the neighborhood directions; According to the two-dimensional coordinates of the extended grid, respectively calculating a plurality of fitting values representing the extended grid elevation by using the fitting lines corresponding to at least two of the neighborhood directions; Obtaining the average value of the multiple fitting values as the extended grid elevation of the extended grid; and Integrating the two-dimensional coordinates of the extended grid and the extended grid elevation to determine the extended grid information.

7. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, it realizes the steps in the method for establishing a local map as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by the processor to execute the steps in the method for establishing a local map as described in any one of claims 1-5.

Images