Electronic apparatus for scheduling and path planning of moving object, and moving object

Abstract

The invention relates to an electronic apparatus and a moving object. The electronic device includes: a memory in which computer program codes are stored; and a processing circuit configured to run the code and execute the following steps: acquiring image information of a plurality of frames of images from an image sensor mounted on a moving body; acquiring motion information of a motion sensor of the moving body, wherein the motion information and the image information of the multiple frames of images are associated with each other through timestamps; fusing the motion information with the image information to obtain depth information which can be applied to each frame of image in the multiple frames of images; and obtaining scene map information of a scene where the moving body is located based on the motion information, the image information and the depth information. The moving body comprises at least one image sensor used for acquiring image information of multiple frames of images of a scene where the moving body is located; a motion sensor that acquires motion information of the moving body; the invention further discloses the electronic equipment.

Description

Technical Field

[0001] The present invention relates to scheduling and path planning of a mobile body, and in particular to an electronic device for scheduling and path planning of a mobile body using image information and motion information, and a mobile body equipped with the electronic device. Background Art

[0002] With the continuous development of machine vision and artificial intelligence technologies, mobile objects such as AGVs (Automatic Guided Vehicles) are increasingly being used in factories, warehouses, restaurants, and other scenarios to replace human labor in the handling, loading, and delivery of goods or merchandise. Therefore, intelligent scheduling and path planning of AGVs have become key factors affecting the efficiency and accuracy of AGV operations.

[0003] Typically, an intelligent system for AGV scheduling includes electronic devices and remote control devices mounted on the AGV. These devices include at least one sensor that acquires environmental perception information in real time, as well as a processor or processing circuit that processes the environmental perception information acquired by the sensor. These processors or processing circuits perform scheduling and path planning for each AGV by invoking intelligent control models such as scheduling models. Typically, the upper-level scheduling model determines the optimization results of the overall horizontal transportation operation, while the lower-level routing model determines the actual completion time of each operation task for each AGV. For example, the main controller can use certain criteria (providing strategies and constraints) to uniformly assign scores to each grid position in the scene where the AGV is located and determine the next direction of travel of the AGV based on the scores of adjacent grid positions, thereby achieving scheduling of the AGV. For example, swarm intelligence can also be used to determine the current optimal route (optimization process) for each intelligent agent in the system. Therefore, multiple AGVs need to communicate with each other, share their respective environmental perception information, work collaboratively, and improve work efficiency.

[0004] To implement the aforementioned scheduling and routing, most existing AGVs use laser navigation technology or a combination of laser and vision-assisted navigation technology. While these navigation technologies can navigate and plan movement paths for AGVs using various methods, they typically require expensive lidar equipment and / or depth image sensors and are highly dependent on lighting conditions and the image quality of the image sensor. In other words, these methods are susceptible to changes in the external environment. Furthermore, the lack of fixed reference points for AGV navigation can result in lower accuracy. Furthermore, the image processing models used in existing technologies are often not lightweight enough to be placed on edge devices. Summary of the Invention

[0005] Technical issues

[0006] To this end, the object of the present invention is to provide an electronic device for scheduling and path planning of a mobile body and a mobile body equipped with the electronic device, so that the selection and routing of the optimal path can be achieved at a lower cost and has stronger resistance to communication and on-site interference.

[0007] Technical solutions to technical problems

[0008] One aspect of the present invention provides an electronic device, comprising: a memory storing computer program code; and a processing circuit configured to run the code and perform the following steps: acquiring image information of multiple frames of images from an image sensor mounted on a mobile body; acquiring motion information of a motion sensor of the mobile body, the motion information being associated with the image information of the multiple frames of images through a timestamp; fusing the motion information with the image information to acquire depth information that can be applied to each frame of the multiple frames of images; and obtaining scene map information of the scene in which the mobile body is located based on the motion information, the image information, and the depth information.

[0009] Preferably, the mobile object may be an AGV, and the processing circuit may be configured to collect wheel step information of the AGV from a step sensor of the AGV as the motion information.

[0010] Preferably, the image sensor may be a monocular RGB sensor.

[0011] Preferably, the electronic device may be the mobile object or may be mounted on the mobile object.

[0012] Preferably, the electronic device further includes a communication interface, and the processing circuit is further configured to control receiving, through the communication interface, the scene map information associated with the area passed by the other mobile body during movement from the other mobile body.

[0013] Preferably, the processing circuit obtains the scene map information of the scene in which the mobile object is located, including the following steps: training a depth estimation model based on the depth information of each frame image to obtain a multi-frame depth map corresponding to the multi-frame image of the scene; gridding the scene based on the motion information; acquiring environmental perception information corresponding to each grid position based on the image information and the depth map obtained at each divided grid position; and encoding the adjacent grids of each grid position according to the environmental perception information, and guiding the movement of the mobile object according to the encoding of the adjacent grids.

[0014] Preferably, in the process of encoding the grid in the scene by the processing circuit, the processing circuit may establish global map coordinates shared by two different moving objects in the scene based on an intersection of the encoding paths of the two moving objects.

[0015] Preferably, in the process of establishing the global map coordinates, the processing circuit may be configured to use any corner of the scene as a reference origin of the global map coordinates.

[0016] Preferably, based on the global map coordinates, the processing circuit may splice the scene map information obtained by the electronic device itself with the scene map information received from other processing circuits through the communication interface.

[0017] Preferably, in the process of encoding the grid in the scene by the processing circuit, the processing circuit may be configured to set the encoded value according to the position of the moving target of the moving object and the position of the obstacle in the scene.

[0018] Preferably, the processing circuit can be configured to: first, give priority to assigning a larger initial score to the grid in the adjacent grids that is closer to the moving target; second, adjust the initial score according to the depth information of the obstacle at each grid in the adjacent grids, so that the initial score of the grid in the adjacent grids that is closer to the obstacle is reduced more.

[0019] Preferably, after obtaining the scene map information of the scene in which the mobile body is located, the processing circuit can be configured to execute, when receiving a task instruction through the communication interface: calculating the shortest path between the current position of the mobile body and the position of the task target; and calculating the number of grids passed by the shortest path.

[0020] Preferably, the shortest path is obtained by the processing circuit calling a predetermined shortest path algorithm stored in the memory for calculation.

[0021] Preferably, after calculating the shortest path, the processing circuit can send the calculated number of grids passed by the shortest path to other electronic devices in the scene through the communication interface, and receive the number of grids passed by the shortest path from other mobile objects to the location of the task target calculated by the other electronic devices through the communication interface. When the number of grids passed by the shortest path calculated by the processing circuit is less than the number of grids passed by the shortest path from other mobile objects to the location of the task target calculated by the other electronic devices through the communication interface, the electronic device controls the mobile object to move toward the location of the task target and executes the task instruction.

[0022] Preferably, the scene map information may include classification information indicating whether each of the divided grid positions is unobstructed or blocked, and the calculation of the shortest path takes the classification information into consideration and avoids the blocked grid positions.

[0023] Preferably, when a new obstacle appears in the image information of the scene acquired by the electronic device, the electronic device can update the depth estimation model and the environmental perception information, and send the updated depth estimation model and the environmental perception information to other electronic devices in the scene through the communication interface.

[0024] Preferably, the environmental perception information may include at least the image information and the depth map of obstacles, other moving objects and / or targets existing around each of the grid positions.

[0025] Preferably, the processing circuit may be configured to use the distance of one rotation of the wheels of the moving object as the unit of the grid division when performing the grid division of the scene.

[0026] Another aspect of the present invention provides a mobile object, which includes: at least one image sensor, which obtains image information of multiple frames of images of the scene in which the mobile object is located; a motion sensor, which obtains motion information of the mobile object; and an electronic device, which is the electronic device as described above.

[0027] Beneficial effects

[0028] The electronic device and mobile object according to the present invention utilize the mobile object's onboard image sensor and inherent motion information to obtain image depth information, construct a scene map based on this depth information, and subsequently plan and select the mobile object's motion path. This eliminates the need for expensive additional equipment such as lidar and depth image sensors, thereby enabling the selection and routing of the mobile object's optimal motion path at a low cost.

[0029] Furthermore, unlike traditional AGV scheduling and path planning methods that rely on a central control center, the electronic devices and mobile objects of the present invention share access to previously encoded environmental perception information, leveraging swarm intelligence to select the optimal path. Consequently, the electronic devices and mobile objects of the present invention only need to communicate with each other within the scene, rather than with a main controller. This provides greater robustness, improved resistance to communication and field interference, and greater adaptability to diverse network environments and various operating conditions in different work scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figures 1A to 1D A schematic diagram showing an application scenario of an electronic device according to the present invention;

[0031] Figure 2 A schematic diagram for illustrating the working principle of a photoelectric encoder used in an electronic device according to the present invention;

[0032] Figure 3 A schematic diagram showing the principle of fusing image information and motion information by a processing circuit of an electronic device according to an embodiment of the present invention is shown;

[0033] Figure 4 A schematic diagram for illustrating the principle and process of an electronic device obtaining scene map information and constructing a scene map according to an embodiment of the present invention;

[0034] Figure 5 is a schematic diagram for explaining the principle and process of an electronic device obtaining scene map information and constructing a scene map according to an embodiment of the present invention;

[0035] Figure 6 is a schematic diagram for explaining the principle and process of an electronic device obtaining scene map information and constructing a scene map according to an embodiment of the present invention;

[0036] Figure 7 is a schematic diagram for explaining the principle and process of an electronic device obtaining scene map information and constructing a scene map according to an embodiment of the present invention;

[0037] Figure 8 is a schematic diagram illustrating the principle of environmental coding performed by an electronic device according to the present invention;

[0038] Figure 9 FIG. 1 is a schematic diagram showing an example in which an electronic device performs path planning for a mobile object according to the present invention. DETAILED DESCRIPTION

[0039] The present invention will be further described below with reference to the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it. However, the embodiments are not intended to limit the present invention. The embodiments of the present invention and the technical features in the embodiments may be combined with each other unless there is a conflict.

[0040] It should be noted that in the following detailed description and claims, the use of expressions such as "approximately" and "approximately" is intended to take into account factors such as manufacturing tolerances and processing accuracy that are understood by those skilled in the art, and will not lead to ambiguity in the description or unclear scope of protection. In addition, the directions such as "upper", "lower", "left", and "right" that appear in this document are only for the convenience of description and are defined in conjunction with the directions illustrated in the accompanying drawings or the common installation or working conditions of the device. After reading this document, those skilled in the art will obviously be able to easily know the direction corresponding to the direction described in this document when the device presents an orientation inconsistent with the orientation described in this document due to flipping or movement.

[0041] Hereinafter, the description will be given in the following order.

[0042] 1. Principles and Structures of Electronic Devices and Mobile Objects According to Embodiments of the Present Invention

[0043] 2. Acquisition of depth information by electronic devices

[0044] 3. Construction of scene map

[0045] 4. Path planning and selection of mobile objects

[0046] Hereinafter, the principles and structures of electronic equipment and mobile objects according to embodiments of the present invention will be described first.

[0047] 1. Principles and Structures of Electronic Devices and Mobile Objects According to Embodiments of the Present Invention

[0048] The electronic device for scheduling and path planning a mobile object according to the present invention utilizes swarm intelligence technology to perform scheduling and path planning for mobile objects. The concept of swarm intelligence originates from observations of social organisms such as insect colonies in nature. Colonies of social organisms often exhibit macroscopically intelligent behavioral characteristics through the cooperation of unintelligent individuals. Therefore, by employing such swarm intelligence technology, the electronic device according to the present invention can implement the scheduling and path planning functions for mobile objects with a relatively simple configuration and at a relatively low cost.

[0049] Figures 1A to 1D : is a schematic diagram showing the application scenarios of the electronic device for scheduling and path planning of mobile bodies according to the present invention. These application scenarios may be operations such as transportation, loading and unloading of goods in environments such as factories and warehouses. In these application scenarios, there are multiple mobile bodies such as AGV carts moving therein. These mobile bodies may be in different positions and different states. For example, some mobile bodies may be in a state of performing tasks, some mobile bodies may be in a state of returning to the standby area after completing the tasks, some mobile bodies may be in a state of waiting in the standby area, some mobile bodies may be in a state of charging in the charging area, and so on. In the prior art, these mobile bodies in different positions and different states in the scene need to be uniformly coordinated and scheduled by the main controller, and task allocation and movement path planning are performed. However, the electronic device for scheduling and path planning of mobile bodies according to the present invention utilizes swarm intelligence technology to enable multiple mobile bodies to schedule and plan their movement paths through distributed control without a main controller. As will be explained in detail below, such distributed control is achieved by multiple electronic devices sharing information and jointly building and maintaining a common scene map. Figure 1A In the scene shown, if an AGV finds an obstacle in its moving path, it can update the information related to the obstacle in the scene map so that other AGVs in the scene can also know the relevant information of the obstacle. Figures 1B to 1D In the illustrated scenario, AGV 1 receives an instruction to execute a high-priority task. In this scenario, through information exchange between multiple electronic devices for mobile object scheduling and path planning according to the present invention, AGV 2 is able to make way for AGV 1, and AGV 3 remains in place without affecting the path of AGV 1, thereby achieving coordinated movement and planning among multiple AGVs.

[0050] The electronic device according to the present invention can be mounted on various mobile bodies, or can be the mobile bodies themselves. In the following, the AGV is used as an example of a mobile body for explanation. However, the mobile body is not limited to the AGV.

[0051] like Figures 1A to 1D As shown, because the electronic devices according to the present invention utilize swarm intelligence technology, these mobile objects equipped with the electronic devices according to the present invention can communicate with each other and share their respective environmental perception information. For example, such communication is performed via wireless communication interfaces provided in the electronic devices. As a result, these mobile objects can complete their respective tasks through distributed control without the need for a remote controller or control center.

[0052] Specifically, the electronic device for scheduling and path planning a mobile object according to an embodiment of the present invention is, for example, disposed on the mobile object. In this case, the electronic device may be provided with memory and processing circuitry. The processing circuitry may, for example, be a processing circuit equipped with an intelligent chip such as the IMX500 and possessing intelligent computing capabilities. Furthermore, the mobile object is also provided with an image sensor. The image sensor is capable of capturing image information of the mobile object's surroundings in real time. This image information includes multiple frames of images captured by the image sensor. The image information of a captured frame may also include time information associated with the timestamp when the frame was captured. The image sensor may be an infrared image sensor, an RGB image sensor, or a combination thereof. For example, the image sensor may be a monocular RGB sensor. The electronic device for scheduling and path planning a mobile object according to an embodiment of the present invention may acquire image information acquired by these image sensors. Specifically, the processing circuit of the electronic device for scheduling and path planning a mobile object according to an embodiment of the present invention can access and read the image information by invoking a program stored in memory, and then perform the various processing described below.

[0053] 2. Acquisition of depth information by electronic devices

[0054] The electronic device for scheduling and path planning a mobile object according to an embodiment of the present invention can obtain motion information of the mobile object. This motion information can be acquired by a motion sensor mounted on the mobile object, or it can be wheel stepping information acquired by a stepping sensor within the mobile object's own stepping control device. For example, this motion information can directly or easily be used to obtain or calculate information such as the drive data of the mobile object's stepper motor, the mobile object's speed, acceleration, and azimuth, and can also include timestamp information of the mobile object's motion process.

[0055] For example, such motion information can be obtained by a stepping motor and a photoelectric encoder provided in a moving object such as an AGV. Figure 2 This is a schematic diagram used to illustrate the working principle of common photoelectric encoders. Figure 2As shown, the photoelectric encoder includes at least a light-emitting element 201, a photosensitive element 202 for sensing the light emitted by the light-emitting element, and an optical code disk 203. The optical code disk 203 is provided with many equidistant apertures or transparent parts along the circumference, and the optical code disk 203 is coaxial with the output shaft of the stepper motor of the AGV. When the motor rotates, the optical code disk 203 rotates at the same speed as the motor. The light signal emitted by the light-emitting element 201 passes through the apertures or transparent parts of the optical code disk 203 to generate a light pulse signal. The photosensitive element 202 detects such a light pulse signal and converts it into a digital signal output. The processor can know the current speed of the motor by counting the number of pulses detected by the photoelectric encoder per second, and then record the number of rotations of the AGV wheel.

[0056] In addition, in order to determine the rotation direction of the motor, that is, the moving direction of the AGV, such as Figure 2 As shown, the optical code disk 203 can also provide two pulse signals with a phase difference of 90°, namely, the A-phase signal and the B-phase signal. For example, when the optical code disk 203 rotates clockwise, the photosensitive element 202 first detects the A-phase signal and then detects the B-phase signal. When the optical code disk 203 rotates counterclockwise, the photosensitive element 202 first detects the B-phase signal and then detects the A-phase signal. By determining the order of the A-phase signal and the B-phase signal, the rotation direction of the motor can be determined; by counting the number of pulses of the A-phase signal and the B-phase signal, the rotation angle can be determined. In addition, as Figure 2 As shown, a Z-phase signal, which outputs one pulse per rotation of the optical code disk 203, can also be provided. The Z-phase signal can be used for reference point positioning to determine the starting point of the movement. By obtaining this information, the processing circuit of the electronic device can obtain more comprehensive motion information such as the AGV's travel direction and distance.

[0057] The processing circuitry of the electronic device has the capability to fuse the image information with the motion information, thereby obtaining similar depth information that can be applied to each of the multiple image frames. This similar depth information has a similar effect to that obtained by capturing images using expensive depth image sensors. However, because the stepper motors and photoelectric encoders are already built into mobile devices such as AGVs, costs and system complexity are significantly reduced.

[0058] Figure 3 The schematic diagram shows the principle of the processing circuit of the electronic device according to the embodiment of the present invention to fuse image information with motion information. The image sensor installed on the mobile body such as AGV can capture image information of the surrounding environment, and the image information includes multiple frames of images. When the mobile body such as AGV moves in the scene, the image sensor can capture the same object at different times. Figure 3As shown, the wall in front of the AGV car is used as an example of the shooting object. The AGV car drives towards the wall from a distance at timestamp T1, and drives in front of the wall at timestamp T5. During this process, the image sensor on the car shoots the wall at multiple times (for example, at timestamps T1 to T5) to obtain an RGB image of the wall. When the AGV car arrives in front of the wall at time T5, the processing circuit obtains the distance Dist1 between the AGV car and the target wall at time T1 based on the motion information of the AGV in the time period from T1 to T5, such as the number of rotations of the wheels. Then, the processing circuit merges the image frame containing the image information of the wall obtained by the image sensor on the car at time T1 with the distance Dist1 from the AGV car to the target wall at time T1 to generate a label Dist1@T1. Similarly, the processing circuit fuses the image frames containing the wall image information obtained by the image sensor on the trolley at T2, T3, and T4 with the distance Dist2, Dist3, and Dist4 from the AGV trolley to the target wall at T2, T3, and T4, respectively, and generates depth information labels Dist2@T2, Dist3@T3, and Dist4@T4, respectively. In this way, based on the RGB image, each frame of the image is associated with the corresponding timestamp and distance information, and image information similar to the depth image captured by a depth camera with a depth sensor can be obtained (also referred to as "depth information" in this article), thereby enabling the AGV to acquire the ability to perceive the distance of objects in the surrounding scene.

[0059] In this way, when an AGV is constantly moving in a scene, the AGV, by traversing all the moving routes in the scene, can eventually obtain the distance information of any position on any moving route in the scene relative to a specific target in the scene. When multiple AGVs are moving in the scene, the processing circuits of each AGV can share the depth information they have obtained through the communication interface, thereby jointly completing the acquisition of the distance information of any position on any moving route in the scene relative to a specific target in the scene. For example, a grayscale image containing a moving target can be obtained, and the value of each pixel on the grayscale image represents the distance.

[0060] 3. Construction of scene map

[0061] According to an embodiment of the present invention, the processing circuit of the electronic device can construct a scene map of the scene in which a mobile object such as an AGV is located based on the obtained image information and depth information, thereby realizing the scheduling and path planning of a mobile object such as an AGV in the scene.

[0062] Figures 4 to 7Schematic diagram for explaining the principle and process of an electronic device obtaining scene map information and constructing a scene map according to an embodiment of the present invention. Figure 4 As shown, there are multiple moving bodies such as AGV carts in the scene, and these moving bodies explore the scene map along a certain exploration path. Among them, for example, AGV cart A and AGV cart B respectively move along their respective exploration paths, and continuously take pictures of the road ahead (i.e., obtain image information) during the process of moving forward, and at the same time mark each frame of image information with a depth information label as described above, thereby generating depth information for each frame of image. Similarly, other moving bodies in the scene also collect the above data during operation. These data can be used to train the same depth estimation model. All AGV carts share this depth estimation model to generate depth maps of various locations in the scene. Such a depth map is part of the environmental perception information of various locations in the scene. Based on the environmental perception information, the processing circuit of the electronic device according to an embodiment of the present invention can control the movement of the moving body and ultimately complete the construction of the scene map.

[0063] For example, environmental perception information includes the positions and distances of obstacles, other moving objects, and moving targets around a moving object, identified based on image information and depth information. The depth estimation model can employ any suitable model known in the art for processing depth data. For example, Monodepth2, Lite-Mono models, and the like can be employed for depth estimation. The Lite-Mono model employs a continuous dilated convolution (CDC) module and a local-global feature interaction (LGFI) module. The former is used to extract rich multi-scale local features, while the latter utilizes a self-attention mechanism to encode random global information into local features. It employs convolutional neural networks (CNNs) to infer depth maps from grayscale images. Using the annotated depth, a regression CNN can be trained to predict the depth value of each pixel in the grayscale image. To highlight the main purpose of the present invention, a detailed description of the depth estimation models that can be employed by the present invention will not be provided herein. Such a depth estimation model can be stored in the memory of an electronic device for scheduling and path planning a moving object according to the present invention and can be called by the processing circuitry of the electronic device.

[0064] In the process of the mobile body exploring the entire scene map, the electronic device for scheduling and path planning of the mobile body according to the present invention can grid the scene environment passed by. For example, Figure 5 The schematic diagram shows the process of meshing a scene by an electronic device for scheduling and path planning of a mobile object according to the present invention. Figure 5As shown in , during the movement of each AGV, the electronic device grids the environment in units of wheel step distance (i.e., the distance the wheel rotates one circle) based on the acquired motion information. In other words, each grid corresponds to the distance traveled by the vehicle in one rotation. In other words, the environment in which the mobile object is located is divided into several grids representing the step units of the mobile object. Figure 5 In the diagram, each square pattern, circle pattern, and triangle pattern represents a grid.

[0065] In the process of completing the gridding of the scene by the processing circuit of the electronic device according to the present invention, the processing circuit can also set a score for each adjacent grid based on the environmental perception information of the adjacent grid obtained at the corresponding grid position (including image information, depth map, timestamp, identified specific object information, etc.) to perform environmental coding. Figure 8 FIG. 1 is a schematic diagram showing the principle of environmental coding performed by an electronic device according to the present invention. Figure 8 As shown in , when a mobile object is at a certain grid position, the processing circuit assigns values ​​to the adjacent grids based on the following two principles, based on the environmental perception information of the adjacent grids obtained at that position. First, the adjacent grids are assigned corresponding initial scores based on the distance from the set mobile target. The closer the grid is to the set target, the higher the score assigned. The highest score can be 5, for example. Second, if the environmental perception information in a certain direction contains an obstacle (which may also be a specific object that needs to be avoided, such as a region boundary or another mobile object), the scores of the adjacent grids in that direction are reduced. The scores can be reduced differently based on the distance from the obstacle. The closer the distance to the obstacle, the greater the reduction in the score. The lowest score can be 0, for example. In other words, the processing circuit of the electronic device according to the present invention assigns values ​​to the adjacent grids of each grid position based on the distance from the set target and the distance from the obstacle, thereby achieving environmental encoding for each grid position. During the encoding process, the mobile target of the mobile object needs to be pre-set. For a specific scene environment, its boundaries are fixed. Therefore, the boundary corners of the scene are usually set as the mobile targets used in the process of constructing the scene map.

[0066] Through such environmental coding, the electronic device according to the embodiment of the present invention can enable the mobile body to know the next direction of movement at each grid position. As the mobile body moves, the grid position of the mobile body changes in real time, and the coding of the adjacent grids of the grid position is also continuously updated accordingly. Since the electronic device for scheduling and path planning of mobile bodies according to the present invention adopts swarm intelligence technology, when there are multiple mobile bodies equipped with the electronic device in the application scenario, each mobile body can share the environmental perception information obtained by each mobile body. In this way, not only can new mobile bodies be added to participate in tasks in new scenes at any time, but also multiple cars can jointly build the same scene map.

[0067] Figure 6 FIG. 4 shows an example of how the electronic device according to the present invention confirms the position of a new moving object when a new moving object is added to a scene. Figure 6 As shown in , a new AGV C is placed in the scene. In this case, the new AGV C can share the gridding and coding information of the scene with all previous AGVs, and can quickly confirm its own position in the map based on its relative position in the current environment. For example, Figure 6 As shown in , the new AGV C obtains image information at its grid position, finds the edge of the scene as a reference target, and obtains depth information as described above while moving toward the reference target, thereby obtaining its relative position relative to the reference target. Furthermore, by sharing the environmental perception information of all previous AGVs, it knows its position in the entire scene. In this case, the newly added AGV C can directly use the grid information generated by other AGVs before it joins, as well as the score codes of the adjacent grids of each grid, without the need for recalculation.

[0068] When multiple mobile bodies explore a scene map together, in order to construct a scene map shared by the multiple mobile bodies, it is necessary to establish a unified global map coordinate for the gridding of each mobile body. Figure 7 FIG. 1 shows a schematic diagram of establishing global coordinates of an electronic device according to the present invention when there are multiple moving objects in a scene. Figure 7 As shown, the boundaries of each scene are known and determined, so the farthest travel position of the mobile body in two orthogonal directions (for example, the horizontal direction and the vertical direction in the figure) in a certain scene (i.e., the four corners in the figure) is also determined. Therefore, any corner can be used as the reference origin of the global map coordinates. In addition, the coordinates of the encoding paths of different mobile bodies are combined into a unified global map coordinate based on the intersection of the encoding paths of different mobile bodies. For example, Figure 7As shown in , the cross icon shows the path intersection X of AGV trolley A and AGV trolley B. In order to combine the coordinates of the coded paths of AGV trolley A and AGV trolley B into a unified global map coordinate, first, based on the reference origin of the selected global map coordinate, the global coordinates of the intersection X relative to the reference origin are obtained by gridding AGV trolley A or AGV trolley B. Then, based on the relative coordinates of each position on the path of AGV trolley A and AGV trolley B relative to the intersection X and the global coordinates of the intersection X relative to the reference origin, the unified global map coordinates of each grid position on the path of AGV trolley A and AGV trolley B are obtained. In a similar way, each grid position on the path of all AGV trolleys in the scene can obtain a unified global map coordinate. By sharing the global map coordinates of the grid positions of the scene map that it has explored in real time, each mobile body can realize the splicing of the scene map explored by each mobile body and finally construct a complete map of the entire scene. In the complete scene map constructed in this way, all possible paths of mobile objects have been gridded, and the scene map information of each grid position contains or is associated with the environmental perception information at that position. Figure 6 and 7 As shown in , based on the environmental perception information of each grid position, each grid position can also be marked as an unobstructed position and a blocked position, so that the blocked position can be avoided in subsequent path planning.

[0069] In this case, as will be described in detail below, the electronic device according to the present invention carried in each mobile object such as an AGV vehicle working in the scenario can be controlled through distributed nodes to independently plan paths for each vehicle and achieve collaborative planning among multiple vehicles.

[0070] 4. Path planning and selection of mobile objects

[0071] Below, we will refer to Figure 9 The process of path planning and selection of a mobile object performed by the electronic device according to the present invention is described.

[0072] After the electronic device according to the present invention completes the construction of the scene map, the mobile objects in the scene can enter normal operating mode. In this case, the electronic device according to the embodiment of the present invention can sense its own position information in the global map, and the processing circuit of the electronic device has target detection and distance estimation capabilities. Therefore, when each mobile object receives a task, it can determine the mobile object that will be sent to the target to perform the task through distributed control based on its own position and the target position, and other mobile objects will not appear in the movement path of the mobile object performing the task.

[0073] Specifically, when a designated target appears in the scene, the target's location information is transmitted to the electronic devices of each mobile object. Each electronic device at a different location in the scene immediately acquires image data of its surroundings using its image sensor. By combining this acquired image data with the constructed global map data, each electronic device can determine the location of each mobile object in the scene. Next, the processing circuitry of each electronic device invokes a path planning algorithm stored in its memory to determine the shortest path from the location of the mobile object where each electronic device resides to the target. Any suitable shortest path algorithm from the prior art, such as the Dijkstra algorithm or the Bellman-Ford algorithm, can be used as the path planning algorithm. For example, the classic Dijkstra algorithm can be used. The Dijkstra algorithm is a typical shortest path algorithm proposed by Dutch computer scientist Edsger W. Dijkstra in 1959. It is used to calculate the shortest path from one node to another. Its key feature is its greedy strategy, expanding outwards from a starting point (based on a breadth-first traversal) until reaching the end point. For example, the algorithm declares an array dis to store the shortest distance from the origin to each vertex and a set of vertices for which the shortest path has been found: T = {}. Initially, the path weight of the origin s is assigned to 0 (dis[s] = 0). If there is a path (s, m) that can be directly reached from the origin s, then dis[m] is set to w(s, m), and the path lengths of all other vertices that s cannot directly reach are set to infinity. Initially, the set T contains only the origin s. Then, the minimum value is selected from the dis array, which is the shortest path from the origin s to the vertex corresponding to that value, and the point is added to T, thus completing a vertex. Then, it is necessary to check whether the newly added vertex can reach other vertices and whether the path length to other points through the newly added vertex is shorter than that of the direct path from the origin. If so, the values ​​of these vertices in dis are replaced. Then, the minimum value is found from dis again, and the above steps are repeated until T contains all the vertices of the graph. In order to highlight the main purpose of the present invention, the shortest path algorithm that can be used is not described in detail here. It should be noted that when calculating the shortest path for each electronic device, since the blocked grid locations have been marked in the global map, the calculated shortest paths are all shortest paths that avoid the marked blocked locations.

[0074] After each electronic device calculates the shortest path to the target for the mobile body in which it is located, the processing circuit of each electronic device calculates the number of grids passed in the shortest path based on the grid division of the scene map previously performed. Since the size of each grid corresponds to the wheel stepping distance of the AGV car, the fewer the number of grids passed, the shorter the shortest path of the AGV car to the target. By calculating the number of grids contained in the shortest path to the target through real-time interaction, each mobile body can determine the shortest optimal path among the shortest paths of all AGV cars to the target in a distributed node control manner, and the corresponding AGV car will go to the target to perform the current task, and the paths of other AGV cars will not interfere with and affect the path of the AGV car to perform the current task (similar to Figures 1B to 1D ). Figure 9 An example of an electronic device performing path planning for a mobile object according to the present invention is shown. Figure 9 As shown in the figure, there are multiple possible paths for both AGVs to reach the destination. The electronic devices according to the embodiments of the present invention, carried on the two AGVs, perform the aforementioned path planning and determine the shortest paths A and B, respectively, to the destination, as indicated by the arrows in the figure. Then, by further calculating and comparing the number of map grids traversed by shortest paths A and B, shortest path A is ultimately determined to be the shorter optimal path.

[0075] The above-mentioned path planning and selection process is carried out in a dynamic manner. That is to say, every time each AGV car passes through a map grid, the electronic device according to the embodiment of the present invention will re-perform the above-mentioned path planning and selection calculation to confirm that the path of the AGV car currently performing the task is the optimal path to the target. If the calculated optimal path changes due to the appearance of new obstacles in the map or changes in the target position, the corresponding task will be immediately replaced by the most recently calculated moving body on the optimal path. When a new obstacle appears in the scene that does not exist in the current scene map (similar to Figure 1A (As shown in the example above), the electronic device that first discovers the obstacle repeatedly executes the method described above to update the depth map and environmental perception information of the obstacle's location, and also updates the corresponding map code. In other words, the electronic device according to the present invention also has the ability to maintain and update the scene map in real time.

[0076] The electronic device and the mobile object equipped with the electronic device according to the present invention have been described above with reference to the accompanying drawings. The electronic device and the mobile object according to the present invention can be constructed as follows.

[0077] (1) An electronic device comprising:

[0078] a memory storing computer program code; and

[0079] The processing circuit is configured to run the code and perform the following steps:

[0080] acquiring image information of a plurality of frames of images from an image sensor mounted on a moving object;

[0081] acquiring motion information of a motion sensor of the moving object, wherein the motion information and the image information of the multiple frames of images are associated with each other through a timestamp;

[0082] fusing the motion information with the image information to obtain depth information applicable to each frame of the multiple frames; and

[0083] Scene map information of the scene where the moving object is located is obtained based on the motion information, the image information and the fused depth information. (2)

[0085] The electronic device as described in (1) above, wherein the moving object is an AGV, and the processing circuit is configured to collect wheel step information of the AGV from a step sensor of the AGV as the motion information. (3)

[0087] The electronic device as described in (1) or (2) above, wherein the image sensor is a monocular RGB sensor. (4)

[0089] The electronic device as described in any one of (1) to (3) above, wherein the electronic device is the mobile object or is mounted on the mobile object. (5)

[0091] The electronic device as described in any one of (1) to (4) above, wherein the electronic device further includes a communication interface, and

[0092] The processing circuit is further configured to control the receiving of the scene map information associated with an area passed by the other mobile body during movement from the other mobile body through the communication interface. (6)

[0094] As described in the electronic device (5) above, wherein the processing circuit obtains the scene map information of the scene in which the mobile object is located, comprising the following steps:

[0095] Training a depth estimation model based on the depth information of each frame of image to obtain a multi-frame depth map corresponding to the multi-frame image of the scene;

[0096] Gridding the scene based on the motion information;

[0097] Acquire environmental perception information corresponding to each of the divided grid positions based on the image information and the depth map obtained at each of the divided grid positions; and

[0098] Encoding the adjacent grids of each of the grid positions according to the environmental perception information, and guiding the movement of the mobile object according to the encoding of the adjacent grids. (7)

[0100] An electronic device as described in (6) above, wherein, in the process of the processing circuit encoding the grid in the scene, the processing circuit establishes global map coordinates shared by the two moving bodies based on the intersection of the encoding paths of two different moving bodies in the scene. (8)

[0102] The electronic device as described in (7) above, wherein, in the process of establishing the global map coordinates, any corner of the scene is used as the reference origin of the global map coordinates. (9)

[0104] An electronic device as described in (7) above, wherein, based on the global map coordinates, the processing circuit splices the scene map information obtained by the electronic device itself with the scene map information received from other processing circuits through the communication interface. (10)

[0106] An electronic device as described in any one of (6) to (9) above, wherein, in the process of the processing circuit encoding the grid in the scene, the processing circuit is configured to set the encoded value according to the position of the moving target of the mobile body and the position of the obstacle in the scene. (11)

[0108] The electronic device as described in (10) above, wherein the processing circuit is configured to:

[0109] First, the grids in the adjacent grids that are closer to the moving target are preferentially assigned larger initial scores;

[0110] Secondly, the initial score is adjusted according to the depth information of the obstacle at each of the adjacent grids, so that the initial score of the grid closer to the obstacle in the adjacent grids is reduced more. (12)

[0112] The electronic device as described in any one of (6) to (11) above, wherein, after obtaining scene map information of the scene in which the mobile object is located, the processing circuit is configured to execute, upon receiving a task instruction through the communication interface:

[0113] Calculating the shortest path between the current position of the mobile object and the position of the mission target; and

[0114] The number of the grids passed by the shortest path is calculated. (13)

[0116] The electronic device as described in (12) above, wherein the shortest path is obtained by the processing circuit calling a predetermined shortest path algorithm stored in the memory for calculation. (14)

[0118] An electronic device as described in (12) or (13) above, wherein, after calculating the shortest path, the processing circuit sends the calculated number of grids passed by the shortest path to other electronic devices in the scene through the communication interface, and receives the number of grids passed by the shortest path from other mobile bodies to the location of the task target calculated by the other electronic devices through the communication interface. (15)

[0120] An electronic device as described in (14) above, wherein, when the number of grids passed by the shortest path calculated by the processing circuit is less than the number of grids passed by the shortest path from other mobile bodies to the position of the task target calculated by the other electronic devices received through the communication interface, the electronic device controls the mobile body to move toward the position of the task target and executes the task instruction. (16)

[0122] An electronic device as described in any one of (12) to (15) above, wherein the scene map information includes classification information indicating whether each of the divided grid locations is unobstructed or blocked, and

[0123] The calculation of the shortest path takes the classification information into account and avoids blocked grid locations. (17)

[0125] An electronic device as described in any one of (12) to (16) above, wherein, when a new obstacle appears in the image information of the scene acquired by the electronic device, the electronic device updates the depth estimation model and the environmental perception information, and sends the updated depth estimation model and the environmental perception information to the other electronic devices in the scene through the communication interface. (18)

[0127] An electronic device as described in any one of (6) to (17) above, wherein the environmental perception information includes at least the image information and the depth map of obstacles, other moving objects and / or targets existing around each of the grid positions. (19)

[0129] An electronic device as described in any one of (6) to (18) above, wherein the processing circuit is configured to use the distance of one rotation of the wheels of the moving body as the unit of the grid division when performing the grid division of the scene.

[0130] (20) A mobile object comprising:

[0131] at least one image sensor, configured to acquire image information of multiple frames of images of a scene in which the moving object is located;

[0132] A motion sensor, configured to obtain motion information of the moving object;

[0133] An electronic device, wherein the electronic device is the electronic device described in (1) to (19).

[0134] It is obvious to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or basic characteristics of the present invention. For example, the electronic device for scheduling and path planning of a mobile body according to the present invention can be mounted on the mobile body, or it can be the mobile body itself. In this case, the electronic device itself includes components such as an image sensor and a motion sensor. Therefore, from any point of view, the embodiments described herein should be regarded as exemplary and non-restrictive, and the scope of the present invention is defined by the appended claims rather than the above description, and it is intended that all changes that fall within the meaning and scope of the equivalent elements of the claims be included in the present invention. Any figure mark in the claims should not be regarded as limiting the claim to which it relates.

[0135] In addition, it should be understood that although this specification is described in terms of implementation methods, not every implementation method contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

Claims

1. An electronic device, wherein: The electronic device comprises: a memory storing computer program code; and The processing circuit is configured to run the code and perform the following steps: acquiring image information of a plurality of frames of images from an image sensor mounted on a moving object; acquiring motion information of a motion sensor of the moving object, wherein the motion information and the image information of the multiple frames of images are associated with each other through a timestamp; fusing the motion information with the image information to obtain depth information applicable to each frame of the multiple frames; and Scene map information of the scene where the moving object is located is obtained based on the motion information, the image information and the depth information.

2. The electronic device according to claim 1, wherein The mobile object is an AGV, and the processing circuit is configured to collect wheel step information of the AGV from a step sensor of the AGV as the motion information.

3. The electronic device according to claim 1, wherein The image sensor is a monocular RGB sensor.

4. The electronic device according to claim 1, wherein The electronic device is the mobile object or is mounted on the mobile object.

5. The electronic device according to any one of claims 1 to 4, wherein: The electronic device further includes a communication interface, and The processing circuit is further configured to control the receiving of the scene map information associated with an area passed by the other mobile body during movement from the other mobile body through the communication interface.

6. The electronic device according to claim 5, wherein: The processing circuit obtains the scene map information of the scene where the mobile object is located, including the following steps: Training a depth estimation model based on the depth information of each frame of image to obtain a multi-frame depth map corresponding to the multi-frame image of the scene; Gridding the scene based on the motion information; Acquire environmental perception information corresponding to each of the divided grid positions based on the image information and the depth map obtained at each of the divided grid positions; and Encoding the adjacent grids of each of the grid positions according to the environmental perception information, and guiding the movement of the mobile object according to the encoding of the adjacent grids.

7. The electronic device according to claim 6, wherein: In the process of encoding the grid in the scene by the processing circuit, the processing circuit establishes a global map coordinate shared by two different moving objects in the scene based on an intersection of the encoding paths of the two moving objects.

8. The electronic device according to claim 7, wherein: In the process of establishing the global map coordinates, the processing circuit is configured to use any corner of the scene as a reference origin of the global map coordinates.

9. The electronic device according to claim 7, wherein: Based on the global map coordinates, the processing circuit splices the scene map information obtained by the electronic device itself with the scene map information received from other processing circuits through the communication interface.

10. The electronic device according to claim 6, wherein In the process of encoding the grid in the scene by the processing circuit, the processing circuit is configured to set the encoded value according to the position of the moving target of the moving object and the position of the obstacle in the scene.

11. The electronic device according to claim 10, wherein: The processing circuit is configured to: First, the grids in the adjacent grids that are closer to the moving target are preferentially assigned larger initial scores; Secondly, the initial score is adjusted according to the depth information of the obstacle at each of the adjacent grids, so that the initial score of the grid closer to the obstacle in the adjacent grids is reduced more.

12. The electronic device according to claim 6, wherein: After obtaining the scene map information of the scene in which the mobile object is located, the processing circuit is configured to execute, upon receiving a task instruction through the communication interface: Calculating the shortest path between the current position of the mobile object and the position of the mission target; and The number of the grids passed by the shortest path is calculated.

13. The electronic device according to claim 12, wherein: The shortest path is obtained by the processing circuit calling a predetermined shortest path algorithm stored in the memory to perform calculations.

14. The electronic device according to claim 12, wherein: After calculating the shortest path, the processing circuit sends the calculated number of grids passed by the shortest path to other electronic devices in the scene through the communication interface, and receives the number of grids passed by the shortest path from other mobile objects to the location of the task target calculated by the other electronic devices through the communication interface.

15. The electronic device according to claim 14, wherein When the number of grids passed by the shortest path calculated by the processing circuit is less than the number of grids passed by the shortest path from the other mobile body to the position of the task target calculated by the other electronic device received through the communication interface, the electronic device controls the mobile body to move toward the position of the task target and executes the task instruction.

16. The electronic device according to claim 12, wherein: The scene map information includes classification information indicating whether each of the divided grid locations is unobstructed or blocked, and The calculation of the shortest path takes the classification information into account and avoids blocked grid locations.

17. The electronic device according to claim 12, wherein: When a new obstacle appears in the image information of the scene acquired by the electronic device, the electronic device updates the depth estimation model and the environmental perception information, and sends the updated depth estimation model and the environmental perception information to other electronic devices in the scene through the communication interface.

18. The electronic device according to claim 6, wherein: The environmental perception information includes at least the image information and the depth map of obstacles, other moving objects and / or targets around each of the grid positions.

19. The electronic device according to claim 6, wherein: The processing circuit is configured to use a distance corresponding to one rotation of the wheels of the moving object as a unit of the grid division when performing the grid division of the scene.

20. A mobile object, comprising: at least one image sensor, configured to acquire image information of multiple frames of images of a scene in which the moving object is located; A motion sensor, configured to obtain motion information of the moving object; as well as An electronic device, wherein the electronic device is the electronic device according to claims 1 to 19.

Images