Electronic device for scheduling and path planning of mobile unit, and mobile unit

Abstract

An electronic device and a mobile unit. The electronic device comprises: a memory, having computer program code stored therein; and a processing circuit, configured to run the code and perform the following steps: obtaining image information of a plurality of images from an image sensor mounted on a mobile unit; obtaining motion information of a motion sensor of the mobile unit, the motion information and the image information of the plurality of images being associated with each other by means of timestamps; fusing the motion information with the image information, so as to obtain depth information applicable to each of the plurality of images; and on the basis of the motion information, the image information, and the depth information, obtaining scene map information of an environment in which the mobile unit is located. The mobile unit comprises: at least one image sensor, configured to obtain image information of a plurality of images of an environment in which the mobile unit is located; a motion sensor, configured to obtain motion information of the mobile unit; and the electronic device.

Description

Electronic device for scheduling and path planning of mobile body and mobile body

[0001] Citation of Related Applications

[0002] This application claims the benefit of Chinese Patent Application No. 202410211896.1 filed with the State Intellectual Property Office of the People's Republic of China on February 26, 2024, the entire contents of which are hereby incorporated by reference into this document in their entirety. Technical Field

[0003] The present invention relates to scheduling and path planning of a mobile body, and in particular to an electronic device for scheduling and path planning of a mobile body using image information and motion information, and a mobile body equipped with the electronic device. Background Art

[0004] With the continuous development of machine vision and artificial intelligence technologies, mobile objects such as AGVs (Automatic Guided Vehicles) are increasingly being used in factories, warehouses, restaurants, and other scenarios to replace manual labor in the handling, loading, and delivery of goods or merchandise. Therefore, intelligent scheduling and path planning for AGVs have become key factors affecting the efficiency and accuracy of AGV operations.

[0005] Typically, an intelligent system for AGV scheduling includes electronic devices and remote control devices mounted on the AGV. These devices include at least one sensor that acquires environmental perception information in real time, as well as a processor or processing circuit that processes the environmental perception information acquired by the sensor. These processors or processing circuits perform scheduling and path planning for each AGV by invoking intelligent control models such as scheduling models. Typically, the upper-level scheduling model determines the optimization results of the overall horizontal transportation operation, while the lower-level routing model determines the actual completion time of each operation task for each AGV. For example, the main controller can use certain criteria (providing strategies and constraints) to uniformly assign scores to each grid position in the scene where the AGV is located and determine the next direction of travel of the AGV based on the scores of adjacent grid positions, thereby achieving scheduling of the AGV. For example, swarm intelligence can also be used to determine the current optimal route (optimization process) for each intelligent agent in the system. Therefore, multiple AGVs need to communicate with each other, share their respective environmental perception information, work collaboratively, and improve work efficiency.

[0006] To implement the aforementioned scheduling and routing, most existing AGVs use laser navigation technology or a combination of laser and vision-assisted navigation technology. While these navigation technologies can navigate and plan movement paths for AGVs using various methods, they typically require expensive lidar equipment and / or depth image sensors and are highly dependent on lighting conditions and the image quality of the image sensor. In other words, these methods are susceptible to changes in the external environment. Furthermore, the lack of fixed reference points for AGV navigation can result in lower accuracy. Furthermore, the image processing models used in existing technologies are often not lightweight enough to be placed on edge devices. Summary of the Invention Technical issues

[0007] To this end, the object of the present invention is to provide an electronic device for scheduling and path planning of a mobile body and a mobile body equipped with the electronic device, so that the selection and routing of the optimal path can be achieved at a lower cost and has stronger resistance to communication and on-site interference.

[0008] Technical solutions to technical problems

[0009] One aspect of the present invention provides an electronic device, comprising: a memory storing computer program code; and a processing circuit configured to run the code and perform the following steps: acquiring image information of multiple frames of images from an image sensor mounted on a mobile body; acquiring motion information of a motion sensor of the mobile body, the motion information being associated with the image information of the multiple frames of images through a timestamp; fusing the motion information with the image information to acquire depth information that can be applied to each frame of the multiple frames of images; and obtaining scene map information of the scene in which the mobile body is located based on the motion information, the image information, and the depth information.

[0010] Preferably, the mobile object may be an AGV, and the processing circuit may be configured to collect wheel step information of the AGV from a step sensor of the AGV as the motion information.

[0011] Preferably, the image sensor may be a monocular RGB sensor.

[0012] Preferably, the electronic device may be the mobile object or may be mounted on the mobile object.

[0013] Preferably, the electronic device further includes a communication interface, and the processing circuit is further configured to control receiving, through the communication interface, the scene map information associated with the area passed by the other mobile body during movement from the other mobile body.

[0014] Preferably, the processing circuit obtains the scene map information of the scene in which the mobile object is located, including the following steps: training a depth estimation model based on the depth information of each frame image to obtain a multi-frame depth map corresponding to the multi-frame image of the scene; gridding the scene based on the motion information; acquiring environmental perception information corresponding to each grid position based on the image information and the depth map obtained at each divided grid position; and encoding the adjacent grids of each grid position according to the environmental perception information, and guiding the movement of the mobile object according to the encoding of the adjacent grids.

[0015] Preferably, in the process of encoding the grid in the scene by the processing circuit, the processing circuit may establish global map coordinates shared by two different moving objects in the scene based on an intersection of the encoding paths of the two moving objects.

[0016] Preferably, in the process of establishing the global map coordinates, the processing circuit may be configured to use any corner of the scene as a reference origin of the global map coordinates.

[0017] Preferably, based on the global map coordinates, the processing circuit may splice the scene map information obtained by the electronic device itself with the scene map information received from other processing circuits through the communication interface.

[0018] Preferably, in the process of encoding the grid in the scene by the processing circuit, the processing circuit may be configured to set the encoded value according to the position of the moving target of the moving object and the position of the obstacle in the scene.

[0019] Preferably, the processing circuit can be configured to: first, give priority to assigning a larger initial score to the grid in the adjacent grids that is closer to the moving target; second, adjust the initial score according to the depth information of the obstacle at each grid in the adjacent grids, so that the initial score of the grid in the adjacent grids that is closer to the obstacle is reduced more.

[0020] Preferably, after obtaining the scene map information of the scene in which the mobile body is located, the processing circuit can be configured to execute, when receiving a task instruction through the communication interface: calculating the shortest path between the current position of the mobile body and the position of the task target; and calculating the number of grids passed by the shortest path.

[0021] Preferably, the shortest path is obtained by the processing circuit calling a predetermined shortest path algorithm stored in the memory for calculation.

[0022] Preferably, after calculating the shortest path, the processing circuit can send the calculated number of grids passed by the shortest path to other electronic devices in the scene through the communication interface, and receive the number of grids passed by the shortest path from other mobile objects to the location of the task target calculated by the other electronic devices through the communication interface. When the number of grids passed by the shortest path calculated by the processing circuit is less than the number of grids passed by the shortest path from other mobile objects to the location of the task target calculated by the other electronic devices through the communication interface, the electronic device controls the mobile object to move toward the location of the task target and executes the task instruction.

[0023] Preferably, the scene map information may include classification information indicating whether each of the divided grid positions is unobstructed or blocked, and the calculation of the shortest path takes the classification information into consideration and avoids the blocked grid positions.

[0024] Preferably, when a new obstacle appears in the image information of the scene acquired by the electronic device, the electronic device can update the depth estimation model and the environmental perception information, and send the updated depth estimation model and the environmental perception information to other electronic devices in the scene through the communication interface.

[0025] Preferably, the environmental perception information may include at least the image information and the depth map of obstacles, other moving objects and / or targets existing around each of the grid positions.

[0026] Preferably, the processing circuit may be configured to use the distance of one rotation of the wheels of the moving object as the unit of the grid division when performing the grid division of the scene.

[0027] Another aspect of the present invention provides a mobile object, which includes: at least one image sensor, which obtains image information of multiple frames of images of the scene in which the mobile object is located; a motion sensor, which obtains motion information of the mobile object; and an electronic device, which is the electronic device described above. Beneficial effects

[0028] The electronic device and mobile object according to the present invention utilize the mobile object's onboard image sensor and inherent motion information to obtain image depth information, construct a scene map based on this depth information, and subsequently plan and select the mobile object's motion path. This eliminates the need for expensive additional equipment such as lidar and depth image sensors, thereby enabling the selection and routing of the mobile object's optimal motion path at a low cost.

[0029] Furthermore, unlike traditional AGV scheduling and path planning methods that rely on a central control center, the electronic devices and mobile objects of the present invention share access to previously encoded environmental perception information, leveraging swarm intelligence to select the optimal path. Consequently, the electronic devices and mobile objects of the present invention only need to communicate with each other within the scene, rather than with a main controller. This provides greater robustness, improved resistance to communication and field interference, and greater adaptability to diverse network environments and various operating conditions in different work scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] 1A to 1D are schematic diagrams showing application scenarios of electronic devices according to the present invention;

[0031] FIG2 is a schematic diagram for illustrating the working principle of a photoelectric encoder used in an electronic device according to the present invention;

[0032] FIG3 is a schematic diagram showing the principle of fusing image information and motion information by a processing circuit of an electronic device according to an embodiment of the present invention;

[0033] 4 is a schematic diagram illustrating the principle and process of an electronic device obtaining scene map information and constructing a scene map according to an embodiment of the present invention;

[0034] 5 is a schematic diagram for explaining the principle and process of an electronic device obtaining scene map information and constructing a scene map according to an embodiment of the present invention;

[0035] 6 is a schematic diagram for explaining the principle and process of an electronic device obtaining scene map information and constructing a scene map according to an embodiment of the present invention;

[0036] 7 is a schematic diagram for explaining the principle and process of an electronic device obtaining scene map information and constructing a scene map according to an embodiment of the present invention;

[0037] FIG8 is a schematic diagram showing the principle of environmental coding performed by an electronic device according to the present invention;

[0038] FIG. 9 is a schematic diagram showing an example of an electronic device performing path planning for a moving object according to the present invention. DETAILED DESCRIPTION

[0039] The present invention will be further described below with reference to the accompanying drawings and specific embodiments so that those skilled in the art can better understand and implement the present invention. However, the embodiments are not intended to limit the present invention. The embodiments of the present invention and the technical features in the embodiments may be combined with each other unless there is a conflict.

[0040] It should be noted that in the following detailed description and claims, the use of expressions such as "approximately" and "approximately" is intended to take into account factors such as manufacturing tolerances and processing accuracy that are understood by those skilled in the art, and will not lead to ambiguity in the description or unclear scope of protection. In addition, the directions such as "upper", "lower", "left", and "right" that appear in this document are only for the convenience of description and are defined in conjunction with the directions illustrated in the accompanying drawings or the common installation or working conditions of the device. After reading this document, those skilled in the art will obviously be able to easily know the direction corresponding to the direction described in this document when the device presents an orientation inconsistent with the orientation described in this document due to flipping or movement.

[0041] Hereinafter, the description will be given in the following order.

[0042] 1. Principles and Structures of Electronic Devices and Mobile Objects According to Embodiments of the Present Invention

[0043] 2. Acquisition of depth information by electronic devices

[0044] 3. Construction of scene map

[0045] 4. Path planning and selection of mobile objects

[0046] Hereinafter, the principles and structures of electronic equipment and mobile objects according to embodiments of the present invention will be described first.

[0047] 1. Principles and Structures of Electronic Devices and Mobile Objects According to Embodiments of the Present Invention

[0048] The electronic device for scheduling and path planning a mobile object according to the present invention utilizes swarm intelligence technology to perform scheduling and path planning for mobile objects. The concept of swarm intelligence originates from observations of social organisms such as insect colonies in nature. Colonies of social organisms often exhibit macroscopically intelligent behavioral characteristics through the cooperation of unintelligent individuals. Therefore, by employing such swarm intelligence technology, the electronic device according to the present invention can implement the scheduling and path planning functions for mobile objects with a relatively simple configuration and at a relatively low cost.

[0049] Figures 1A to 1D are schematic diagrams illustrating application scenarios of an electronic device for scheduling and path planning a mobile object according to the present invention. These application scenarios may involve operations such as transporting and loading and unloading goods in environments such as factories and warehouses. In these application scenarios, multiple mobile objects, such as AGVs, are moving around. These mobile objects may be in different locations and states. For example, some mobile objects may be executing a task, some may be returning to a standby area after completing a task, some may be on standby in a standby area, some may be charging in a charging area, and so on. In the prior art, these mobile objects in different locations and states in a scenario require unified coordination and scheduling by a main controller, which also performs task allocation and path planning. However, the electronic device for scheduling and path planning a mobile object according to the present invention utilizes swarm intelligence technology, enabling multiple mobile objects to autonomously schedule and plan their paths through distributed control without a main controller. As will be explained in detail below, this distributed control is achieved by multiple electronic devices sharing information and jointly building and maintaining a common scene map. In the scenario shown in FIG1A , if an AGV finds an obstacle in its moving path, it can update the information related to the obstacle in the scene map so that other AGVs in the scene can also know the relevant information of the obstacle. In the scenarios shown in FIG1B to FIG1D , AGV No. 1 receives an instruction to execute a task with a high priority. In this case, through the information interaction between multiple electronic devices for scheduling and path planning of mobile bodies according to the present invention, AGV No. 2 can make way for AGV No. 1, and AGV No. 3 can stay in place without affecting the travel path of AGV No. 1, thereby achieving coordination and planning of the moving paths between multiple AGVs.

[0050] The electronic device according to the present invention can be mounted on various mobile bodies, or can be the mobile bodies themselves. In the following, the AGV is used as an example of a mobile body for explanation. However, the mobile body is not limited to the AGV.

[0051] As shown in Figures 1A to 1D , because the electronic devices according to the present invention utilize swarm intelligence technology, these mobile objects equipped with these electronic devices can communicate with each other and share their respective environmental perception information. For example, this mutual communication occurs via wireless communication interfaces provided in the electronic devices. As a result, these mobile objects can complete their respective tasks through distributed control without the need for a remote controller or control center.

[0052] Specifically, the electronic device for scheduling and path planning a mobile object according to an embodiment of the present invention is, for example, disposed on the mobile object. In this case, the electronic device may be provided with memory and processing circuitry. The processing circuitry may, for example, be a processing circuit equipped with an intelligent chip such as the IMX500 and possessing intelligent computing capabilities. Furthermore, the mobile object is also provided with an image sensor. The image sensor is capable of capturing image information of the mobile object's surroundings in real time. This image information includes multiple frames of images captured by the image sensor. The image information of a captured frame may also include time information associated with the timestamp when the frame was captured. The image sensor may be an infrared image sensor, an RGB image sensor, or a combination thereof. For example, the image sensor may be a monocular RGB sensor. The electronic device for scheduling and path planning a mobile object according to an embodiment of the present invention may acquire image information acquired by these image sensors. Specifically, the processing circuit of the electronic device for scheduling and path planning a mobile object according to an embodiment of the present invention can access and read the image information by invoking a program stored in memory, and then perform the various processing described below.

[0053] 2. Acquisition of depth information by electronic devices

[0054] The electronic device for scheduling and path planning a mobile object according to an embodiment of the present invention can obtain motion information of the mobile object. This motion information can be acquired by a motion sensor mounted on the mobile object, or it can be wheel stepping information acquired by a stepping sensor within the mobile object's own stepping control device. For example, this motion information can directly or easily be used to obtain or calculate information such as the drive data of the mobile object's stepper motor, the mobile object's speed, acceleration, and azimuth, and can also include timestamp information of the mobile object's motion process.

[0055] For example, such motion information can be obtained by a stepper motor and photoelectric encoder installed in a mobile object such as an AGV. Figure 2 is a schematic diagram used to illustrate the working principle of a common photoelectric encoder. As shown in Figure 2, the photoelectric encoder includes at least a light-emitting element 201, a photosensitive element 202 for sensing the light emitted by the light-emitting element, and an optical code disk 203. The optical code disk 203 is provided with a plurality of equally spaced apertures or transparent portions along the circumference. The optical code disk 203 is coaxial with the output shaft of the AGV's stepper motor. When the motor rotates, the optical code disk 203 rotates at the same speed as the motor. The light signal emitted by the light-emitting element 201 passes through the apertures or transparent portions of the optical code disk 203, generating a light pulse signal. The photosensitive element 202 detects this light pulse signal and converts it into a digital signal for output. By counting the number of pulses detected by the photoelectric encoder per second, the processor can determine the current motor speed and thus record the number of rotations of the AGV wheel.

[0056] Furthermore, to determine the motor's rotational direction, and therefore the AGV's direction of movement, as shown in Figure 2, the optical code disk 203 can also provide two pulse signals, namely, the A-phase signal and the B-phase signal, with a 90° phase difference. For example, when the optical code disk 203 rotates clockwise, the photosensitive element 202 first detects the A-phase signal and then the B-phase signal. When the optical code disk 203 rotates counterclockwise, the photosensitive element 202 first detects the B-phase signal and then the A-phase signal. By determining the order of the A-phase and B-phase signals, the motor's rotational direction can be determined; by counting the number of pulses in the A-phase and B-phase signals, the rotation angle can be determined. Furthermore, as shown in Figure 2, a Z-phase signal, which outputs one pulse per rotation of the optical code disk 203, can also be provided. The Z-phase signal can be used for reference point positioning to determine the starting point of movement. By acquiring this information, the electronic device's processing circuitry can obtain richer motion information, such as the AGV's direction and distance traveled.

[0057] The processing circuitry of the electronic device has the capability to fuse the image information with the motion information, thereby obtaining similar depth information that can be applied to each of the multiple image frames. This similar depth information has a similar effect to that obtained by capturing images using expensive depth image sensors. However, because the stepper motors and photoelectric encoders are already built into mobile devices such as AGVs, costs and system complexity are significantly reduced.

[0058] Figure 3 illustrates a schematic diagram of the principle of fusing image information with motion information by a processing circuit of an electronic device according to an embodiment of the present invention. An image sensor mounted on a mobile object, such as an AGV, can capture image information of the surrounding environment, including multiple frames of images. As the AGV or other mobile object moves within a scene, the image sensor can capture the same subject at different times. As shown in Figure 3, the wall in front of the AGV is used as an example of a subject for capture. The AGV begins moving toward the wall from a distance at timestamp T1 and arrives in front of the wall at timestamp T5. During this process, the image sensor on the AGV captures the wall at multiple times (e.g., between timestamps T1 and T5) to obtain RGB images of the wall. When the AGV arrives in front of the wall at timestamp T5, the processing circuit determines the distance Dist1 between the AGV and the target wall at timestamp T1 based on the acquired motion information of the AGV during the time period T1 to T5, such as the number of wheel rotations. Next, the processing circuit fuses the image frame containing the wall image information obtained by the image sensor on the trolley at time T1 with the distance Dist1 from the AGV trolley to the target wall at time T1, and generates the label Dist1@T1. Similarly, the processing circuit fuses the image frames containing the wall image information obtained by the image sensor on the trolley at times T2, T3, and T4 with the distance Dist2, Dist3, and Dist4 from the AGV trolley to the target wall at times T2, T3, and T4, respectively, and generates depth information labels Dist2@T2, Dist3@T3, and Dist4@T4, respectively. In this way, on the basis of the RGB image, each frame of the image is associated with the corresponding timestamp and distance information, and image information similar to the depth image taken by a depth camera with a depth sensor can be obtained (also referred to as "depth information" in this article), thereby enabling the AGV to acquire the ability to perceive the distance of objects in the surrounding scene.

[0059] In this way, when an AGV is constantly moving in a scene, the AGV, by traversing all the moving routes in the scene, can eventually obtain the distance information of any position on any moving route in the scene relative to a specific target in the scene. When multiple AGVs are moving in the scene, the processing circuits of each AGV can share the depth information they have obtained through the communication interface, thereby jointly completing the acquisition of the distance information of any position on any moving route in the scene relative to a specific target in the scene. For example, a grayscale image containing a moving target can be obtained, and the value of each pixel on the grayscale image represents the distance.

[0060] 3. Construction of scene map

[0061] According to an embodiment of the present invention, the processing circuit of the electronic device can construct a scene map of the scene in which a mobile object such as an AGV is located based on the obtained image information and depth information, thereby realizing the scheduling and path planning of a mobile object such as an AGV in the scene.

[0062] Figures 4 to 7 are schematic diagrams illustrating the principles and processes of obtaining scene map information and constructing a scene map by an electronic device according to an embodiment of the present invention. As shown in Figure 4, there are multiple mobile objects, such as AGVs, in the scene, and each of these mobile objects explores the scene map along a specific exploration path. For example, AGV A and AGV B each advance along their respective exploration paths, continuously taking photos of the road ahead (i.e., obtaining image information) as they advance, and simultaneously labeling each frame of image information with a depth information tag as described above, thereby generating depth information for each frame of image information. Similarly, other mobile objects in the scene also collect the above-mentioned data during operation. This data can be used to train the same depth estimation model. All AGVs share this depth estimation model to generate depth maps for each location in the scene. Such depth maps are part of the environmental perception information for each location in the scene. Based on this environmental perception information, the processing circuitry of the electronic device according to an embodiment of the present invention can control the movement of the mobile object and ultimately complete the construction of the scene map.

[0063] For example, environmental perception information includes the positions and distances of obstacles, other moving objects, and moving targets around a moving object, identified based on image information and depth information. The depth estimation model can employ any suitable model known in the art for processing depth data. For example, Monodepth2, Lite-Mono models, and the like can be employed for depth estimation. The Lite-Mono model employs a continuous dilated convolution (CDC) module and a local-global feature interaction (LGFI) module. The former is used to extract rich multi-scale local features, while the latter utilizes a self-attention mechanism to encode random global information into local features. It employs convolutional neural networks (CNNs) to infer depth maps from grayscale images. Using the annotated depth, a regression CNN can be trained to predict the depth value of each pixel in the grayscale image. To highlight the main purpose of the present invention, a detailed description of the depth estimation models that can be employed by the present invention will not be provided herein. Such a depth estimation model can be stored in the memory of an electronic device for scheduling and path planning a moving object according to the present invention and can be called by the processing circuitry of the electronic device.

[0064] In the process of the mobile body exploring the entire scene map, the electronic device for scheduling and path planning of the mobile body according to the present invention can grid the scene environment passed by. For example, Figure 5 shows a schematic diagram of the process of gridding the scene by the electronic device for scheduling and path planning of the mobile body according to the present invention. As shown in Figure 5, during the movement of each AGV car, the electronic device grids the environment in units of the step distance of the wheel (that is, the distance the wheel rotates one circle) based on the acquired motion information. In other words, each grid corresponds to the distance traveled by the vehicle in one rotation. In other words, the environment in which the mobile body is located is divided into several grids representing the step units of the mobile body. In Figure 5, each square pattern, circular pattern and triangular pattern represents a grid.

[0065] During the process of gridding a scene by the processing circuit of the electronic device according to the present invention, the processing circuit can also assign scores to each adjacent grid based on the environmental perception information of the adjacent grid obtained at the corresponding grid position (including image information, depth map, timestamp, identified specific object information, etc.) to perform environmental coding. Figure 8 shows a schematic diagram of the principle of environmental coding by the electronic device according to the present invention. As shown in Figure 8, when a mobile object is at a certain grid position, the processing circuit assigns values ​​to the adjacent grids based on the following two principles based on the environmental perception information of the adjacent grids obtained at that position. First, the adjacent grids are assigned corresponding initial scores based on the distance from the set mobile target. The closer the grid is to the set target, the higher the score is assigned. The highest score can be 5, for example. Second, if there is an obstacle in the environmental perception information in a certain direction (which may also be a specific object that needs to be avoided, such as a region boundary or other mobile objects), the scores of the adjacent grids in that direction are reduced. The score reduction can be different based on the distance from the obstacle. The closer the distance to the obstacle, the greater the score reduction. The lowest score can be 0, for example. In other words, the processing circuit of the electronic device according to the present invention assigns values ​​to the adjacent grids of each grid position based on the distance to the set target and the distance to the obstacle, thereby achieving environmental encoding for each grid position. During the encoding process, the moving target of the moving object needs to be pre-set. For a specific scene environment, its boundaries are fixed. Therefore, the boundary corners of the scene are usually set as the moving targets used in the process of constructing the scene map.

[0066] Through such environmental coding, the electronic device according to the embodiment of the present invention can enable the mobile body to know the next direction of movement at each grid position. As the mobile body moves, the grid position of the mobile body changes in real time, and the coding of the adjacent grids of the grid position is also continuously updated accordingly. Since the electronic device for scheduling and path planning of mobile bodies according to the present invention adopts swarm intelligence technology, when there are multiple mobile bodies equipped with the electronic device in the application scenario, each mobile body can share the environmental perception information obtained by each mobile body. In this way, not only can new mobile bodies be added to participate in tasks in new scenes at any time, but also multiple cars can jointly build the same scene map.

[0067] FIG6 shows an example of an electronic device according to the present invention confirming the position of a new moving body when a new moving body is added to a scene. As shown in FIG6 , a new AGV C is placed in the scene. In this case, the new AGV C can share the gridding and coding information of the scene by all previous AGVs, and can quickly confirm its own position in the map based on its relative position in the current environment. For example, as shown in FIG6 , the new AGV C obtains image information at its grid position, finds the edge of the scene as a reference target, and obtains depth information as described above while moving toward the reference target, thereby obtaining its relative position relative to the reference target, and then, by sharing the environmental perception information of all previous AGVs, obtains its own position in the entire scene. In this case, the newly added AGV C can directly use the gridding information generated by other AGVs before it joins and the score coding of the adjacent grids of each grid, without the need to recalculate.

[0068] In the case where multiple mobile bodies jointly explore the scene map, in order to construct a scene map shared by multiple mobile bodies, it is necessary to grid-unify the global map coordinates for each mobile body. Figure 7 shows a schematic diagram of establishing global coordinates for an electronic device according to the present invention when there are multiple mobile bodies in a scene. As shown in Figure 7, the boundaries of each scene are known and determined, and the farthest travel position of the mobile body in two orthogonal directions (for example, the horizontal direction and the vertical direction in the figure) in a certain scene (that is, the four corners in the figure) is also determined. Therefore, any corner can be used as the reference origin of the global map coordinates. In addition, the coordinates of the coded paths of different mobile bodies are combined into a unified global map coordinate based on the intersection of the coded paths of different mobile bodies. For example, as shown in Figure 7, the cross icon shows the path intersection X of AGV trolley A and AGV trolley B. In order to combine the coordinates of the coded paths of AGV car A and AGV car B into a unified global map coordinate, first, based on the reference origin of the selected global map coordinate, the global coordinates of the intersection X relative to the reference origin are obtained by gridding AGV car A or AGV car B. Then, based on the relative coordinates of each position on the path of AGV car A and AGV car B relative to the intersection X and the global coordinates of the intersection X relative to the reference origin, the unified global map coordinates of each grid position on the path of AGV car A and AGV car B are obtained. In a similar way, each grid position on the path of all AGV cars in the scene can obtain a unified global map coordinate. By sharing the global map coordinates of the grid positions of the scene map explored by each mobile body in real time, it is possible to achieve the splicing of the scene maps explored by each mobile body and ultimately construct a complete map of the entire scene. In the complete scene map constructed in this way, all possible paths of the mobile body have been gridded, and the scene map information of each grid position contains or is associated with the environmental perception information at that position. In addition, as shown in Figures 6 and 7, each grid position can also be marked as an unobstructed position and a blocked position based on the environmental perception information of each grid position, so that the blocked position can be avoided in subsequent path planning.

[0069] In this case, as will be described in detail below, the electronic device according to the present invention carried in each mobile object such as an AGV vehicle working in the scenario can be controlled through distributed nodes to independently plan paths for each vehicle and achieve collaborative planning among multiple vehicles.

[0070] 4. Path planning and selection of mobile objects

[0071] 9 , a process of path planning and selection of a moving object performed by the electronic device according to the present invention will be described.

[0072] After the electronic device according to the present invention completes the construction of the scene map, the mobile objects in the scene can enter normal operating mode. In this case, the electronic device according to the embodiment of the present invention can sense its own position information in the global map, and the processing circuit of the electronic device has target detection and distance estimation capabilities. Therefore, when each mobile object receives a task, it can determine the mobile object that will be sent to the target to perform the task through distributed control based on its own position and the target position, and other mobile objects will not appear in the movement path of the mobile object performing the task.

[0073] Specifically, when a designated target appears in the scene, the target's location information is transmitted to the electronic devices of each mobile object. Each electronic device at a different location in the scene immediately acquires image data of its surroundings using its image sensor. By combining this acquired image data with the constructed global map data, each electronic device can determine the location of each mobile object in the scene. Next, the processing circuitry of each electronic device invokes a path planning algorithm stored in its memory to determine the shortest path from the location of the mobile object where each electronic device resides to the target. Any suitable shortest path algorithm from the prior art, such as the Dijkstra algorithm or the Bellman-Ford algorithm, can be used as the path planning algorithm. For example, the classic Dijkstra algorithm can be used. The Dijkstra algorithm is a typical shortest path algorithm proposed by Dutch computer scientist Edsger W. Dijkstra in 1959. It is used to calculate the shortest path from one node to another. Its key feature is its greedy strategy, expanding outwards from a starting point (based on a breadth-first traversal) until reaching the end point. For example, the algorithm declares an array dis to store the shortest distance from the origin to each vertex and a set of vertices for which the shortest path has been found: T = {}. Initially, the path weight of the origin s is assigned to 0 (dis[s] = 0). If there is a path (s, m) that can be directly reached from the origin s, then dis[m] is set to w(s, m), and the path lengths of all other vertices that s cannot directly reach are set to infinity. Initially, the set T contains only the origin s. Then, the minimum value is selected from the dis array, which is the shortest path from the origin s to the vertex corresponding to that value, and the point is added to T, thus completing a vertex. Then, it is necessary to check whether the newly added vertex can reach other vertices and whether the path length to other points through the newly added vertex is shorter than that of the direct path from the origin. If so, the values ​​of these vertices in dis are replaced. Then, the minimum value is found from dis again, and the above steps are repeated until T contains all the vertices of the graph. In order to highlight the main purpose of the present invention, the shortest path algorithm that can be used is not described in detail here. It should be noted that when calculating the shortest path for each electronic device, since the blocked grid locations have been marked in the global map, the calculated shortest paths are all shortest paths that avoid the marked blocked locations.

[0074] After each electronic device calculates the shortest path to the target for the mobile body in which it is located, the processing circuit of each electronic device calculates the number of grids passed in the respective shortest path based on the grid division of the scene map previously performed. Since the size of each grid corresponds to the wheel stepping distance of the AGV car, the fewer the number of grids passed, the shorter the shortest path for the AGV car to reach the target. By calculating the number of grids contained in the shortest path to the target calculated by real-time interaction, each mobile body can determine the shortest optimal path among the shortest paths of all AGV cars to the target in a distributed node control manner, and the corresponding AGV car goes to the target to perform the current task, and the paths of other AGV cars will not interfere with and affect the path of the AGV car to perform the current task (similar to the scene shown in Figures 1B to 1D). Figure 9 shows an example of an electronic device performing path planning for a mobile body according to the present invention. As shown in Figure 9, there are multiple possible paths for both AGV cars to the target. The electronic devices according to the embodiments of the present invention, carried by the two AGVs, perform the aforementioned path planning and determine the shortest paths A and B, respectively, to the destination, as indicated by the arrows in the diagram. Then, by further calculating and comparing the number of map grids traversed by shortest paths A and B, shortest path A is ultimately determined to be the shorter optimal path.

[0075] The above-mentioned path planning and selection process is carried out in a dynamic manner. That is to say, every time each AGV car passes through a map grid, the electronic device according to the embodiment of the present invention will re-perform the above-mentioned path planning and selection calculation to confirm that the path of the AGV car currently performing the task is the optimal path to the target. If the calculated optimal path changes due to the appearance of new obstacles in the map or changes in the target position, it will be immediately replaced by the mobile body on the most recently calculated optimal path to perform the corresponding task. When a new obstacle that does not exist in the current scene map appears in the scene (similar to the situation shown in Figure 1A), the electronic device of the mobile body that first discovers the obstacle updates the depth map and environmental perception information of the obstacle's location by repeatedly executing the method described above, and updates the corresponding map code. In other words, the electronic device according to the present invention also has the ability to perform real-time maintenance and updates of the scene map.

[0076] The electronic device and the mobile object equipped with the electronic device according to the present invention have been described above with reference to the accompanying drawings. The electronic device and the mobile object according to the present invention can be constructed as follows.

[0077] (1) An electronic device comprising:

[0078] a memory storing computer program code; and

[0079] The processing circuit is configured to run the code and perform the following steps:

[0080] acquiring image information of a plurality of frames of images from an image sensor mounted on a moving object;

[0081] acquiring motion information of a motion sensor of the moving object, wherein the motion information and the image information of the multiple frames of images are associated with each other through a timestamp;

[0082] fusing the motion information with the image information to obtain depth information applicable to each frame of the multiple frames; and

[0083] Scene map information of the scene where the moving object is located is obtained based on the motion information, the image information and the fused depth information.

[0084] (2)

[0085] The electronic device as described in (1) above, wherein the moving object is an AGV, and the processing circuit is configured to collect wheel step information of the AGV from a step sensor of the AGV as the motion information.

[0086] (3)

[0087] The electronic device as described in (1) or (2) above, wherein the image sensor is a monocular RGB sensor.

[0088] (4)

[0089] The electronic device as described in any one of (1) to (3) above, wherein the electronic device is the mobile object or is mounted on the mobile object.

[0090] (5)

[0091] The electronic device as described in any one of (1) to (4) above, wherein the electronic device further includes a communication interface, and

[0092] The processing circuit is further configured to control the receiving of the scene map information associated with an area passed by the other mobile body during movement from the other mobile body through the communication interface.

[0093] (6)

[0094] As described in the electronic device (5) above, wherein the processing circuit obtains the scene map information of the scene in which the mobile object is located, comprising the following steps:

[0095] Training a depth estimation model based on the depth information of each frame of image to obtain a multi-frame depth map corresponding to the multi-frame image of the scene;

[0096] Gridding the scene based on the motion information;

[0097] Acquire environmental perception information corresponding to each of the divided grid positions based on the image information and the depth map obtained at each of the divided grid positions; and

[0098] Encoding the adjacent grids of each of the grid positions according to the environmental perception information, and guiding the movement of the mobile object according to the encoding of the adjacent grids.

[0099] (7)

[0100] An electronic device as described in (6) above, wherein, in the process of the processing circuit encoding the grid in the scene, the processing circuit establishes global map coordinates shared by the two moving bodies based on the intersection of the encoding paths of two different moving bodies in the scene.

[0101] (8)

[0102] The electronic device as described in (7) above, wherein, in the process of establishing the global map coordinates, any corner of the scene is used as the reference origin of the global map coordinates.

[0103] (9)

[0104] An electronic device as described in (7) above, wherein, based on the global map coordinates, the processing circuit splices the scene map information obtained by the electronic device itself with the scene map information received from other processing circuits through the communication interface.

[0105] (10)

[0106] An electronic device as described in any one of (6) to (9) above, wherein, in the process of the processing circuit encoding the grid in the scene, the processing circuit is configured to set the encoded value according to the position of the moving target of the mobile body and the position of the obstacle in the scene.

[0107] (11)

[0108] The electronic device as described in (10) above, wherein the processing circuit is configured to:

[0109] First, the grids in the adjacent grids that are closer to the moving target are preferentially assigned larger initial scores;

[0110] Secondly, the initial score is adjusted according to the depth information of the obstacle at each of the adjacent grids, so that the initial score of the grid closer to the obstacle in the adjacent grids is reduced more.

[0111] (12)

[0112] The electronic device as described in any one of (6) to (11) above, wherein, after obtaining the scene map information of the scene in which the mobile object is located, the processing circuit is configured to execute, upon receiving a task instruction through the communication interface:

[0113] Calculating the shortest path between the current position of the mobile object and the position of the mission target; and

[0114] The number of the grids passed by the shortest path is calculated.

[0115] (13)

[0116] The electronic device as described in (12) above, wherein the shortest path is obtained by the processing circuit calling a predetermined shortest path algorithm stored in the memory for calculation.

[0117] (14)

[0118] An electronic device as described in (12) or (13) above, wherein, after calculating the shortest path, the processing circuit sends the calculated number of grids passed by the shortest path to other electronic devices in the scene through the communication interface, and receives the number of grids passed by the shortest path from other mobile bodies to the location of the task target calculated by the other electronic devices through the communication interface.

[0119] (15)

[0120] An electronic device as described in (14) above, wherein, when the number of grids passed by the shortest path calculated by the processing circuit is less than the number of grids passed by the shortest path from other mobile bodies to the position of the task target calculated by the other electronic devices received through the communication interface, the electronic device controls the mobile body to move toward the position of the task target and executes the task instruction.

[0121] (16)

[0122] An electronic device as described in any one of (12) to (15) above, wherein the scene map information includes classification information indicating whether each of the divided grid locations is unobstructed or blocked, and

[0123] The calculation of the shortest path takes the classification information into account and avoids blocked grid locations.

[0124] (17)

[0125] An electronic device as described in any one of (12) to (16) above, wherein, when a new obstacle appears in the image information of the scene acquired by the electronic device, the electronic device updates the depth estimation model and the environmental perception information, and sends the updated depth estimation model and the environmental perception information to the other electronic devices in the scene through the communication interface.

[0126] (18)

[0127] An electronic device as described in any one of (6) to (17) above, wherein the environmental perception information includes at least the image information and the depth map of obstacles, other moving objects and / or targets existing around each of the grid positions.

[0128] (19)

[0129] An electronic device as described in any one of (6) to (18) above, wherein the processing circuit is configured to use the distance of one rotation of the wheels of the moving body as the unit of the grid division when performing the grid division of the scene.

[0130] (20) A mobile object comprising:

[0131] at least one image sensor, configured to acquire image information of multiple frames of images of a scene in which the moving object is located;

[0132] A motion sensor, configured to obtain motion information of the moving object;

[0133] An electronic device, wherein the electronic device is the electronic device described in (1) to (19).

[0134] It is obvious to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. For example, the electronic device for scheduling and path planning of a mobile body according to the present invention can be mounted on the mobile body, or it can be the mobile body itself. In this case, the electronic device itself includes components such as an image sensor and a motion sensor. Therefore, from any point of view, the embodiments described herein should be regarded as exemplary and non-restrictive, and the scope of the present invention is defined by the appended claims rather than the above description, and it is intended that all changes that fall within the meaning and scope of the equivalent elements of the claims be included in the present invention. Any figure mark in the claims should not be regarded as limiting the claim to which it relates.

[0135] In addition, it should be understood that although this specification is described in terms of implementation methods, not every implementation method contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

Claims

1. An electronic device, wherein: The electronic device comprises: a memory storing computer program code; and The processing circuit is configured to run the code and perform the following steps: acquiring image information of a plurality of frames of images from an image sensor mounted on a moving object; acquiring motion information of a motion sensor of the moving object, wherein the motion information and the image information of the multiple frames of images are associated with each other through a timestamp; fusing the motion information with the image information to obtain depth information applicable to each frame of the multiple frames; and Scene map information of the scene where the moving object is located is obtained based on the motion information, the image information and the depth information.

2. The electronic device according to claim 1, wherein The mobile object is an AGV, and the processing circuit is configured to collect wheel step information of the AGV from a step sensor of the AGV as the motion information.

3. The electronic device according to claim 1, wherein The image sensor is a monocular RGB sensor.

4. The electronic device according to claim 1, wherein The electronic device is the mobile object or is mounted on the mobile object.

5. The electronic device according to any one of claims 1 to 4, wherein: The electronic device further includes a communication interface, and The processing circuit is further configured to control the receiving of the scene map information associated with an area passed by the other mobile body during movement from the other mobile body through the communication interface.

6. The electronic device according to claim 5, wherein: The processing circuit obtains the scene map information of the scene where the mobile object is located, including the following steps: Training a depth estimation model based on the depth information of each frame of image to obtain a multi-frame depth map corresponding to the multi-frame image of the scene; Gridding the scene based on the motion information; Acquire environmental perception information corresponding to each of the divided grid positions based on the image information and the depth map obtained at each of the divided grid positions; and Encoding the adjacent grids of each of the grid positions according to the environmental perception information, and guiding the movement of the mobile object according to the encoding of the adjacent grids.

7. The electronic device according to claim 6, wherein: In the process of encoding the grid in the scene by the processing circuit, the processing circuit establishes a global map coordinate shared by two different moving objects in the scene based on an intersection of the encoding paths of the two moving objects.

8. The electronic device according to claim 7, wherein: In the process of establishing the global map coordinates, the processing circuit is configured to use any corner of the scene as a reference origin of the global map coordinates.

9. The electronic device according to claim 7, wherein: Based on the global map coordinates, the processing circuit splices the scene map information obtained by the electronic device itself with the scene map information received from other processing circuits through the communication interface.

10. The electronic device according to claim 6, wherein In the process of encoding the grid in the scene by the processing circuit, the processing circuit is configured to set the encoded value according to the position of the moving target of the moving object and the position of the obstacle in the scene.

11. The electronic device according to claim 10, wherein: The processing circuit is configured to: First, the grids in the adjacent grids that are closer to the moving target are preferentially assigned larger initial scores; Secondly, the initial score is adjusted according to the depth information of the obstacle at each of the adjacent grids, so that the initial score of the grid closer to the obstacle in the adjacent grids is reduced more.

12. The electronic device according to claim 6, wherein: After obtaining the scene map information of the scene in which the mobile object is located, the processing circuit is configured to execute, upon receiving a task instruction through the communication interface: Calculating the shortest path between the current position of the mobile object and the position of the mission target; and The number of the grids passed by the shortest path is calculated.

13. The electronic device according to claim 12, wherein: The shortest path is obtained by the processing circuit calling a predetermined shortest path algorithm stored in the memory to perform calculations.

14. The electronic device according to claim 12, wherein: After calculating the shortest path, the processing circuit sends the calculated number of grids passed by the shortest path to other electronic devices in the scene through the communication interface, and receives the number of grids passed by the shortest path from other mobile objects to the location of the task target calculated by the other electronic devices through the communication interface.

15. The electronic device according to claim 14, wherein When the number of grids passed by the shortest path calculated by the processing circuit is less than the number of grids passed by the shortest path from the other mobile body to the position of the task target calculated by the other electronic device received through the communication interface, the electronic device controls the mobile body to move toward the position of the task target and executes the task instruction.

16. The electronic device according to claim 12, wherein: The scene map information includes classification information indicating whether each of the divided grid locations is unobstructed or blocked, and The calculation of the shortest path takes the classification information into account and avoids blocked grid locations.

17. The electronic device according to claim 12, wherein: When a new obstacle appears in the image information of the scene acquired by the electronic device, the electronic device updates the depth estimation model and the environmental perception information, and sends the updated depth estimation model and the environmental perception information to other electronic devices in the scene through the communication interface.

18. The electronic device according to claim 6, wherein: The environmental perception information includes at least the image information and the depth map of obstacles, other moving objects and / or targets around each of the grid positions.

19. The electronic device according to claim 6, wherein: The processing circuit is configured to use a distance corresponding to one rotation of the wheels of the moving object as a unit of the grid division when performing the grid division of the scene.

20. A mobile object, comprising: at least one image sensor, configured to acquire image information of multiple frames of images of a scene in which the moving object is located; A motion sensor, configured to obtain motion information of the moving object; as well as An electronic device, wherein the electronic device is the electronic device according to claims 1 to 19.

Images