Navigation method and system, and related device

Generate bird's-eye view and circular view by rotating the image sensor to solve the problem of repeated exploration and inefficiency in map-free navigation technology, realize all-round environmental perception and navigation path finding, and improve navigation flexibility and efficiency.

WO2024240161A1PCT designated stage expired Publication Date: 2024-11-28HUAWEI TECH CO LTD
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
PCT/CN2024/094574
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-24
Filing Date
2024-05-22
Publication Date
2024-11-28

AI Technical Summary

Technical Problem

Existing mapless navigation technology can easily lead to repeated exploration, collision and inefficiency when environmental changes are changed, and the sensor can only perceive the forward field of view, making it difficult to find navigation paths.

Method used

By rotating the image sensor to acquire images of different acquisition angles, generate bird's-eye view and circular view, determine angle information and motion information based on these images and navigation targets, adjust the acquisition angle of the image sensor and the movement direction of the navigation system, and realize all-round environmental perception and The path is found.

Benefits of technology

There is no need to load maps in advance, save memory resources, improve navigation flexibility and efficiency, avoid repeated exploration and collision, and achieve all-round environmental perception and precise navigation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2024094574_28112024_PF_FP_ABST
    Figure CN2024094574_28112024_PF_FP_ABST
Patent Text Reader

Abstract

Provided in the present application are a navigation method and system, and a related device. The method comprises the following steps: a navigation system obtaining a plurality of images after photographing a first scene from a plurality of collection angles; according to the plurality of images, determining a first aerial view and a first surround view of the first scene; in combination with the first aerial view, the first surround view and a navigation target, determining first angle information and first motion information; and finally, adjusting the collection angles according to the first angle information, and moving towards the navigation target according to the first motion information, so that the navigation system can continue to collect new images on the basis of the adjusted collection angles so as to obtain new collection angles and new motion information, and so on until reaching a position indicated by the navigation target. The navigation system can perceive all-around environment information from front, rear, left, right and top views, so as to quickly and accurately determine a navigation result, thereby preventing the problems of repeated exploration, collision and low efficiency.
Need to check novelty before this filing date? Find Prior Art

Description

A navigation method, system and related equipment

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on May 24, 2023, with application number 202310593462.8 and application name “A Navigation Method, System and Related Equipment”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of artificial intelligence (AI), and in particular to a navigation method, system, and related equipment. Background Art

[0003] The rapid development and popularization of fields such as autonomous driving navigation, robotic navigation, and drone navigation have prompted continuous exploration and innovation in navigation technology. Traditional navigation technology is typically map-based, requiring the preparation and loading of map data in advance, which takes up a large amount of memory. Furthermore, map data needs to be updated when the environment changes, limiting the flexibility of the navigation system. Consequently, mapless navigation technology has emerged. Mapless navigation refers to an environmentally aware navigation method implemented using various sensors and algorithms. This method eliminates the need to load map information in advance, conserving memory resources. Furthermore, it can navigate and avoid obstacles even when the environment changes. It offers high flexibility and adaptability, and is often used in fields such as autonomous driving, robotic navigation, drone navigation, and smart homes.

[0004] However, current map-free navigation technology can only perceive environmental information within the forward field of view, requiring the robot to run for a period of time before finding the navigation route. This leads to problems such as repeated exploration, easy collisions, and low efficiency.

[0005] Summary of the Invention

[0006] The present application provides a navigation method, system and related equipment for solving the problems of repeated exploration, easy collision and low efficiency when a robot runs for a period of time without map navigation.

[0007] In a first aspect, a navigation method is provided, comprising the following steps: a moving device acquires a plurality of first images, wherein the plurality of first images are a plurality of images obtained after the moving device instructs an image sensor to shoot a first scene from a plurality of acquisition angles; a processing device determines a first bird's-eye view and a first surrounding view of the first scene based on the plurality of first images; the processing device determines first angle information and first motion information based on the first bird's-eye view, the first surrounding view, and a navigation target, wherein the first angle information is used to instruct the image sensor to adjust the acquisition angle, and the first motion information is used to instruct the device where the moving device is located to move toward the navigation target.

[0008] The method described in the first aspect is implemented, and multiple images at different acquisition angles are obtained by rotating the image sensor. Then, a bird's-eye view and a surrounding view are obtained based on the multiple images. Then, angle information and motion information are determined based on the bird's-eye view, the surrounding view, and the navigation target. Based on the angle information, the next acquisition angle of the image sensor is adjusted. Based on the motion information, the device where the navigation system is located is instructed to move toward the navigation target, so that the image sensor can continue to acquire new images based on the adjusted acquisition angle, obtain new acquisition angles and motion information, and so on, until the position indicated by the navigation target is reached. This makes the entire navigation process unnecessary to load a map in advance, saves memory resources, and has high flexibility. In addition, the navigation system can perceive all-round environmental information such as the front, back, left, right, and overhead perspectives. The device can directly find the navigation path, avoiding repeated exploration, collisions, and inefficiency.

[0009] In one possible implementation, the multiple first images are multiple images obtained after the image sensor changes the acquisition angle, and the image sensor changes the acquisition angle as the device rotates, or the device includes a base, and the image sensor changes the acquisition angle as the base rotates.

[0010] For example, assuming that the motion device is deployed on a robot, the image sensor (such as a camera) can be installed on the head of the robot, and multiple images from different perspectives can be obtained by rotating the robot head. Alternatively, the image sensor can be installed on a rotatable base at the bottom of the robot. When the robot does not rotate, the base can be rotated so that the image sensor can rotate along the base to capture images, thereby obtaining multiple images from multiple different perspectives. For another example, assuming that the motion device is deployed on an autonomous driving vehicle, the image sensor (such as a camera) can be installed on a rotatable bracket in the autonomous driving vehicle, so that the image sensor can capture images as the bracket rotates, thereby obtaining multiple images from multiple different perspectives. For another example, a sweeping robot can also have a rotating base and a built-in camera, which captures images as the base rotates to obtain multiple images from different perspectives. It should be understood that the above examples are for illustration only and are not specifically limited in this application.

[0011] In the above implementation, the image sensor rotates with the base to change the acquisition angle. The width of the rotatable base where the image sensor is located can be smaller than the maximum width of the robot. In this way, when the robot enters a small space or a dead-end environment, the robot is constrained by obstacles and cannot rotate. By rotating the base where the image sensor is located, the robot can obtain a field of view behind it, avoiding the problem of navigation failure due to the robot being stuck in a blind spot.

[0012] In one possible implementation, the processing device inputs the first surrounding view and the first bird's-eye view into a navigation model to obtain first angle information and first motion information, wherein the navigation model includes a feature extraction network and an inference network, the feature extraction network includes a first network and a second network, the first network is used to generate a first feature based on the first surrounding view, the second network is used to generate a second feature based on the first surrounding view and the first bird's-eye view, and the inference network is used to obtain the first angle information and the first motion information based on the first feature, the second feature and the navigation target.

[0013] The above implementation method performs reasoning based on the first ring view and the first bird's-eye view, so that the navigation system can not only perceive the environmental information within the front field of view, but also perceive the environmental information in the rear, left, right and overhead perspectives, and perceive the environmental information around the device where the motion device is located (such as a robot) in all directions, so that the device where the motion device is located can directly find the navigation path, avoiding repeated exploration, collisions, and inefficiency.

[0014] In one possible implementation, the first network includes a first encoder of a first autoencoding network, the input data of the first encoder includes a first ring view, and the first feature is the output result of the first encoder; the second network includes a decoder of the first autoencoding network, an encoder of the second autoencoding network, and a decoder of the second autoencoding network, the second feature is the output result of the second decoder, and the input data of the second network includes the first feature and the first bird's-eye view.

[0015] In the above implementation, the first feature is the feature extracted by the first network from the ring view. Compared with the original ring view, the first feature can eliminate noise or other redundant information. Extracting the visual features of the bird's-eye view based on the first feature can not only make the information contained in the second feature more comprehensive, but also improve the efficiency and accuracy of the second feature extraction.

[0016] In one possible implementation, in the second network, the output end of the first decoder is connected to the input end of the second encoder, and the output end of the second encoder is connected to the input end of the second decoder, wherein the input data of the first decoder includes the first feature, and the input data of the second encoder includes the output result of the first decoder and the first bird's-eye view.

[0017] In the above implementation method, the surround view is input into the first encoder of the first network, the first encoder outputs the first feature, the first feature is input into the first decoder of the second network, the output result of the first decoder is obtained, and then the output result of the first decoder and the bird's-eye view are input into the second encoder of the second network to obtain the output result of the second encoder, and then the output result of the second encoder is input into the second decoder, and the second decoder outputs the second feature. This method uses the first feature and the bird's-eye view as the input data of the second network, so that the finally extracted second feature can have richer surround view information and bird's-eye view information, and the decision inferred based on this feature is more accurate.

[0018] In one possible implementation, the first decoder includes multiple first convolutional layers, the second encoder includes multiple second convolutional layers, and the output result of each first convolutional layer is sequentially input into each second convolutional layer in the order of the convolutional layers.

[0019] For example, the first decoder includes 3 convolutional layers, which are L11, L12 and L13 in order, and the second encoder includes 3 convolutional layers, which are L21, L22 and L23 in order. Then the output result of L11 is input into L21, the output result of L12 is input into L22, and the output result of L13 is input into L23. It should be understood that the above examples are for illustration only and this application does not make specific limitations.

[0020] It should be understood that by inputting the output results of each first convolutional layer of the first decoder into each convolutional layer of the second encoder, the second encoder can fully extract the image features in the ring view and the bird's-eye view. The use of multiple convolutional layers to refine the extracted feature information can better handle images with noise and deformation, has higher robustness, and the decisions inferred based on this feature are more accurate.

[0021] In one possible implementation, the processing device determines the first angle information and the first motion information based on the first bird's-eye view, the first surround view, the navigation target, and the sensor information. The sensor information includes information collected by sensors other than the image sensor. The other sensors include one or more of a position sensor, an acceleration sensor, a gyroscope sensor, a geomagnetic sensor, an infrared sensor, an IMU, an odometer, an angular velocity sensor, and a linear velocity sensor.

[0022] Optionally, the above-mentioned multiple sensor information also includes information obtained from third-party platforms, such as a weather information service platform that can provide relevant data for weather forecasts; a time information service platform that can provide the current time, date, holiday information, etc. of each country; a medical institution service platform that can provide notification information issued by each medical institution (such as hospital vacation notices, newly hired expert information), etc., and can also include other public data platforms that can provide information such as traffic planning data, road construction information, traffic flow data, parking lot locations, charging locations, census data, etc., which are not given as examples here; a third-party network service platform can be a service platform that provides network data, such as an instant messaging service platform, an online audio and video service platform, an online shopping platform, a logistics tracking platform, a search engine, a map service platform, a food delivery service platform, etc. It should be understood that the above-mentioned government public data platform and third-party network service platform are only used for illustration, and the specific details can be determined according to the actual application scenario, and this application does not make specific limitations.

[0023] The above implementation method combines the bird's-eye view, the surround view and the sensor information to determine the angle information and motion information, and integrates multiple aspects of information to determine the movement direction of the device and the subsequent acquisition angle, so that the navigation information generated is more accurate.

[0024] In one possible implementation, the first motion information includes a motion speed. When the motion device adjusts the acquisition angle of the image sensor according to the first angle information, the motion device moves toward the navigation target using a preset speed. When the acquisition angle of the image sensor does not need to be adjusted, the motion device moves toward the navigation target using the motion speed.

[0025] In the above implementation, the preset speed may be a relatively low speed that can ensure the quality of image acquisition, so that the images acquired by the motion device can be high-quality images, thereby improving the accuracy of subsequent navigation decisions.

[0026] In one possible implementation, the motion device instructs the image sensor to capture multiple second images, where the multiple second images are multiple images obtained by photographing the second scene using one or more target capture angles indicated by the first angle information. The processing device determines a second bird's-eye view and a second surround view of the second scene based on the multiple second images. The processing device determines the second angle information and the second motion information based on the second bird's-eye view, the second surround view, and the navigation target. The second angle information is used to instruct the image sensor to adjust the capture angle, and the second motion information is used to instruct the device where the motion device is located to move toward the navigation target.

[0027] The above implementation method periodically collects environmental information, periodically receives environmental information and infers it, obtains angle information and motion information, periodically receives angle information and motion information, and processes the angle information and motion information. Therefore, the generated motion information can be the angle information and motion information within the period. As the environmental information is periodically collected, the generated motion information can also be continuously changed. For example, the navigation route can be continuously adjusted to achieve the purpose of real-time navigation and improve the stability of the solution.

[0028] In one possible implementation, the motion device and the processing device are deployed on the same computing device, and the computing device includes any one of an autonomous vehicle, a drone, a mobile robot, and a movable smart home appliance; or, the motion device is deployed on a first computing device, and the first computing device includes any one of an autonomous vehicle, a drone, a mobile robot, and a movable smart home appliance; the processing device is deployed on a second computing device, and the second computing device includes any one of a virtual machine, a bare metal server, and a container.

[0029] For example, when the device is a vehicle, the vehicle can rotate the base of the on-board camera to collect multiple first images at multiple angles to generate a first bird's-eye view and a first surround view, and then determine the first angle information and the first motion information based on the first bird's-eye view, the first surround view and the navigation target, and then adjust the motion direction and the motion speed based on the first motion information to move toward the navigation target, and at the same time adjust the acquisition angle of the image sensor according to the first angle information to collect the second image to generate a second bird's-eye view and a second surround view, and then determine the second angle information and the second motion information based on the second bird's-eye view, the second surround view and the navigation target, and then continue to move toward the navigation target based on the second motion information, and so on, until the vehicle reaches the destination indicated by the navigation target.

[0030] To give another example, when the device is a robot, the robot can collect multiple first images by rotating itself or rotating the base of the image sensor to generate a first bird's-eye view and a first surrounding view, and then determine the first angle information and the first motion information based on the first bird's-eye view, the first surrounding view and the navigation target, and then adjust the motion direction and the motion speed based on the first motion information to move toward the navigation target, and at the same time adjust the collection angle of the image sensor according to the first angle information, collect the second image to generate a second bird's-eye view and a second surrounding view, and then determine the second angle information and the second motion information based on the second bird's-eye view, the second surrounding view and the navigation target, and then continue to move toward the navigation target based on the second motion information, and so on, until the robot reaches the destination indicated by the navigation target.

[0031] It should be understood that other types of equipment such as drones and sweeping robots can also implement the solutions described in this application according to the above examples, and they are not illustrated one by one here.

[0032] In the above implementation, the motion device and the processing device can be deployed in the same or different devices, and can be flexibly deployed according to specific application scenarios, and is applicable to a wide range of scenarios.

[0033] In one possible implementation, the first feature, the second feature, the sensor information, the navigation target, and the historical sensor information in the input data can be fused to obtain a fusion result, and then the fusion result can be input into the inference network. In a specific implementation, since the feature dimensions of the first feature, the second feature, the sensor information, the navigation target, and the historical sensor information may be different, the feature dimensions of the first feature, the second feature, the sensor information, the navigation target, and the historical sensor information can be unified first, and the high-dimensional features can be reduced in dimension. For example, the second feature is a feature extracted based on a bird's-eye view. The bird's-eye view is a two-dimensional plane perspective. Therefore, the second feature may be a two-dimensional feature, while the sensor information is usually a one-dimensional feature. Therefore, the second feature needs to be compressed and reduced in dimension before being spliced ​​and fused with the first feature. In a specific implementation, the second feature can be input into a third autoencoder network, and the compressed and reduced second feature can be obtained according to the output result of the third encoder of the third autoencoder network. Of course, the second feature can also be compressed and reduced in dimension by other methods to make it consistent with the data dimension of other input data, so that feature fusion can be performed. This application does not make specific limitations.

[0034] The above implementation method extracts and encodes features from the bird's-eye view and the surround view, enabling the fusion and inference of images, features, and sensor information in different formats, thus avoiding problems such as poor model calculation efficiency and accuracy caused by data from multiple different sources.

[0035] In a second aspect, a motion device is provided, which is applied to a navigation system. The navigation system includes a motion device and a processing device. The motion device includes: an acquisition unit, which is used to obtain multiple first images, where the multiple first images are multiple images obtained after the image sensor shoots a first scene from multiple acquisition angles. The acquisition unit is used to send the multiple first images to the processing device. The acquisition unit is used to receive first angle information sent by the processing device, and instruct the image sensor to adjust the acquisition angle according to the first angle information based on the first angle information. The motion unit is used to receive first motion information sent by the processing device, and instruct the device where the motion device is located to move toward the navigation target based on the first motion information.

[0036] The motion device described in the second aspect is implemented, and multiple images at different acquisition angles are obtained by rotating the image sensor, and then the images are sent to the processing device, so that the processing device can obtain a bird's-eye view and a surrounding view based on the multiple images, and then determine the angle information and motion information based on the bird's-eye view, the surrounding view and the navigation target. The motion device can adjust the next acquisition angle of the image sensor based on the angle information, and instruct the device where the navigation system is located to move toward the navigation target based on the motion information, so that the image sensor can continue to acquire new images based on the adjusted acquisition angle, obtain new acquisition angles and motion information, and so on, until it reaches the position indicated by the navigation target, so that the entire navigation process does not require loading a map in advance, saving memory resources and high flexibility. In addition, the navigation system can perceive all-round environmental information such as the front, back, left, right and overhead perspectives, and the device can directly find the navigation path to avoid repeated exploration, collisions and low efficiency.

[0037] In one possible implementation, the multiple first images are multiple images obtained after the image sensor changes the acquisition angle, and the image sensor changes the acquisition angle as the device rotates, or the device includes a base, and the image sensor changes the acquisition angle as the base rotates.

[0038] In one possible implementation, the first motion information includes a motion speed, a motion unit for moving toward the navigation target using a preset speed when the acquisition angle of the image sensor is adjusted according to the first angle information, and a motion unit for moving toward the navigation target using the motion speed when the acquisition angle of the image sensor does not need to be adjusted.

[0039] In one possible implementation, the acquisition unit is further used to instruct the image sensor to acquire multiple second images, where the multiple second images are multiple images obtained after shooting the second scene using one or more target acquisition angles indicated by the first angle information. The acquisition unit is used to send the multiple second images to the processing device. The acquisition unit is used to receive the second angle information sent by the processing device, and to instruct the image sensor to adjust the acquisition angle according to the second angle information based on the second angle information. The motion unit is used to receive the second motion information sent by the processing device, and to instruct the device where the motion device is located to move toward the navigation target based on the second motion information.

[0040] In one possible implementation, the motion device and the processing device are deployed on the same computing device, and the computing device includes any one of an autonomous vehicle, a drone, a mobile robot, and a movable smart home appliance; or, the motion device is deployed on a first computing device, and the first computing device includes any one of an autonomous vehicle, a drone, a mobile robot, and a movable smart home appliance; the processing device is deployed on a second computing device, and the second computing device includes any one of a virtual machine, a bare metal server, and a container.

[0041] According to a third aspect, a processing device is provided, which is applied to a navigation system. The navigation system includes a processing device and a motion device. The processing device includes: an image processing unit for receiving a plurality of first images sent by the motion device, the plurality of first images being a plurality of images obtained after the motion device instructs an image sensor to shoot a first scene from a plurality of acquisition angles; the image processing unit is further used to determine a first bird's-eye view and a first surrounding view of the first scene based on the plurality of first images; a navigation unit is used to determine first angle information and first motion information based on the first bird's-eye view, the first surrounding view and the navigation target, and to send the first angle information and the first motion information to the motion device, wherein the first angle information is used to instruct the image sensor to adjust the acquisition angle, and the first motion information is used to instruct the device where the motion device is located to move toward the navigation target.

[0042] The processing device described in the third aspect implements, after receiving the bird's-eye view and surround view sent by the motion device, obtains the bird's-eye view and surround view based on multiple images, and then determines the angle information and motion information based on the bird's-eye view, surround view and navigation target, and then sends it to the motion device, so that the motion device can adjust the next acquisition angle of the image sensor based on the angle information, and instruct the device where the navigation system is located to move toward the navigation target based on the motion information, so that the image sensor can continue to acquire new images based on the adjusted acquisition angle, obtain new acquisition angle and motion information, and so on, until it reaches the position indicated by the navigation target, so that the entire navigation process does not need to load the map in advance, saves memory resources, and has high flexibility. In addition, the navigation system can perceive all-round environmental information such as the front, back, left, right and overhead perspective, and the device can directly find the navigation path to avoid repeated exploration, collision, and inefficiency.

[0043] In one possible implementation, the multiple first images are multiple images obtained after the image sensor changes the acquisition angle, and the image sensor changes the acquisition angle as the device rotates, or the device includes a base, and the image sensor changes the acquisition angle as the base rotates.

[0044] In one possible implementation, the navigation unit is used to input the first surrounding view and the first bird's-eye view into a navigation model to obtain first angle information and first motion information, wherein the navigation model includes a feature extraction network and an inference network, the feature extraction network includes a first network and a second network, the first network is used to generate a first feature based on the first surrounding view, the second network is used to generate a second feature based on the first surrounding view and the first bird's-eye view, and the inference network is used to obtain the first angle information and the first motion information based on the first feature, the second feature and the navigation target.

[0045] In one possible implementation, the first network includes a first encoder of a first autoencoding network, the input data of the first encoder includes a first ring view, the first feature is the output result of the first encoder, the second network includes a decoder of the first autoencoding network, an encoder of the second autoencoding network, and a decoder of the second autoencoding network, the second feature is the output result of the second decoder, and the input data of the second network includes the first feature and the first bird's-eye view.

[0046] In one possible implementation, in the second network, the output end of the first decoder is connected to the input end of the second encoder, and the output end of the second encoder is connected to the input end of the second decoder, wherein the input data of the first decoder includes the first feature, and the input data of the second encoder includes the output result of the first decoder and the first bird's-eye view.

[0047] In one possible implementation, the first decoder includes multiple first convolutional layers, the second encoder includes multiple second convolutional layers, and the output result of each first convolutional layer is sequentially input into each second convolutional layer in the order of the convolutional layers.

[0048] In one possible implementation, the navigation unit is used to determine the first angle information and the first motion information based on the first bird's-eye view, the first surround view, the navigation target, and the sensor information, where the sensor information includes information collected by sensors other than the image sensor, and the other sensors include one or more of a position sensor, an acceleration sensor, a gyroscope sensor, a geomagnetic sensor, an infrared sensor, an IMU, an odometer, an angular velocity sensor, and a linear velocity sensor.

[0049] In one possible implementation, an image processing unit is used to receive multiple second images sent by a moving device, where the multiple second images are multiple images obtained by photographing a second scene using one or more target acquisition angles indicated by the first angle information. The image processing unit is also used to determine a second bird's-eye view and a second surrounding view of the second scene based on the multiple second images. The navigation unit is used to determine second angle information and second motion information based on the second bird's-eye view, the second surrounding view, and the navigation target, and send the second angle information and second motion information to the moving device, wherein the second angle information is used to instruct the image sensor to adjust the acquisition angle, and the second motion information is used to instruct the device where the moving device is located to move toward the navigation target.

[0050] In one possible implementation, the motion device and the processing device are deployed on the same computing device, and the computing device includes any one of an autonomous vehicle, a drone, a mobile robot, and a movable smart home appliance; or, the motion device is deployed on a first computing device, and the first computing device includes any one of an autonomous vehicle, a drone, a mobile robot, and a movable smart home appliance; the processing device is deployed on a second computing device, and the second computing device includes any one of a virtual machine, a bare metal server, and a container.

[0051] In a fourth aspect, a navigation system is provided, which includes a motion device and a processing device, wherein the motion device implements the functions implemented by the unit module described in the second aspect, and the processing device implements the functions implemented by the unit module described in the third aspect.

[0052] In a fifth aspect, a computing device is provided, comprising a processor and a memory, wherein the memory is used to store code, and the processor is used to execute the code to implement the method described in the first aspect.

[0053] In the sixth aspect, a computing device cluster is provided, comprising at least one first computing device and at least one second computing device, each first computing device comprising a first processor and a first memory, the first memory being used to store a first code, and the first processor being used to execute the first code to implement a unit module as described in the second aspect; each second computing device comprising a second processor and a second memory, the second memory being used to store a second code, and the second processor being used to execute the second code to implement a unit module as described in the third aspect.

[0054] In a seventh aspect, a computer-readable storage medium is provided, in which instructions are stored. When the instructions are executed by a computing device or a computing device cluster, the method described in the first aspect is implemented.

[0055] Based on the implementation methods provided in the above aspects, this application can also be further combined to provide more implementation methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] FIG1 is a schematic diagram of the architecture of a navigation system provided by the present application;

[0057] FIG2 is a deployment method of a navigation system provided by this application;

[0058] FIG3 is another deployment method of a navigation system provided by this application;

[0059] FIG4 is a schematic diagram of a flowchart of steps of a navigation method provided by the present application;

[0060] FIG5 is a schematic diagram of the navigation model structure in the navigation method provided by this application;

[0061] FIG6 is a schematic diagram of the steps for adjusting the acquisition viewing angle in the navigation method provided by the present application;

[0062] FIG7 is a schematic diagram of the structure of a computing device provided by the present application;

[0063] FIG8 is a schematic diagram of the structure of a computing device cluster provided by the present application;

[0064] FIG9 is a schematic diagram of the structure of another computing device cluster provided by the present application. DETAILED DESCRIPTION

[0065] The following describes the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0066] The rapid development and popularization of fields such as autonomous driving navigation, robot navigation, and drone navigation have prompted continuous exploration and innovation in navigation technology. Traditional navigation technology is usually map-based navigation, which requires the preparation and loading of map data in advance. When the environment changes, the map data needs to be updated, which limits the flexibility of the navigation system. Mapless navigation technology is an emerging navigation method based on environmental perception. It can achieve navigation functions without loading map information in advance. It uses various sensors (such as cameras, radars, lidars, etc.) to obtain environmental information, and realizes precise navigation and obstacle avoidance through real-time processing and analysis algorithms, thereby improving safety and efficiency. Compared with traditional map-based navigation technology, mapless navigation technology has higher flexibility, adaptability, and personalization, and can better adapt to complex and changing environments.

[0067] However, the sensors of map-free navigation technology are usually fixed-angle sensors that can only perceive environmental information within the forward field of view. As a result, robots, vehicles, or drones need to travel a certain distance before finding a new navigation path. This leads to problems such as repeated exploration, easy collisions, and low efficiency. In complex obstacle environments, it is also easy to get stuck in blind spots, resulting in navigation failure. For example, when a robot enters a narrow space under the guidance of a fixed perspective, it is constrained by obstacles and cannot switch the camera's field of view. At this time, the robot cannot switch to a new navigation path under the constant perspective, and the robot falls into a blind spot, resulting in navigation failure.

[0068] In summary, traditional map-based navigation requires offline map construction of the scene, which is cumbersome and memory-intensive. Mapless navigation technology uses environmental information obtained by sensors, which has a lot of repetition and redundancy, making it easy to repeat exploration and fall into local blind spots, resulting in poor navigation accuracy and low efficiency.

[0069] In order to solve the above-mentioned problems of poor accuracy and low efficiency of map-free navigation, the present application provides a navigation system, which obtains multiple images at different acquisition angles by rotating the image sensor, and then obtains a bird's-eye view and a surround view based on the multiple images, and then determines the angle information and motion information based on the bird's-eye view, the surround view and the navigation target, adjusts the next acquisition angle of the image sensor based on the angle information, and instructs the device where the navigation system is located to move toward the navigation target based on the motion information, so that the image sensor can continue to acquire new images based on the adjusted acquisition angle, obtain new acquisition angle and motion information, and so on, until it reaches the position indicated by the navigation target. The entire navigation process does not require loading a map in advance, saves memory resources, and has high flexibility. In addition, the navigation system can perceive all-round environmental information such as the front, back, left, right, and overhead perspective. The device can directly find the navigation path to avoid repeated exploration, collision, and low efficiency.

[0070] FIG1 is a schematic diagram of the architecture of a navigation system provided by this application. As shown in FIG1 , the navigation system 100 may include a motion device 110 and a processing device 120. A communication connection exists between the motion device 110 and the processing device 120, which may be a wired connection or a wireless connection, which is not specifically limited in this application. The number of motion devices 110 and processing devices 120 may be one or more, and one processing device 120 may establish a communication connection with one or more motion devices 110, which is not specifically limited in this application.

[0071] The motion device 110 can be deployed on a device with mobile function, which can respond to and execute navigation instructions issued by the processing device 120, such as mobile robots, vehicles, drones, smart appliances, etc., where mobile robots are robots that can move in the environment, and may include those with tracked / wheeled mobile mechanical devices, such as logistics robots, agricultural robots, inspection robots, voice guide robots, etc. Vehicles may include self-driving cars, and smart appliances may include sweeping robots. Examples are not given here one by one.

[0072] The processing device 120 can be deployed on a device with computing capabilities, such as a computing device, a terminal device, a computing device cluster, a robot, a drone, an autonomous vehicle, etc. Among them, the computing device includes a bare metal server (BMS), a virtual machine, a container or an edge computing device. Among them, BMS refers to a general physical server, such as an ARM server or an X86 server; a virtual machine refers to a complete computer system with complete hardware system functions implemented by network function virtualization (NFV) technology and simulated by software, running in a completely isolated environment; a container refers to a group of processes that are resource-constrained and isolated from each other; an edge computing device refers to a device that is closer to the data source and end user and has low latency and high bandwidth characteristics, such as an intelligent router, an edge server, etc. The processing device 120 can also be deployed in a computing device cluster, which may include multiple of the above-mentioned computing devices. Terminal devices include personal computers, smart phones, handheld processing devices, tablet computers, mobile notebooks, augmented reality (AR) devices, virtual reality (VR) devices, integrated handheld game consoles, wearable devices, vehicle-mounted devices, smart conferencing devices, smart advertising devices, smart home appliances, etc. Smart home appliances can be sweeping robots, mopping robots, etc., which are not specifically limited here.

[0073] Optionally, the motion device 110 and the processing device 120 can be deployed on different devices, and the motion device 110 and the processing device 120 are connected via a network 130. For example, as shown in FIG2 , FIG2 is a deployment method of a navigation system provided by the present application, the processing device 120 can be an edge computing device, a computing device cluster, a terminal device, etc., and the motion device 110 can be a robot, a transportation device, a drone, etc. This application does not make specific limitations. For example, the processing device 120 is a computing device cluster, such as a cloud data center, and the motion device 110 is a transportation device, such as an autonomous vehicle. For another example, the processing device 120 is a terminal device, such as a user's smartphone, and the motion device 110 is a robot, which has a mobile device. For another example, the processing device 120 is an edge computing device, and the motion device 110 is a drone. The edge computing device can be deployed on both sides of the road. It should be understood that the above examples are for illustration purposes only and are not specifically limited in this application.

[0074] Optionally, the motion device 110 and the processing device 120 may be deployed on the same device, and the motion device 110 and the processing device 120 may be connected via a bus, which may be an internal bus or an external bus, and is not specifically limited in this application. For example, FIG3 is another deployment method of a navigation system provided by this application. As shown in FIG3, the processing device 120A and the motion device 110A may be deployed on the robot 200, the processing device 120B and the motion device 110B may be deployed on the same transportation device 300, and the processing device 120C and the motion device 110C may be deployed on the same drone 400.

[0075] It should be understood that FIG. 2 and FIG. 3 are for illustration purposes only, and the motion device 110 and the processing device 120 may also be deployed on other types of equipment, which is not specifically limited in this application.

[0076] In one possible implementation, the navigation system can be implemented as software. Specifically, it can be an independent, dedicated navigation software tool. Users can download the navigation software tool to implement the functions of the navigation system shown in Figure 1. It can also be built into other software tools as a plug-in, such as a secondary open functional module or solution for robots, autonomous driving systems, or drones, or as a cloud service on a cloud platform. Users can use the navigation system by upgrading robots, autonomous driving systems, drone systems, or purchasing cloud platform services.

[0077] In one possible implementation, the above-mentioned navigation system can be deployed in the form of hardware inside a robot, an autonomous vehicle or a drone. As a part of the robot, the autonomous vehicle or the drone, users can use the navigation system by purchasing a robot, an autonomous vehicle or a drone.

[0078] Furthermore, the motion device 110 and the processing device 120 can be further divided into multiple unit modules. For example, as shown in FIG1 , the motion device 110 may include a motion unit 111 and an acquisition unit 112, and the processing device 120 may include an image processing unit 121, a feature extraction unit 122, an inference unit 123, and a training unit 124. It should be understood that FIG1 is an exemplary division method, and the motion device 110 and the processing device 120 may also include more or fewer units. For example, the feature extraction unit 122 and the inference unit 123 may be combined into a navigation unit, and the image processing unit 121 may be further divided into a bird's-eye view generation unit and a surround view generation unit. This application does not limit this.

[0079] The image processing unit 121, feature extraction unit 122, inference unit 123, and training unit 124 within the processing device 120 can be implemented via software or hardware. For example, the implementation of the image processing unit 121 within the processing device 120 will be described below using the image processing unit 121 as an example. Similarly, the implementation of the feature extraction unit 122, inference unit 123, and training unit 124 can refer to the implementation of the image processing unit 121.

[0080] The image processing unit 121 is an example of a software functional unit. The image processing unit 121 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Furthermore, the computing instance may be one or more. For example, the image processing unit 121 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one data center or multiple geographically close data centers. Typically, a region may include multiple AZs.

[0081] As an example of a hardware functional unit, the image processing unit 121 may include at least one computing device, such as a server. Alternatively, the image processing unit 121 may be implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0082] The functions of each unit module in the motion device 110 and the processing device 120 are described below.

[0083] The collection unit 112 in the motion device 110 is used to collect multiple pieces of environmental information and send the multiple pieces of environmental information to the processing device 120. The multiple pieces of environmental information are information collected by the same or different sensors on the same scene.

[0084] In a specific implementation, the acquisition unit 112 in the motion device 110 can collect multiple environmental information through sensors on the device where the motion device 110 is located, and then send the multiple environmental information to the information feature extraction unit 122 in the processing device 120. The sensors include but are not limited to image sensors, radar sensors, voice sensors, position sensors, acceleration sensors, gyroscope sensors, geomagnetic sensors, infrared sensors, inertial measurement units (IMUs), odometers, angular velocity sensors, linear velocity sensors, and the like.

[0085] Among them, an image sensor is a device used to capture images, including cameras, webcams, etc.; a voice sensor is a sensor used to capture human voice, such as a microphone, a handset, etc.; a position sensor is a sensor that measures the position or movement distance of an object, such as the global positioning system (GPS) or a trajectory sensor; an accelerometer is a sensor that can measure the acceleration of an object, which is usually used to detect the motion state of an object. For example, in self-driving cars, accelerometers can be used to detect vehicle acceleration, braking, and steering operations to help achieve accurate vehicle control and navigation. In mobile robots, accelerometers can be used to detect the robot's shaking, vibration, and tilt to support the robot's stability and safety. A gyroscope sensor is a sensor that can measure the angular velocity of an object and can be used to detect information such as the device's rotation and directional changes. A geomagnetic sensor is a sensor that can measure the strength and direction of the Earth's magnetic field and is widely used in fields such as navigation and location recognition. An infrared sensor is a sensor that can sense infrared radiation from surrounding objects and is often used in applications such as measuring temperature and detecting obstacles. An IMU is a device that integrates sensors such as accelerometers and gyroscopes to provide information about acceleration, angular velocity, and direction. An odometer is a sensor that measures the distance and direction of a robot's movement, usually determining the robot's position and direction through the rotation of its wheels or tracks. An angular velocity sensor is a sensor that can measure the rotational speed of an object around a specific axis. A linear velocity sensor is a sensor that can measure the speed of an object as it moves along a specific direction.

[0086] It should be understood that the examples of the above sensors are for illustration only. The equipment where the motion device 110 is located may also include other information acquisition sensors that can sense specified measured quantities (physical quantities, chemical quantities, state quantities, and mechanical quantities) and convert them into usable output signals according to certain rules. Examples are not given here one by one.

[0087] In one possible implementation, the multiple environmental information may include multiple images, each of which includes images captured from multiple different perspectives of the same scene. These images may be obtained by continuously changing the image sensor's capture angle. Optionally, the scene content captured by these multiple images is adjacent, with each image capturing scene content that is horizontally or vertically adjacent in spatial location. The edges and features of these multiple images can be stitched together to form a complete circumferential or panoramic view, or image processing techniques can be used to fill in gaps or discontinuities to form a complete circumferential or panoramic view.

[0088] The sensor on the device where the motion device 110 is located includes an image sensor with a variable acquisition angle. The image sensor can change the acquisition angle by rotating the device where the motion device 110 is located, or by rotating the base where the image sensor is located. For example, assuming that the motion device 110 is deployed on a robot, the image sensor (such as a camera) can be installed on the head of the robot, and multiple images from different perspectives can be obtained by rotating the robot head. Alternatively, the image sensor can be installed on a rotatable base at the bottom of the robot. When the robot is not rotating, the base can also be rotated so that the image sensor can rotate along the base to capture images, thereby obtaining multiple images from multiple different perspectives. For another example, assuming that the motion device 110 is deployed on an autonomous vehicle, the image sensor (such as a camera) can be installed on a rotatable bracket in the autonomous vehicle so that the image sensor can capture images as the bracket rotates, thereby obtaining multiple images from multiple different perspectives. For another example, a sweeping robot can also have a rotating base and a built-in camera, and capture images as the base rotates to obtain multiple images from different perspectives. It should be understood that the above examples are for illustration only and are not specifically limited in this application.

[0089] Optionally, the width of the rotatable base where the image sensor is located can be smaller than the maximum width of the robot. In this way, when the robot enters a small space or a dead end environment, the robot is constrained by obstacles and cannot rotate. By rotating the base where the image sensor is located, the robot's rear view can be obtained, avoiding the problem of navigation failure due to the robot being stuck in a blind spot.

[0090] For example, when the device is a vehicle, the vehicle can rotate the base of the on-board camera to collect multiple first images at multiple angles to generate a first bird's-eye view and a first surround view, and then determine the first angle information and the first motion information based on the first bird's-eye view, the first surround view and the navigation target, and then adjust the motion direction and the motion speed based on the first motion information to move toward the navigation target, and at the same time adjust the acquisition angle of the image sensor according to the first angle information to collect the second image to generate a second bird's-eye view and a second surround view, and then determine the second angle information and the second motion information based on the second bird's-eye view, the second surround view and the navigation target, and then continue to move toward the navigation target based on the second motion information, and so on, until the vehicle reaches the destination indicated by the navigation target.

[0091] To give another example, when the device is a robot, the robot can collect multiple first images by rotating itself or rotating the base of the image sensor to generate a first bird's-eye view and a first surrounding view, and then determine the first angle information and the first motion information based on the first bird's-eye view, the first surrounding view and the navigation target, and then adjust the motion direction and the motion speed based on the first motion information to move toward the navigation target, and at the same time adjust the collection angle of the image sensor according to the first angle information, collect the second image to generate a second bird's-eye view and a second surrounding view, and then determine the second angle information and the second motion information based on the second bird's-eye view, the second surrounding view and the navigation target, and then continue to move toward the navigation target based on the second motion information, and so on, until the robot reaches the destination indicated by the navigation target.

[0092] It should be understood that other types of equipment such as drones and sweeping robots can also implement the solutions described in this application according to the above examples, and they are not illustrated one by one here.

[0093] It should be noted that the above-mentioned multiple images can also be obtained by capturing the same scene through multiple image sensors with different capturing perspectives. That is to say, multiple image sensors with different capturing perspectives are deployed on the device where the motion device 110 is located. However, this will increase the hardware cost. Therefore, it can be selected according to the actual application scenario, and this application does not make specific limitations.

[0094] In one possible implementation, the multiple environmental information may further include multiple sensor information, including information collected by sensors other than image sensors. For example, information collected by the aforementioned voice sensor, position sensor, acceleration sensor, gyroscope sensor, geomagnetic sensor, infrared sensor, IMU, odometer, angular velocity sensor, linear velocity sensor, and the like is not specifically limited in this application.

[0095] Furthermore, the above-mentioned multiple sensor information also includes information obtained from third-party platforms, which may include government public data platforms and third-party network service platforms. Among them, the government public data platform can provide various public data, such as a weather information service platform that can provide weather forecast related data; a time information service platform that can provide the current time, date, holiday information, etc. of each country; a medical institution service platform that can provide notification information issued by each medical institution (such as hospital vacation notices, newly hired expert information), etc., and may also include other public data platforms that can provide information such as traffic planning data, road construction information, traffic flow data, parking lot locations, charging locations, census data, etc., which are not listed here one by one; the third-party network service platform can be a service platform that provides network data, such as an instant messaging service platform, an online audio and video service platform, an online shopping platform, a logistics tracking platform, a search engine, a map service platform, a food delivery service platform, etc. It should be understood that the above-mentioned government public data platform and third-party network service platform are only used for illustration, and the specific application scenarios can be determined according to the actual application scenarios, and this application does not make specific limitations.

[0096] In one possible implementation, multiple environmental information may also include a navigation target, which may include a destination. The destination may be a specific address, such as a certain address, a certain store, a certain restaurant, my home, a living room, a kitchen, etc., or it may be specific coordinates, such as global positioning system (GPS) coordinates, or it may be a directional feature of the destination, such as 100 meters east, 100 meters left, etc.; the navigation target may also include navigation conditions, such as the shortest path, the shortest time, avoiding congestion, avoiding obstacles, etc. The navigation target may also include richer content, which will not be illustrated one by one here.

[0097] The image processing unit 121 of the processing device 120 is configured to receive the multiple pieces of environmental information sent by the acquisition unit 112 and generate a surround view and a bird's-eye view according to the multiple images in the multiple pieces of environmental information.

[0098] Optionally, the surround view may be obtained by aligning and stitching multiple images in the horizontal or vertical direction. The surround view is typically an image with a wide-angle field of view. The wide-angle field of view here may be a field of view that exceeds what a camera or human eye can capture, such as a field of view exceeding 160 degrees horizontally or exceeding 175 degrees vertically. It should be understood that the wider the field of view of the surround view, the richer and more comprehensive the visual information it provides. In some embodiments, the surround view may display a scene from a 360-degree perspective.

[0099] It should be understood that if there is a gap or discontinuous gap between two images, it can be filled by image processing technology. The image processing technology here may include but is not limited to pixel interpolation, Poisson reconstruction, texture synthesis, image restoration, deep learning and other methods. For example, linear interpolation or quadratic interpolation methods can be used to calculate the value of the missing pixel based on the pixels around the missing pixel, and then fill it into the missing area, or an AI model can be trained based on a deep learning method to enable it to have the ability to predict the missing area. The above examples are for illustration only and are not specifically limited in this application.

[0100] Optionally, a bird's-eye view refers to an image that shows the whole picture of a scene or area from a bird's-eye view, similar to observing the ground from a bird's perspective. This perspective can provide a wider and more comprehensive field of view. In a specific implementation, a bird's-eye view can be an image obtained by projecting pictures of multiple perspectives onto a plane using inverse perspective mapping (IPM) technology, or it can be obtained by other technical means, such as using three-dimensional reconstruction technology to build a three-dimensional model and then rendering a bird's-eye view from a bird's-eye view. It should be understood that the above examples are for illustration only and are not specifically limited in this application. Among them, a bird's-eye view can be generated based on multiple images, or it can be generated based on an annular view. In other words, a bird's-eye view and annular view can be generated based on parallel processing of multiple images, or an annular view can be generated first and then a bird's-eye view can be generated. This application does not make specific limitations.

[0101] It can be understood that based on the bird's-eye view and the panoramic view, the visual features of the device where the motion device 110 is located under an all-round perspective can be obtained, making the visual features obtained by the motion device 110 more comprehensive and the navigation information generated thereby more accurate.

[0102] The feature extraction unit 122 of the processing device 120 is used to extract features from the surround view and the bird's-eye view, obtain a first feature and a second feature, and then send the first feature and the second feature to the inference unit 123 .

[0103] In one possible implementation, the surrounding view and the bird's-eye view can be input into a feature extraction network to obtain the first feature and the second feature. The feature extraction network includes a first network and a second network. The first network receives input data including the surrounding view and outputs data including the first feature. The second network receives input data including the bird's-eye view and outputs data including the second feature.

[0104] Optionally, the input data of the second network may also include a surrounding view, that is, the input data of the second network includes a bird's-eye view and a surrounding view, and the output data includes a second feature. It should be understood that extracting the visual features of the bird's-eye view in combination with the surrounding view can make the information contained in the second feature more comprehensive, and the navigation information generated thereby is more accurate.

[0105] In one possible implementation, the input data of the second network may include the first feature and the bird's-eye view, and the output data includes the second feature. It should be understood that the first feature is the feature extracted by the first network from the ring view. Compared with the original ring view, the first feature can exclude noise or other redundant information. Extracting the visual features of the bird's-eye view based on the first feature can not only make the information contained in the second feature more comprehensive, but also improve the efficiency and accuracy of the second feature extraction.

[0106] Optionally, the first network and the second network can be determined based on an autoencoder network (AE), wherein the autoencoder network can extract features from the input data, and these features can be used for subsequent tasks such as classification, clustering, etc., and can also represent high-dimensional input data as low-dimensional encoding to achieve data compression and dimensionality reduction. The autoencoder network can also have other functions, such as the data reconstruction autoencoder network can realize data reconstruction and denoising, and the data generation autoencoder network can realize image style conversion and other applications, which are not specifically limited in this application.

[0107] In a specific implementation, the self-encoding network may include an encoder and a decoder, wherein the encoder is used to convert the input data into a low-dimensional vector representation, and the output result of the encoder has the function set by the self-encoding network, such as dimensionality reduction, feature extraction, denoising, style conversion, etc. in the aforementioned content. The decoder is used to receive the low-dimensional vector output by the encoder and map the low-dimensional vector back to the space of the input data to obtain the output result. If the output result of the decoder is very different from the input data, it means that the output result of the encoder is incorrect and needs further optimization. In some cases, the input data may lose some information due to noise, distortion, and other problems during transmission or storage. At this time, the decoder can try to reconstruct the original data from the encoding result of the input data with these lost information. In addition, in some application scenarios, the output result of the decoder is the result after denoising or data compression, which can be understood as higher-quality input data, and can generate data in the corresponding format according to subsequent processing requirements. Therefore, the output of the decoder and the output of the encoder have their own applications. When the input data needs to be encoded and its vector representation is obtained, the encoder output should be used. When a specific format or higher-quality input data is required, the decoder output should be used. The above examples are for illustration and are not specifically limited in this application.

[0108] Optionally, the first network may include a first encoder of a first autoencoding network, and the second network may include a first decoder of the first autoencoding network, a second encoder of the second autoencoding network, and a decoder of the second autoencoding network. Input data of the first network includes a ring view, and output data includes a first feature, where the first feature is an output result of the first encoder.

[0109] In a specific implementation, the output of the first decoder in the second network is connected to the input of the second encoder, and the output of the second encoder is connected to the input of the second decoder. The input data of the first decoder includes the first feature, the input data of the second encoder includes the output of the first decoder and a bird's-eye view, the input data of the second decoder includes the output of the second encoder, and the output of the second decoder is the second feature.

[0110] Simply put, the surround view is input into the first encoder of the first network, and the first encoder outputs the first feature. The first feature is then input into the first decoder of the second network, obtaining the output of the first decoder. The output of the first decoder and the bird's-eye view are then input into the second encoder of the second network, obtaining the output of the second encoder. The output of the second encoder is then input into the second decoder, and the second decoder outputs the second feature. It should be understood that using the first feature and the bird's-eye view as input data for the second network allows the ultimately extracted second feature to contain richer surround view and bird's-eye view information, making decisions based on this feature more accurate.

[0111] Optionally, the autoencoding network can be multiple convolutional layers, the first decoder may include multiple first convolutional layers, and the second encoder may include multiple second convolutional layers. In one possible implementation, the output result of each first convolutional layer is sequentially input into multiple second convolutional layers, and there is a corresponding relationship between each first convolutional layer and the second convolutional layer. For example, the first decoder includes 3 convolutional layers, which are L11, L12 and L13 in order, and the second encoder includes 3 convolutional layers, which are L21, L22 and L23 in order. Then the output result of L11 is input into L21, the output result of L12 is input into L22, and the output result of L13 is input into L23. It should be understood that the above examples are for illustration only and are not specifically limited in this application.

[0112] It should be noted that in the above implementation, the first number of the first convolutional layers of the first decoder is the same as the second number of the second convolutional layers of the second encoder. When the first number is greater than the second number, the input data of each second convolutional layer can be the output results of one or more first convolutional layers. When the first number is less than the second number, part of the input data of the second convolutional layer is the output data of the first convolutional layer, and another part of the input data of the second convolutional layer can be the output data of the previous second convolutional layer.

[0113] It should be understood that the first feature is the output result of the first encoder, and the first feature can be a feature vector. The second feature is the output result of the second decoder. The function of the decoder is to map the vector output by the encoder back to the space of the input data. Therefore, the second feature can be a two-dimensional depth bird's-eye view. Since the input data of the second encoder includes not only a bird's-eye view, but also the output result of the first decoder, the two-dimensional depth bird's-eye view additionally contains the image features of the surrounding view compared to the bird's-eye view, and the decision inferred based on this feature is more accurate.

[0114] The inference unit 123 can obtain an inference result based on the first feature, the second feature and the navigation target, and the inference result includes angle information and motion information, wherein the acquisition angle is used for the acquisition unit 112 of the motion device 110 to adjust the acquisition angle of the image sensor and obtain multiple new images, and the motion information is used for the motion unit 111 of the motion device 110 to navigate and move to the target position.

[0115] It can be understood that the reasoning unit 123 performs reasoning based on the first feature and the second feature, and the first feature and the second feature are obtained from the bird's-eye view and the surround view, so that the navigation system can not only perceive the environmental information within the front field of view, but also perceive the environmental information in the rear, left, right and overhead perspectives, and perceive the environmental information around the device where the motion device is located (such as a robot) in all directions, so that the device where the motion device is located can directly find the navigation path, avoiding repeated exploration, collision, and inefficiency.

[0116] In one possible implementation, the inference unit 123 may obtain an inference result based on the first feature, the second feature, the sensor information, and the navigation target. It should be understood that the bird's-eye view and surround view can provide multi-faceted environmental information, and the sensor information can provide kinematic information of the device and other auxiliary reference information, enabling the navigation system to obtain more accurate inference results and improve navigation accuracy and efficiency.

[0117] In a specific implementation, the reasoning unit 123 can input the first feature, the second feature, the sensor information and the navigation target into the reasoning network to obtain the reasoning result, wherein the reasoning network may include a convolutional neural network (CNN), a recurrent neural network (RNN), a support vector machine (SVM), a decision tree (decision tree), a deep learning neural network (DNN), a long short-term memory (LSTM) network, etc., which is not specifically limited in this application.

[0118] Optionally, when the inference network is an LSTM network, the input data of the inference network also includes historical sensor information, that is, the inference unit 123 can input the first feature, the second feature, the sensor information, the navigation target and the historical sensor information into the inference network to obtain an inference result, wherein the collection time of the historical sensor information is earlier than the collection time of the above-mentioned sensor information. It should be understood that the LSTM network is a network based on historical data to capture long-term dependencies in a time series. Using the LSTM network as an inference network can take into account past historical sensor information to obtain more accurate inference results, while solving the gradient disappearance problem and improving the accuracy of the model.

[0119] Optionally, the inference unit 123 can fuse the first feature, the second feature, the sensor information, the navigation target, and the historical sensor information in the input data to obtain a fusion result, and then input the fusion result into the inference network. In a specific implementation, since the feature dimensions of the first feature, the second feature, the sensor information, the navigation target, and the historical sensor information may be different, the feature dimensions of the first feature, the second feature, the sensor information, the navigation target, and the historical sensor information can be unified first, and the high-dimensional features can be reduced in dimension. For example, the second feature is a feature extracted based on a bird's-eye view. The bird's-eye view is a top view from a two-dimensional plane perspective, so the second feature may be a two-dimensional feature, while the sensor information is usually a one-dimensional feature. Therefore, the second feature needs to be compressed and reduced in dimension, and then spliced ​​and fused with the first feature. In a specific implementation, the second feature can be input into a third autoencoder network, and the compressed and reduced second feature can be obtained according to the output result of the third encoder of the third autoencoder network. Of course, the second feature can also be compressed and reduced in dimension by other means to make it consistent with the data dimension of other input data, so that feature fusion can be performed. This application does not make specific limitations.

[0120] It should be understood that by extracting and encoding features from the bird's-eye view and the surround view, images, features, and sensor information in different formats can be fused and inferred, avoiding problems such as poor model calculation efficiency and accuracy caused by data from multiple different sources.

[0121] In one embodiment, the angle information includes one or more target acquisition angles, which are used for the acquisition unit 112 to adjust the acquisition angle of the image sensor. In a specific implementation, the target acquisition angle in the above angle information can be a specific numerical value, for example, the acquisition angle of the normal viewing angle is 0°, the acquisition angle is +120° when rotated 120 degrees to the left based on the normal viewing angle, and the acquisition angle is -120° when rotated 120 degrees to the right based on the normal viewing angle. The acquisition angle can also be replaced by a noun, for example, the acquisition angle can be a left viewing angle, which means rotating 120 degrees to the left based on the normal viewing angle, and the acquisition angle can be a right viewing angle, which means rotating 120 degrees to the right based on the normal viewing angle. It should be understood that the above examples are for illustration purposes only, and this application does not limit the specific form of the acquisition angle.

[0122] Optionally, the angle information also includes the correspondence between the acquisition sequence and the target acquisition angle. Referring to the above content, it can be seen that the image sensor can continuously change the acquisition angle to obtain multiple images. Therefore, the one or more target acquisition angles output here can be multiple acquisition angles continuously changed by the image sensor, or one or more of the multiple acquisition angles continuously changed above. Therefore, the angle information can also include the correspondence between the acquisition sequence and the target acquisition angle. For example, the angle information may include the first acquisition angle being 0°, the second acquisition angle being +120°, and the third acquisition angle being -120°.

[0123] It should be noted that the target acquisition angle in the angle information can be the acquisition angle after the image sensor is adjusted, or it can be the angle that needs to be adjusted based on the current acquisition angle. For example, if the current acquisition angle is -120° and the acquisition angle needs to be adjusted to -180°, then the target acquisition angle can be -180°, or the target acquisition angle can be -60°. It should be understood that the above examples are for illustration only and this application does not make any specific limitations.

[0124] In one embodiment, the motion information may include the driving route, direction of movement, speed, acceleration, angular velocity, etc., and may also include other information required to indicate the movement of the device. The motion information required by different devices may vary, and this application does not specifically limit this. For example, if the device is a vehicle, the vehicle can adjust the motion direction, speed, acceleration, and other information based on the motion information to move toward the navigation target; if the device is a robot, the robot can adjust the motion direction, motion mode, and speed based on the motion information to move toward the navigation target; if the device is a drone, the drone can adjust the motion direction, speed, altitude, motion posture, etc. based on the motion information, and this application does not specifically limit this.

[0125] The training unit 124 of the processing device 120 is used to train the AI ​​network using the sample set to obtain a trained feature extraction network and inference network.

[0126] In one embodiment, the feature extraction network and the inference network may be different network layers in the same navigation model, where the feature extraction network is the feature extraction layer of the navigation model and the inference network is the fully connected layer of the navigation model. The navigation model is obtained by training using a sample set.

[0127] Optionally, the sample set can be a labeled sample set, comprising input data and labels. The input data includes a known bird's-eye view, a known surrounding view, known sensor information, and a known navigation target, and the labels for the input data include known angle information and known motion information. The input data is fed into the navigation model to obtain an output result. The network parameters are adjusted based on the loss between the output result and the label until the model converges to obtain a trained navigation model.

[0128] Optionally, the sample set can be an unlabeled sample set, which includes a known bird's-eye view, a known surrounding view, known sensor information and a known navigation target. The navigation model can be implemented through a reinforcement learning algorithm. The output results of the navigation model include angle information, motion information and decision values. The decision values ​​are used to evaluate the quality of the output results. The model parameters are adjusted according to the decision values ​​and loss until the model converges to obtain a trained navigation model.

[0129] It should be noted that if the reasoning network in the navigation model is an LSTM network, the input data of the sample set may also include known historical sensor information.

[0130] It should be understood that the feature extraction network and the inference network can also be trained using other machine learning methods, which are not illustrated one by one here, and this application does not limit the training methods of the feature extraction network and the inference network.

[0131] The motion unit 111 of the motion device 110 is used to receive motion information of the inference unit 123, and the acquisition unit 112 is used to receive angle information of the inference unit, wherein the motion unit 111 can instruct the device where the motion device 110 is located to move according to the motion information, and the acquisition unit 112 can adjust the acquisition angle of the image sensor according to the angle information.

[0132] It should be noted that when the acquisition unit 112 acquires multiple images for the first time, it may acquire the images according to a preset acquisition angle, and then acquire the images according to the acquisition angle adjusted by the angle information sent by the inference unit 123 .

[0133] Optionally, when the angle information indicates that the image sensor's acquisition angle needs to be adjusted, the motion unit 111 can move toward the navigation target at a preset speed. After the acquisition schedule is adjusted, the motion unit 111 can then move at the speed specified in the motion information. The preset speed can be a relatively low speed that ensures image acquisition quality, thereby improving the quality of multiple image acquisitions.

[0134] Optionally, when the angle information shows that the acquisition angle of the image sensor does not need to be adjusted, the motion unit 111 may instruct the device where the motion apparatus 110 is located to move toward the navigation target according to the motion information.

[0135] Optionally, the acquisition unit 112 can periodically collect environmental information, the processing device 120 can periodically receive the environmental information and infer it to obtain angle information and motion information, the acquisition unit and the motion unit 111 periodically receive the angle information and motion information, and process the angle information and motion information. Therefore, the motion information generated by the inference unit can be the angle information and motion information within the period. As the environmental information is periodically collected, the generated motion information can also be continuously changed. For example, the navigation route can be continuously adjusted to achieve the purpose of real-time navigation and improve the stability of the solution.

[0136] For example, if the image collected in the first period is the first image and the image collected in the second period is the second image, then the acquisition unit 112 may first collect multiple first images in the first period, where the multiple first images are multiple images obtained after the image sensor shoots the first scene from different acquisition angles. Then, the image processing unit 121 determines the first bird's-eye view and the first surrounding view of the first scene based on the multiple first images, and the feature extraction unit 122 and the reasoning unit 123 then determine the first angle information and the first motion information based on the first bird's-eye view, the first surrounding view and the navigation target.

[0137] Next, the acquisition unit 112 may acquire multiple second images within a second period. These multiple second images are obtained by photographing the second scene using the acquisition angle indicated by the first angle information. The image processing unit 121 then determines a second bird's-eye view and a second surround view of the second scene based on the multiple second images. The feature extraction unit 122 and the inference unit 123 determine the second angle information and the second motion information based on the second bird's-eye view, the second surround view, and the navigation target. This process continues until the device 110 reaches the location indicated by the navigation target.

[0138] In summary, the navigation system provided by the present application obtains multiple images at different acquisition angles by rotating the image sensor, and then obtains a bird's-eye view and a surround view based on the multiple images, and then determines the angle information and motion information based on the bird's-eye view, the surround view and the navigation target, and adjusts the next acquisition angle of the image sensor based on the angle information, and instructs the device where the navigation system is located to move toward the navigation target based on the motion information, so that the image sensor can continue to acquire new images based on the adjusted acquisition angle, obtain new acquisition angles and motion information, and so on, until it reaches the position indicated by the navigation target. The entire navigation process does not require loading a map in advance, saves memory resources, and has high flexibility. Moreover, the navigation system can perceive all-round environmental information such as the front, back, left, right, and overhead perspectives. The device can directly find the navigation path to avoid repeated exploration, collisions, and inefficiency.

[0139] FIG4 is a flowchart of steps of a navigation method provided by the present application. The method can be applied to the navigation system shown in FIG1 to FIG3 . As shown in FIG4 , the method may include the following steps:

[0140] Step S410: The motion device 110 acquires a plurality of environmental information. This step can be implemented by the acquisition unit 112 in the embodiment of FIG1 .

[0141] Optionally, the environmental information may include multiple images, and the above multiple images can be obtained after the image sensor (such as a camera) on the motion device 110 changes the collection angle. The image sensor can change the collection angle by rotating the device where the motion device 110 is located, or by rotating the base where the image sensor is located.

[0142] Furthermore, the width of the rotatable base where the image sensor is located can be smaller than the maximum width of the robot. In this way, when the robot enters a small space or a dead-end environment, the robot is constrained by obstacles and cannot rotate. By rotating the base where the image sensor is located, the robot can obtain a field of view behind it, avoiding the problem of navigation failure due to the robot being stuck in a blind spot.

[0143] Optionally, environmental information may also include navigation targets and sensor information. Navigation targets may include a destination and navigation conditions. The destination may be a specific address, specific coordinates, or a location feature of the destination. Navigation conditions may include the shortest path, shortest time, congestion avoidance, obstacle avoidance, etc. Sensor line information may include information collected by sensors other than image sensors, and may also include information obtained from third-party platforms, such as government public data platforms and third-party network service platforms.

[0144] It should be noted that for a detailed description of the specific implementation of the motion device 110 and the environmental information, reference may be made to the relevant description of the embodiment of FIG1 , which will not be repeated here.

[0145] Step S420: The motion device 110 sends a plurality of pieces of environmental information to the processing device 120. This step can be implemented by the collection unit 112 in the embodiment of FIG1 .

[0146] Step S430: The processing device 120 generates a bird's-eye view and a surrounding view based on the multiple images. This step can be implemented by the image processing unit 121 in the embodiment of FIG1 .

[0147] Optionally, the surround view can be obtained by aligning and stitching multiple images horizontally or vertically. The surround view is typically a wide-angle image, which can exceed the field of view that can be captured by a camera or the human eye. In some embodiments, the surround view can display a scene from a 360-degree perspective. In a specific implementation, if there are gaps or discontinuous gaps between two images, they can be filled using image processing techniques. The image processing techniques here may include but are not limited to pixel interpolation, Poisson reconstruction, texture synthesis, image restoration, deep learning, and other methods, which are not specifically limited in this application.

[0148] Optionally, a bird's-eye view refers to an image that shows the entire scene or area from a bird's-eye view, similar to observing the ground from a bird's eye view. This perspective can provide a wider and more comprehensive field of view. In a specific implementation, a bird's-eye view can be an image obtained by projecting images from multiple perspectives onto a plane using IPM technology, or it can be obtained through other technical means. The bird's-eye view can be generated based on multiple images or a panoramic view, and this application does not specifically limit this.

[0149] It can be understood that based on the bird's-eye view and the panoramic view, the visual features of the device where the motion device 110 is located under an all-round perspective can be obtained, making the visual features obtained by the motion device 110 more comprehensive and the navigation information generated thereby more accurate.

[0150] It should be noted that for the specific implementation of the processing device 120 and the specific description of the bird's-eye view and the surround view, reference can be made to the relevant description in the embodiment of FIG1 , which will not be repeated here.

[0151] Step S440: The processing device 120 determines angle information and motion information based on the bird's-eye view, the surrounding view, and the navigation target. This step can be implemented by the feature extraction unit 122 and the inference unit 123 in the embodiment of FIG1 .

[0152] Optionally, the processing device 120 may input the first surrounding view and the first bird's-eye view into a navigation model to obtain first angle information and first motion information, wherein the navigation model includes a feature extraction network and an inference network, the feature extraction network includes a first network and a second network, the first network is used to generate a first feature based on the first surrounding view, the second network is used to generate a second feature based on the first surrounding view and the first bird's-eye view, and the inference network is used to obtain the first angle information and the first motion information based on the first feature, the second feature and the navigation target.

[0153] It can be understood that reasoning based on the first ring view and the first bird's-eye view allows the navigation system to perceive not only the environmental information within the front field of view, but also the environmental information in the rear, left, right and overhead perspectives, and to fully perceive the environmental information around the device where the motion device is located (such as a robot), so that the device where the motion device is located can directly find the navigation path, avoiding repeated exploration, collisions, and inefficiency.

[0154] Optionally, the input data of the second network may also include a surrounding view, that is, the input data of the second network includes a bird's-eye view and a surrounding view, and the output data includes a second feature. It should be understood that extracting the visual features of the bird's-eye view in combination with the surrounding view can make the information contained in the second feature more comprehensive, and the navigation information generated thereby is more accurate.

[0155] Optionally, the input data of the second network may include the first feature and the bird's-eye view, and the output data includes the second feature. It should be understood that the first feature is the feature extracted by the first network from the ring view. Compared with the original ring view, the first feature can exclude noise or other redundant information. Extracting the visual features of the bird's-eye view based on the first feature can not only make the information contained in the second feature more comprehensive, but also improve the efficiency and accuracy of the second feature extraction.

[0156] In a specific implementation, the first network and the second network can be determined based on an autoencoding network. The autoencoding network may include an encoder and a decoder. The description of the autoencoding network can refer to the relevant description in the embodiment of Figure 1, and will not be repeated here. Optionally, the first network may include a first encoder of the first autoencoding network, and the second network may include a first decoder of the first autoencoding network, a second encoder of the second autoencoding network, and a decoder of the second autoencoding network. The input data of the first network includes a ring view, and the output data includes a first feature, which is the output result of the first encoder.

[0157] Optionally, the output of the first decoder in the second network is connected to the input of the second encoder, and the output of the second encoder is connected to the input of the second decoder. The first decoder's input data includes the first feature, the second encoder's input data includes the output of the first decoder and a bird's-eye view, the second decoder's input data includes the output of the second encoder, and the output of the second decoder is the second feature. It should be understood that using the first feature and the bird's-eye view as input data for the second network allows the ultimately extracted second feature to contain richer surround and bird's-eye view information, making decisions based on this feature more accurate.

[0158] Optionally, the autoencoding network can be multiple convolutional layers, the first decoder may include multiple first convolutional layers, and the second encoder may include multiple second convolutional layers. In one possible implementation, the output results of each first convolutional layer are sequentially input into multiple second convolutional layers, and there is a corresponding relationship between each first convolutional layer and the second convolutional layer.

[0159] It should be noted that in the above implementation, the first number of the first convolutional layers of the first decoder is the same as the second number of the second convolutional layers of the second encoder. When the first number is greater than the second number, the input data of each second convolutional layer can be the output results of one or more first convolutional layers. When the first number is less than the second number, part of the input data of the second convolutional layer is the output data of the first convolutional layer, and another part of the input data of the second convolutional layer can be the output data of the previous second convolutional layer.

[0160] It should be understood that the first feature is the output result of the first encoder, and the first feature can be a feature vector. The second feature is the output result of the second decoder. The function of the decoder is to map the vector output by the encoder back to the space of the input data. Therefore, the second feature can be a two-dimensional depth bird's-eye view. Since the input data of the second encoder includes not only a bird's-eye view, but also the output result of the first decoder, the two-dimensional depth bird's-eye view additionally contains the image features of the surrounding view compared to the bird's-eye view, and the decision inferred based on this feature is more accurate.

[0161] In one possible implementation, an inference result can be obtained based on the first feature, the second feature, the sensor information, and the navigation target. It should be understood that the bird's-eye view and surround view can provide multi-faceted environmental information, and the sensor information can provide kinematic information of the device and other auxiliary reference information, enabling the navigation system to obtain more accurate inference results and improve navigation accuracy and efficiency.

[0162] In a specific implementation, the first feature, the second feature, the sensor information, and the navigation target can be input into an inference network to obtain an inference result. Optionally, when the inference network is an LSTM network, the input data of the inference network also includes historical sensor information, where the historical sensor information is collected earlier than the sensor information.

[0163] Optionally, the first feature, the second feature, the sensor information, the navigation target and the historical sensor information in the input data can be fused to obtain a fusion result, and then the fusion result can be input into the inference network. In a specific implementation, since the feature dimensions of the first feature, the second feature, the sensor information, the navigation target and the historical sensor information may be different, the feature dimensions of the first feature, the second feature, the sensor information, the navigation target and the historical sensor information can be unified first, and the high-dimensional features can be reduced in dimension. For example, the second feature is a feature extracted based on a bird's-eye view. The bird's-eye view is a top view from a two-dimensional plane perspective, so the second feature may be a two-dimensional feature, while the sensor information is usually a one-dimensional feature. Therefore, the second feature needs to be compressed and reduced in dimension, and then spliced ​​and fused with the first feature. In a specific implementation, the second feature can be input into a third autoencoder network, and the compressed and reduced second feature can be obtained according to the output result of the third encoder of the third autoencoder network. Of course, the second feature can also be compressed and reduced in dimension in other ways to make it consistent with the data dimension of other input data, so that feature fusion can be performed. This application does not make specific limitations.

[0164] It should be understood that by extracting and encoding features from the bird's-eye view and the surround view, images, features, and sensor information in different formats can be fused and inferred, avoiding problems such as poor model calculation efficiency and accuracy caused by data from multiple different sources.

[0165] In an embodiment of the present application, the navigation model may be obtained by training using a sample set. Specifically, before step S430, the AI ​​network may be trained using the sample set to obtain a trained navigation model.

[0166] Optionally, the feature extraction network and the inference network can be different network layers in the same navigation model, where the feature extraction network is the feature extraction layer of the navigation model and the inference network is the fully connected layer of the navigation model. The navigation model is obtained by training using a sample set.

[0167] Optionally, the sample set can be a labeled sample set, comprising input data and labels. The input data includes a known bird's-eye view, a known surrounding view, known sensor information, and a known navigation target, and the labels for the input data include known angle information and known motion information. The input data is fed into the navigation model to obtain an output result. The network parameters are adjusted based on the loss between the output result and the label until the model converges to obtain a trained navigation model.

[0168] Optionally, the sample set can be an unlabeled sample set, which includes a known bird's-eye view, a known surrounding view, known sensor information and a known navigation target. The navigation model can be implemented through a reinforcement learning algorithm. The output results of the navigation model include angle information, motion information and decision values. The decision values ​​are used to evaluate the quality of the output results. The model parameters are adjusted according to the decision values ​​and loss until the model converges to obtain a trained navigation model.

[0169] It should be noted that if the reasoning network in the navigation model is an LSTM network, the input data of the sample set may also include known historical sensor information.

[0170] It should be understood that the feature extraction network and the inference network can also be trained using other machine learning methods, which are not illustrated one by one here, and this application does not limit the training methods of the feature extraction network and the inference network.

[0171] For example, FIG5 is a schematic diagram of the navigation model structure in the navigation method provided in the present application. As shown in FIG5 , the navigation model may include a feature extraction network and an inference network.

[0172] In the example shown in Figure 5, the surround view is input into the first encoder, the first encoder outputs the first feature, the output result of the first decoder and the bird's-eye view are input into the second encoder, the second decoder outputs the second feature, and then the second feature is input into the third encoder, the third encoder outputs the reduced-dimensional second feature, and then the first feature, the reduced-dimensional second feature, the sensor information, and the navigation target are input into the inference network to obtain angle information and motion information.

[0173] It should be noted that if the inference network is an LSTM, the input data to the inference network also includes historical sensor information. During training, if the inference network is implemented using a reinforcement learning algorithm, the output of the inference network also includes a reward value, and the model parameters are adjusted based on the reward value. However, during the inference phase, the output of the inference network does not include the reward value. For details, please refer to the description in the previous embodiment and will not be repeated here.

[0174] Among them, the output result of each convolutional layer in the first decoder can be used as input data and input into each convolutional layer in the second encoder. For example, in the example shown in Figure 5, the output result of the first convolutional layer of the first decoder can be used as input data of the second convolutional layer of the second encoder, and the output result of the second convolutional layer of the first decoder can be used as input data of the third convolutional layer of the second encoder, and so on. It should be understood that Figure 5 is for illustration, and this application does not limit the number of convolutional layers in each encoder and each decoder, nor does it limit the output of each convolutional layer of the first decoder and the input of each convolutional layer of the second encoder.

[0175] It should be understood that the network structure shown in Figure 5 is for illustration purposes only. The second encoder may also directly take the annular view and the bird's-eye view as input, and replace the annular view itself with the output result of the first decoder. This is because in some cases, the annular view may lose some information due to noise, distortion, and other problems during transmission or storage. In this case, the decoder can try to reconstruct the original data. In addition, in some application scenarios, the output result of the decoder is the result of noise reduction or data compression, which can be understood as a higher-quality annular view and can generate the format required by the second decoder. Therefore, compared with the output result of the annular view and the first decoder, using the output result of the first decoder can improve the extraction efficiency and accuracy of the second feature.

[0176] It should be noted that the first feature is the output result of the first encoder, which can be a feature vector. The second feature is the output result of the second decoder. The function of the decoder is to map the vector output by the encoder back to the space of the input data. Therefore, the second feature can be a two-dimensional depth bird's-eye view. Since the input data of the second encoder includes not only the bird's-eye view, but also the output result of the first decoder, the two-dimensional depth bird's-eye view additionally contains the image features of the surrounding view compared to the bird's-eye view, and the decision inferred based on this feature is more accurate.

[0177] However, the two-dimensional depth bird's-eye view image as the second feature is high-dimensional data, and the first feature, navigation target and sensor information are low-dimensional data. Therefore, the two-dimensional depth bird's-eye view image needs to be reduced in dimensionality. Specifically, as shown in Figure 5, the second feature is input into the third encoder, and the output result of the third encoder is the second feature after dimensionality reduction. This feature can be spliced ​​and fused with the first feature, navigation target and sensor information to obtain a fusion result. Based on the fusion result, reasoning is completed to obtain motion information and angle information. By extracting and encoding features of the bird's-eye view and the surrounding view, images, features and sensor information in different formats can be fused and inferred, avoiding problems such as poor model calculation efficiency and poor accuracy caused by data from multiple different sources.

[0178] It should be understood that a connection can also be established between each convolutional layer in the first encoder and each convolutional layer in the second encoder. For example, the second convolutional layer of the first encoder can be connected to the second-to-last convolutional layer of the second encoder, the third convolutional layer of the first encoder can be connected to the third-to-last convolutional layer of the second encoder, and so on. Similarly, a connection can be established between each convolutional layer in the second encoder and each convolutional layer in the second decoder, a connection can be established between each convolutional layer in the first encoder and each convolutional layer in the first decoder, and a connection can be established between each convolutional layer in the third encoder and each convolutional layer in the third decoder. The network structure of each encoder and decoder can be determined according to the actual application scenario, and this application does not limit this.

[0179] It should be noted that the structures of the first encoder, the second encoder, and the third encoder may be the same or different, and the structures of the first decoder, the second decoder, and the third decoder may be the same or different. Figure 5 shows three encoders and decoders with the same structure for easy understanding, but this application is not limited to this.

[0180] Step S450: The processing device 120 sends the angle information and the motion information to the motion device 110. This step can be implemented by the inference unit 123 in the embodiment of FIG1 .

[0181] In one embodiment, the angle information includes one or more target acquisition angles. The target acquisition angle in the above angle information can be a specific numerical value or can be replaced by a noun. For details, please refer to the relevant examples in the embodiment of Figure 1. This application does not limit the specific form of the acquisition angle.

[0182] Optionally, the angle information also includes the correspondence between the acquisition sequence and the target acquisition angle. Referring to the above content, it can be seen that the image sensor can continuously change the acquisition angle to obtain multiple images. Therefore, the one or more target acquisition angles output here can be multiple acquisition angles continuously changed by the image sensor, or one or more of the above-mentioned multiple acquisition angles continuously changed. Therefore, the angle information can also include the correspondence between the acquisition sequence and the target acquisition angle.

[0183] In one embodiment, the motion information may include navigation route, motion direction, motion speed, motion acceleration, motion angular velocity, etc., and may also include other information required to indicate the movement of the device, which is not specifically limited in this application.

[0184] Step S460: The motion device 110 adjusts the acquisition angle according to the angle information and moves toward the navigation target according to the motion information. This step can be implemented by the acquisition unit 112 and the motion unit 111 in the embodiment of FIG1 .

[0185] It should be noted that when capturing multiple images for the first time, the images may be captured at a preset capture angle, and subsequently, the images may be captured at the capture angle adjusted according to the angle information. As previously mentioned, the image sensor acquires multiple images by rotating the base to change the capture angle. Therefore, the adjusted capture angle here may be the target capture angle indicated in the recorded angle information, so that when the image sensor acquires multiple images in the future, the multiple images are captured according to the multiple target capture angles indicated by the angle information.

[0186] Optionally, when the angle information indicates that the image sensor's acquisition angle needs to be adjusted, the camera can move toward the navigation target at a preset speed. After the acquisition schedule is adjusted, the camera can then move at the speed specified in the motion information. The preset speed can be a relatively low speed that ensures image acquisition quality, thereby improving the quality of multiple image acquisitions.

[0187] Optionally, environmental information can be collected periodically, environmental information can be received periodically and inferred to obtain angle information and motion information, angle information and motion information can be received periodically, and the angle information and motion information can be processed. Therefore, the generated motion information can be the angle information and motion information within the period. As the environmental information is collected periodically, the generated motion information can also be continuously changed. For example, the navigation route can be continuously adjusted to achieve the purpose of real-time navigation and improve the stability of the solution.

[0188] For example, if the image collected in the first period is the first image and the image collected in the second period is the second image, then multiple first images can be collected in the first period, and the multiple first images are multiple images obtained after the image sensor shoots the first scene from different collection angles. Then, based on the multiple first images, the first bird's-eye view and the first surrounding view of the first scene are determined, and then the first angle information and the first motion information are determined based on the first bird's-eye view, the first surrounding view and the navigation target.

[0189] Next, multiple second images are captured during a second period. These multiple second images are obtained by photographing the second scene using the capture angle indicated by the first angle information. A second bird's-eye view and a second surround view of the second scene are then determined based on the multiple second images. Second angle information and second motion information are then determined based on the second bird's-eye view, the second surround view, and the navigation target. This process continues in this manner until the device reaches the location indicated by the navigation target.

[0190] Step S470: Acquire multiple pieces of environmental information based on the adjusted acquisition angle. This step can be implemented by the acquisition unit 112 in the embodiment of FIG1 .

[0191] It should be noted that the adjusted acquisition angle can be multiple angles, thereby obtaining multiple images from multiple perspectives. Further, after executing step S470, step S420 can be executed to send the newly acquired multiple environmental information to the processing device 120 for processing. The processing device 120 can execute steps S430 to S450 to generate angle information and motion information and feed it back to the motion device 110. The motion device 110 can adjust the acquisition angle of the image sensor according to the new angle information, and then use the new acquisition angle to obtain multiple environmental information. Steps S420 to S470 are repeated until the device where the motion device 110 is located reaches the position indicated by the navigation target.

[0192] For example, as shown in FIG6 , which is a flowchart illustrating the steps for adjusting the capture angle in the navigation method provided herein, after receiving the angle information and motion information, the motion device 110 may execute step S460 to change the capture angle of the image sensor based on the angle information. Of these, angles 1, 3, and 4 do not require adjustment, while angle 2 needs to be rotated 15 degrees to the right. Therefore, the capture angles of angles 1 through 4 are recorded according to the angle information.

[0193] During step S470, the image sensor is rotated to capture multiple images according to the multiple capture angles indicated by the angle information. For example, FIG6 illustrates four viewing angles, where the image captured by viewing angle 1 is the left front image, the image captured by viewing angle 2 is the right front image, the image captured by viewing angle 3 is the left rear image, and the image captured by viewing angle 4 is the right rear image. FIG6 is provided for illustrative purposes only, and this application does not limit the number of multiple images or the range of the viewing angle scans.

[0194] After executing step S470, steps S420 to S430 can be executed to generate a circular view and a bird's-eye view based on multiple images, and then generate new angle information and motion information based on the circular view, the bird's-eye view and the navigation target. Then, use the acquisition angle indicated by the new angle information to capture multiple new images, and so on, until the device where the motion device 110 is located reaches the position indicated by the navigation target.

[0195] In summary, the navigation method provided by the present application obtains multiple images at different acquisition angles by rotating the image sensor, then obtains a bird's-eye view and a surrounding view based on the multiple images, and then determines the angle information and motion information based on the bird's-eye view, the surrounding view and the navigation target, adjusts the next acquisition angle of the image sensor based on the angle information, and instructs the device where the navigation system is located to move toward the navigation target based on the motion information, so that the image sensor can continue to acquire new images based on the adjusted acquisition angle, obtain new acquisition angles and motion information, and so on, until it reaches the position indicated by the navigation target. The entire navigation process does not require loading a map in advance, saves memory resources, and has high flexibility. Moreover, the navigation system can perceive all-round environmental information such as the front, back, left, right, and overhead perspectives. The device can directly find the navigation path to avoid repeated exploration, collisions, and inefficiency.

[0196] FIG7 is a schematic diagram of the structure of a computing device provided by the present application, wherein the computing device 700 may be the navigation system described above. The navigation system may be deployed on computing devices with computing capabilities and mobility requirements, such as mobile robots, vehicles, drones, smart home appliances, etc., wherein mobile robots are robots capable of moving in an environment, and may include those with tracked / wheeled mobile mechanical devices, such as logistics robots, agricultural robots, inspection robots, voice guide robots, etc., vehicles may include self-driving cars, and smart home appliances may include sweeping robots, which are not described one by one here.

[0197] Furthermore, the computing device 700 includes a processor 701, a storage unit 702, a storage medium 703 and a communication interface 704, wherein the processor 701, the storage unit 702, the storage medium 703 and the communication interface 704 communicate through a bus 705, and also communicate through other means such as wireless transmission.

[0198] The processor 701 may include any one or more processors such as a central processing unit (CPU), a microprocessor (MP), or a digital signal processor (DSP).

[0199] For example, a CPU, an NPU, or a combination of a CPU and a hardware chip. The hardware chip is an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The PLD is a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof. The processor 701 executes various types of digitally stored instructions, such as software or firmware programs stored in the storage unit 702, which enables the computing device 700 to provide a wide variety of services.

[0200] In a specific implementation, as an embodiment, the processor 701 includes one or more CPUs, such as CPU0 and CPU1 shown in FIG. 7 .

[0201] In a specific implementation, as an embodiment, computing device 700 also includes multiple processors, such as processor 701 and processor 706 shown in FIG7 . Each of these processors can be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). A processor here refers to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0202] The storage unit 702 is used to store program codes, and is controlled by the processor 701 to execute the processing steps of the navigation system in any of the embodiments shown in Figures 1 to 6. The program codes include one or more software units. The one or more software units mentioned above are the acquisition unit, motion unit, image processing unit, feature extraction unit, and inference unit in the embodiment of Figure 1, wherein the acquisition unit is used to acquire multiple first images, the image processing unit is used to determine a first bird's-eye view and a first surrounding view based on the multiple first images, the feature extraction unit is used to generate a first feature and a second feature based on the first bird's-eye view and the first surrounding view, the inference unit is used to determine first angle information and first motion information based on the first feature, the second feature, and the navigation target, the acquisition unit is used to acquire multiple second images according to multiple target acquisition angles indicated by the first angle information, and the motion unit is used to move toward the navigation target according to the first motion information. Specifically, the acquisition unit is used to execute steps S410 and S420 in the embodiment of Figure 4 and optional steps thereof, the image processing unit is used to execute step S430 in the embodiment of Figure 4 and optional steps thereof, the feature extraction unit and the inference unit are used to execute step S440 in the embodiment of Figure 4 and optional steps thereof, the inference unit is further used to execute step S450 in the embodiment of Figure 4 and optional steps thereof, the acquisition unit and the motion unit are further used to execute steps S460 and S470 in the embodiment of Figure 4 and optional steps thereof, which will not be repeated here.

[0203] The storage unit 702 includes a read-only memory and a random access memory, and provides instructions and data to the processor 701. The storage unit 702 also includes a non-volatile random access memory. The storage unit 702 is a volatile memory or a non-volatile memory, or includes both volatile and non-volatile memories. Among them, the non-volatile memory is a read-only memory (ROM), a programmable ROM (PROM), an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), or a flash memory. The volatile memory is a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are used, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronized dynamic random access memory (SLDRAM), and direct RAM bus RAM (DR RAM). A hard disk, a universal serial bus (USB), a flash memory, a secure digital memory card (SD card), a memory stick, etc., and a hard disk can be a hard disk drive (HDD), a solid state drive (SSD), a mechanical hard disk (HDD), etc., which is not specifically limited in this application.

[0204] The storage medium 703 is a carrier for storing data, such as a hard disk, a universal serial bus (USB), a flash memory, a secure digital memory card (SD card), a memory stick, etc. The hard disk can be a hard disk drive (HDD), a solid state disk (SSD), a mechanical hard disk (HDD), etc., and this application does not make specific limitations.

[0205] The communication interface 704 can be used to provide information input or output for the processor 701, or alternatively, the communication interface 704 can be used to receive data sent externally and / or send data to the outside, and can be a wired interface such as an Ethernet cable, or a wireless interface (such as Wi-Fi, Bluetooth, general wireless transmission, etc.), or alternatively, the communication interface 704 can also include a transmitter (such as a radio frequency transmitter, antenna, etc.) coupled to the interface, or a receiver, etc.

[0206] Bus 705 is a Peripheral Component Interconnect Express (PCIe) bus, an Extended Industry Standard Architecture (EISA) bus, a unified bus (Ubus or UB), a Compute Express Link (CXL), a Cache Coherent Interconnect for Accelerators (CCIX), or the like. Bus 705 is divided into an address bus, a data bus, a control bus, and the like. In addition to the data bus, bus 705 also includes a power bus, a control bus, and a status signal bus. However, for clarity, all buses are labeled as bus 705 in the figure.

[0207] It should be noted that FIG7 is only one possible implementation of the embodiment of the present application. In actual applications, the computing device 700 may also include more or fewer components, which is not limited here. For matters not shown or described in the embodiment of the present application, please refer to the relevant description in the embodiments of FIG1 to FIG6 above, and will not be repeated here.

[0208] Figure 8 is a structural diagram of a computing device cluster provided by the present application, which includes one or more first computing devices 800 and one or more second computing devices 900. The first computing device 800 and the second computing device 900 establish a communication connection through a network 1000, wherein the first computing device 800 can be deployed with the motion device 110 mentioned above, and the second computing device 900 can be deployed with the processing device 120 mentioned above. The first computing device 800 can be a computing device with mobile needs, such as a mobile robot, a vehicle, a drone, a smart appliance, etc., and the second computing device 900 can be a computing device with computing functions, such as a BMS, a virtual machine, a container, an edge server, a terminal device, etc. The network 1000 can be a wired network or a wireless network, which is not specifically limited in this application.

[0209] As shown in FIG8 , the memory 803 in each first computing device 800 may store the same instructions required for executing the motion device 110 in the navigation method. The instructions may implement one or more software units. The one or more software units are the acquisition unit and motion unit in the embodiment of FIG1 . Specifically, the acquisition unit is used to acquire multiple first images, the image processing unit is used to determine a first bird's-eye view and a first surround view based on the multiple first images, and transmit them to the second computing device 900 . The acquisition unit is also used to receive first angle information and first motion information transmitted by the second computing device 900 , and acquire multiple second images based on multiple target acquisition angles indicated by the first angle information. The motion unit is used to move toward the navigation target based on the first motion information. Specifically, the acquisition unit is used to execute steps S410 and S420 in the embodiment of FIG4 , and their optional steps. The acquisition unit and motion unit are also used to execute steps S460 and S470 in the embodiment of FIG4 , and their optional steps, which are not described in detail here.

[0210] The memory 903 in each second computing device 900 may store the same instructions required for executing the processing device 120 in the navigation method. The instructions may implement one or more software units, which are the image processing unit, feature extraction unit, reasoning unit, and training unit in the embodiment of FIG1 . The image processing unit is configured to determine a first bird's-eye view and a first surround view based on multiple first images; the feature extraction unit is configured to generate first features and second features based on the first bird's-eye view and the first surround view; the reasoning unit is configured to determine first angle information and first motion information based on the first features, the second features, and the navigation target; and the training unit is configured to train the network model required for the feature extraction unit and the reasoning unit to operate. Specifically, the image processing unit is configured to execute step S430 and its optional steps in the embodiment of FIG4 ; the feature extraction unit and the reasoning unit are configured to execute step S440 and its optional steps in the embodiment of FIG4 ; and the reasoning unit is further configured to execute step S450 and its optional steps in the embodiment of FIG4 . These steps will not be described in detail herein.

[0211] The first computing device 800 includes a processor 801, a communication interface 802, a memory 803 and a bus 804, and the second computing device 900 includes a processor 901, a communication interface 902, a memory 803 and a bus 804. For further description of the processor 801, the communication interface 802, the memory 803, the bus 804, the processor 901, the communication interface 902, the memory 903, and the bus 904, please refer to the description of the processor 701, the processor 701, the storage unit 702, the storage medium 703, the communication interface 704 and the bus 705 in the embodiment of Figure 7, which will not be repeated here.

[0212] In the implementation shown in FIG7 , each first computing device 800 in the computing cluster can establish a communication connection with one second computing device 900. That is, the multiple images captured by each first computing device 800 are processed by the corresponding second computing device 900. In another possible implementation, each first computing device 800 can establish a connection with multiple second computing devices 900. That is, the multiple images captured by each first computing device 800 are processed by the corresponding multiple second computing devices 900.

[0213] As shown in FIG8 , FIG8 is another schematic diagram of the structure of a computing device cluster provided by the present application. In the implementation shown in FIG8 , the memories 903 in different second computing devices 900 may also store different instructions, each used to execute a portion of the functions of the processing device 120 in the navigation system. In other words, the instructions stored in the memories 903 in different second computing devices 900 may be combined to implement an image processing unit, a feature extraction unit, an inference unit, and a training unit. One or more second computing devices 900 may establish a communication connection via an external network or an internal network.

[0214] As shown in Figure 9, the second computing device 900A and the second computing device 900B are connected via a network. Specifically, they are connected to the network through the communication interface in each computing device. In this type of possible implementation, the memory 903 in the second computing device 900A stores instructions for executing the functions of the image processing unit, the feature extraction unit, and the inference unit. At the same time, the memory 903 in the second computing device 900B stores instructions for executing the functions of the training unit. The functional description of each unit can be referred to the embodiment of Figure 8, and will not be repeated here.

[0215] It should be noted that the implementation method shown in Figure 9 can be an implementation method when the processing power of the second computing device 900A is insufficient, or the storage space of the second computing device 900A is insufficient, or an implementation method in other business scenarios, and this application does not make specific limitations.

[0216] Embodiments of the present application also provide a computer program product including instructions. The computer program product may be software or a program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on a computing device or a cluster of computing devices, the computing device or cluster of computing devices executes a navigation method.

[0217] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center that contains one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive). The computer-readable storage medium includes instructions that instruct a computing device to execute a navigation method, or instructs a computing device cluster to execute a navigation method.

[0218] The above embodiments can be implemented in whole or in part through software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. A computer program product includes multiple computer instructions. When the computer program instructions are loaded or executed on a computer, the processes or functions according to the embodiments of the present invention are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.

[0219] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent repairs or replacements within the technical scope disclosed in the present invention, and such repairs or replacements should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A navigation method, It is characterized in that The method is applied to a navigation system, the navigation system comprising a motion device and a processing device, and the method comprises: The motion device acquires a plurality of first images, where the plurality of first images are a plurality of images obtained after the motion device instructs an image sensor to photograph a first scene from a plurality of acquisition angles; The processing device determines a first bird's-eye view and a first surround view of the first scene based on the plurality of first images; The processing device determines first angle information and first motion information based on the first bird's-eye view, the first surrounding view and the navigation target, wherein the first angle information is used to instruct the image sensor to adjust the acquisition angle, and the first motion information is used to instruct the device where the motion device is located to move toward the navigation target.

2. The method according to claim 1, It is characterized in that The multiple first images are multiple images obtained after the image sensor changes its acquisition angle, and the image sensor changes its acquisition angle as the device rotates, or the device includes a base, and the image sensor changes its acquisition angle as the base rotates.

3. The method according to claim 1 or 2, It is characterized in that The processing device determines the first angle information and the first motion information based on the first bird's-eye view, the first surrounding view, and the navigation target, including: The processing device inputs the first surrounding view and the first bird's-eye view into a navigation model to obtain the first angle information and the first motion information, wherein the navigation model includes a feature extraction network and an inference network, the feature extraction network includes a first network and a second network, the first network is used to generate a first feature based on the first surrounding view, the second network is used to generate a second feature based on the first surrounding view and the first bird's-eye view, and the inference network is used to obtain the first angle information and the first motion information according to the first feature, the second feature and the navigation target.

4. The method according to claim 3, It is characterized in that The first network includes a first encoder of a first autoencoding network, the input data of the first encoder includes the first ring view, and the first feature is the output result of the first encoder; the second network includes a decoder of the first autoencoding network, an encoder of the second autoencoding network and a decoder of the second autoencoding network, the second feature is the output result of the second decoder, and the input data of the second network includes the first feature and the first bird's-eye view.

5. The method according to claim 4, It is characterized in that In the second network, the output end of the first decoder is connected to the input end of the second encoder, and the output end of the second encoder is connected to the input end of the second decoder, wherein the input data of the first decoder includes the first feature, and the input data of the second encoder includes the output result of the first decoder and the first bird's-eye view.

6. The method according to claim 5, It is characterized in that The first decoder includes multiple first convolutional layers, the second encoder includes multiple second convolutional layers, and the output result of each first convolutional layer is sequentially input into each second convolutional layer in the order of the convolutional layers.

7. The method according to any one of claims 1 to 6, It is characterized in that The processing device determines the first angle information and the first motion information based on the first bird's-eye view, the first surrounding view, and the navigation target, including: The processing device determines first angle information and first motion information based on the first bird's-eye view, the first surround view, the navigation target and sensor information, wherein the sensor information includes information collected by other sensors except the image sensor, and the other sensors include one or more of a position sensor, an acceleration sensor, a gyroscope sensor, a geomagnetic sensor, an infrared sensor, an IMU, an odometer, an angular velocity sensor and a linear velocity sensor.

8. The method according to any one of claims 1 to 7, It is characterized in that The first motion information includes a motion speed, and the method further includes: The motion device moves toward the navigation target at a preset speed when adjusting the acquisition angle of the image sensor according to the first angle information; When the acquisition angle of the image sensor does not need to be adjusted, the motion device moves toward the navigation target using the motion speed.

9. The method according to any one of claims 1 to 8, It is characterized in that The method further comprises: The motion device instructs the image sensor to capture a plurality of second images, where the plurality of second images are a plurality of images obtained after photographing a second scene using one or more target capture angles indicated by the first angle information; The processing device determines a second bird's-eye view and a second surrounding view of the second scene based on the plurality of second images; The processing device determines second angle information and second motion information based on the second bird's-eye view, the second surrounding view and the navigation target, the second angle information is used to instruct the image sensor to adjust the acquisition angle, and the second motion information is used to instruct the device where the motion device is located to move toward the navigation target.

10. The method according to any one of claims 1 to 9, It is characterized in that The motion device and the processing device are deployed on the same computing device, and the computing device includes any one of an autonomous driving vehicle, an unmanned aerial vehicle, a mobile robot, and a movable smart home appliance; or, The motion device is deployed on a first computing device, and the first computing device includes any one of an autonomous driving vehicle, a drone, a mobile robot, and a movable smart home appliance; The processing device is deployed on a second computing device, and the second computing device includes any one of a virtual machine, a bare metal server, and a container.

11. A sports device, It is characterized in that The motion device is applied to a navigation system, the navigation system comprises the motion device and a processing device, and the motion device comprises: A capture unit, used to acquire a plurality of first images, where the plurality of first images are a plurality of images obtained after an image sensor captures a first scene from a plurality of capture angles; The acquisition unit is used to send the multiple first images to the processing device; The acquisition unit is used to receive the first angle information sent by the processing device, and instruct the image sensor to adjust the acquisition angle according to the first angle information based on the first angle information; The motion unit is used to receive the first motion information sent by the processing device, and instruct the device where the motion device is located to move towards the navigation target based on the first motion information.

12. The exercise device according to claim 11, It is characterized in that The multiple first images are multiple images obtained after the image sensor changes its acquisition angle, and the image sensor changes its acquisition angle as the device rotates, or the device includes a base, and the image sensor changes its acquisition angle as the base rotates.

13. The exercise device according to claim 11 or 12, It is characterized in that The first motion information includes motion speed; The motion unit is configured to move toward the navigation target at a preset speed when adjusting the acquisition angle of the image sensor according to the first angle information; The motion unit is used to move toward the navigation target using the motion speed when the acquisition angle of the image sensor does not need to be adjusted.

14. A sports device according to any one of claims 11 to 13, It is characterized in that The acquisition unit is further used to instruct the image sensor to acquire a plurality of second images, where the plurality of second images are a plurality of images obtained after photographing the second scene using one or more target acquisition angles indicated by the first angle information; The acquisition unit is used to send the plurality of second images to the processing device The acquisition unit is configured to receive the second angle information sent by the processing device, and instruct the image sensor to adjust the acquisition angle according to the second angle information based on the second angle information; The motion unit is used to receive the second motion information sent by the processing device, and instruct the device where the motion device is located to move towards the navigation target based on the second motion information.

15. A sports device according to any one of claims 11 to 14, It is characterized in that The motion device and the processing device are deployed on the same computing device, and the computing device includes any one of an autonomous driving vehicle, an unmanned aerial vehicle, a mobile robot, and a movable smart home appliance; or, The motion device is deployed on a first computing device, and the first computing device includes any one of an autonomous driving vehicle, a drone, a mobile robot, and a movable smart home appliance; The processing device is deployed on a second computing device, and the second computing device includes any one of a virtual machine, a bare metal server, and a container.

16. A processing device, It is characterized in that The processing device is applied to a navigation system, the navigation system comprises the processing device and a motion device, and the processing device comprises: An image processing unit, configured to receive a plurality of first images sent by the motion device, wherein the plurality of first images are a plurality of images obtained after the motion device instructs an image sensor to photograph a first scene from a plurality of acquisition angles; The image processing unit is further used to determine a first bird's-eye view and a first surround view of the first scene based on the multiple first images; A navigation unit is used to determine first angle information and first motion information based on the first bird's-eye view, the first surrounding view and the navigation target, and send the first angle information and the first motion information to the motion device, wherein the first angle information is used to instruct the image sensor to adjust the acquisition angle, and the first motion information is used to instruct the device where the motion device is located to move toward the navigation target.

17. The processing device according to claim 16, It is characterized in that The multiple first images are multiple images obtained after the image sensor changes its acquisition angle, and the image sensor changes its acquisition angle as the device rotates, or the device includes a base, and the image sensor changes its acquisition angle as the base rotates.

18. The processing device according to claim 16 or 17, It is characterized in that The navigation unit is used to input the first surrounding view and the first bird's-eye view into a navigation model to obtain the first angle information and the first motion information, wherein the navigation model includes a feature extraction network and an inference network, the feature extraction network includes a first network and a second network, the first network is used to generate a first feature based on the first surrounding view, the second network is used to generate a second feature based on the first surrounding view and the first bird's-eye view, and the inference network is used to obtain the first angle information and the first motion information according to the first feature, the second feature and the navigation target.

19. The processing device according to claim 18, It is characterized in that The first network includes a first encoder of a first autoencoding network, the input data of the first encoder includes the first ring view, and the first feature is the output result of the first encoder; the second network includes a decoder of the first autoencoding network, an encoder of the second autoencoding network and a decoder of the second autoencoding network, the second feature is the output result of the second decoder, and the input data of the second network includes the first feature and the first bird's-eye view.

20. The processing device according to claim 19, It is characterized in that In the second network, the output end of the first decoder is connected to the input end of the second encoder, and the output end of the second encoder is connected to the input end of the second decoder, wherein the input data of the first decoder includes the first feature, and the input data of the second encoder includes the output result of the first decoder and the first bird's-eye view.

21. The processing device according to claim 20, It is characterized in that The first decoder includes multiple first convolutional layers, the second encoder includes multiple second convolutional layers, and the output result of each first convolutional layer is sequentially input into each second convolutional layer in the order of the convolutional layers.

22. A processing device according to any one of claims 16 to 21, It is characterized in that The navigation unit is used to determine first angle information and first motion information based on the first bird's-eye view, the first surround view, the navigation target and sensor information, wherein the sensor information includes information collected by other sensors except the image sensor, and the other sensors include one or more of a position sensor, an acceleration sensor, a gyroscope sensor, a geomagnetic sensor, an infrared sensor, an IMU, an odometer, an angular velocity sensor, and a linear velocity sensor.

23. A processing device according to any one of claims 16 to 22, It is characterized in that An image processing unit is configured to receive a plurality of second images sent by the motion device, wherein the plurality of second images are acquired using the first angle a plurality of images obtained after photographing the second scene at one or more target acquisition angles indicated by the degree information; The image processing unit is further used to determine a second bird's-eye view and a second surrounding view of the second scene based on the multiple second images; A navigation unit is used to determine second angle information and second motion information based on the second bird's-eye view, the second surrounding view and the navigation target, and send the second angle information and the second motion information to the motion device, wherein the second angle information is used to instruct the image sensor to adjust the acquisition angle, and the second motion information is used to instruct the device where the motion device is located to move toward the navigation target.

24. A processing device according to any one of claims 16 to 23, It is characterized in that The motion device and the processing device are deployed on the same computing device, and the computing device includes any one of an autonomous driving vehicle, an unmanned aerial vehicle, a mobile robot, and a movable smart home appliance; or, The motion device is deployed on a first computing device, and the first computing device includes any one of an autonomous driving vehicle, a drone, a mobile robot, and a movable smart home appliance; The processing device is deployed on a second computing device, and the second computing device includes any one of a virtual machine, a bare metal server, and a container.

25. A navigation system, It is characterized in that The system includes a motion device and a processing device, wherein the motion device implements the function implemented by the unit module described in any one of claims 11 to 15, and the processing device implements the function implemented by the unit module described in any one of claims 16 to 24.

26. A computing device, It is characterized in that The method comprises a processor and a memory, wherein the memory is used to store codes, and the processor is used to execute the codes to implement the method according to any one of claims 1 to 10.

27. A computing device cluster, It is characterized in that comprising at least one first computing device and at least one second computing device; Each of the first computing devices includes a first processor and a first memory, wherein the first memory is used to store a first code, and the first processor is used to execute the first code to implement the unit module described in any one of claims 11 to 15; Each of the second computing devices includes a second processor and a second memory, wherein the second memory is used to store a second code, and the second processor is used to execute the second code to implement the unit module described in any one of claims 16 to 24.

28. A computer readable storage medium, It is characterized in that The computer-readable storage medium stores instructions, and the instructions are executed by a computing device or a computing device cluster to implement the method according to any one of claims 1 to 10.

Citation Information

Patent Citations

  • Autonomous mobile beverage dispensing device, method and system

    CN107817805A

  • Panoramic look-around recording system and method for vehicle

    CN110853185A

  • Parking space detection method and device and model training device

    CN111016918A

  • Multi-view fusion scene reconstruction method, sensing network training method and device

    CN114913290A

  • Target sensing method and device, computer equipment, storage medium and program product

    CN116012805A