Navigation method, system and related equipment

By rotating the image sensor to acquire multi-angle images, generate bird's-eye view and circular view, solving the problem of repeated exploration and collision in map-free navigation, and achieving all-round environmental perception and efficient navigation path planning.

CN119022908APending Publication Date: 2024-11-26HUAWEI TECH CO LTD
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202310593462.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-24
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

Map-free navigation technology has problems such as repeated exploration, easy collision, and inefficiency, especially since the sensor can only perceive the forward field of view, it takes a while for the robot to run to find the navigation route.

Method used

By rotating the image sensor to acquire multiple images at different acquisition angles, generate aerial views and circular views, determine angle information and motion information based on these images and navigation targets, adjust the acquisition angle of the image sensor and the motion direction of the navigation system to achieve the full Azimuth environment perception and path planning.

Benefits of technology

Realize navigation without loading the map in advance, saves memory resources, improves the flexibility and efficiency of the navigation system, avoids repeated exploration and collision, and can perceive environmental information from the front, rear, left, right and top view angles, and directly finds the Navigation path.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119022908A_ABST
    Figure CN119022908A_ABST
Patent Text Reader

Abstract

The invention provides a navigation method and system and related equipment, and the method comprises the following steps: the navigation system shoots a first scene from a plurality of collection angles to obtain a plurality of images, and determines a first aerial view and a first look-around view of the first scene according to the plurality of images, the first angle information and the first motion information are determined in combination with the first aerial view, the first surround view and the navigation target, finally, the collection angle is adjusted according to the first angle information, and the navigation target is moved according to the first motion information, so that the navigation system can continue to collect a new image based on the adjusted collection angle. The navigation system can sense all-dimensional environment information such as the front, the rear, the left side, the right side and the overlook view angle, so that the navigation result is accurately and quickly determined, and the problems of repeated exploration, collision and low efficiency are avoided.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence (AI), and in particular to a navigation method, system and related equipment. Background Art

[0002] The rapid development and popularization of fields such as autonomous driving navigation, robot navigation, and drone navigation have prompted continuous exploration and innovation in navigation technology. Traditional navigation technology is usually map-based navigation, which requires the preparation and loading of map data in advance, occupies a large amount of memory, and when the environment changes, the map data needs to be updated again, which limits the flexibility of the navigation system. Therefore, mapless navigation technology came into being. Mapless navigation technology refers to a navigation method based on environmental perception that uses various sensors and algorithms. This method does not require the prior loading of map information, which can save memory resources, and can also navigate and avoid obstacles when the environment changes. It has high flexibility and adaptability and is often used in autonomous driving, robot navigation, drone navigation, smart home and other fields.

[0003] However, the current map-free navigation technology can only perceive environmental information within the forward field of vision, which means that the robot needs to run for a period of time before it can find the navigation route. This leads to problems such as repeated exploration, easy collision and low efficiency. Summary of the invention

[0004] The present application provides a navigation method, system and related equipment for solving the problems of repeated exploration, easy collision and low efficiency when a robot runs for a period of time without map navigation.

[0005] In a first aspect, a navigation method is provided, the method comprising the following steps: a moving device acquires a plurality of first images, the plurality of first images being a plurality of images obtained after the moving device instructs an image sensor to shoot a first scene from a plurality of acquisition angles, a processing device determining a first bird's-eye view and a first surrounding view of the first scene according to the plurality of first images, the processing device determining first angle information and first motion information based on the first bird's-eye view, the first surrounding view and a navigation target, wherein the first angle information is used to instruct the image sensor to adjust the acquisition angle, and the first motion information is used to instruct the device where the moving device is located to move toward the navigation target.

[0006] The method described in the first aspect is implemented, and multiple images at different acquisition angles are obtained by rotating the image sensor, and then a bird's-eye view and a surround view are obtained based on the multiple images. The angle information and motion information are determined based on the bird's-eye view, the surround view and the navigation target, and the next acquisition angle of the image sensor is adjusted based on the angle information. Based on the motion information, the device where the navigation system is located is instructed to move toward the navigation target, so that the image sensor can continue to acquire new images based on the adjusted acquisition angle, obtain new acquisition angles and motion information, and so on, until the position indicated by the navigation target is reached, so that the entire navigation process does not need to load the map in advance, saves memory resources, and has high flexibility. In addition, the navigation system can perceive all-round environmental information such as the front, back, left, right and overhead perspectives, and the device can directly find the navigation path to avoid repeated exploration, collisions, and inefficiency.

[0007] In a possible implementation, the multiple first images are multiple images obtained after the image sensor changes the acquisition angle, and the image sensor changes the acquisition angle as the device rotates, or the device includes a base, and the image sensor changes the acquisition angle as the base rotates.

[0008] For example, assuming that the motion device is deployed on a robot, an image sensor (such as a camera) can be installed on the head of the robot, and multiple images from different perspectives can be obtained by rotating the robot head. Alternatively, the image sensor can be installed on a rotatable base at the bottom of the robot. When the robot does not rotate, the base can be rotated so that the image sensor can rotate along the base to capture images, and multiple images from multiple different perspectives can be obtained. For another example, assuming that the motion device is deployed on an autonomous driving vehicle, an image sensor (such as a camera) can be installed on a rotatable bracket in the autonomous driving vehicle, so that the image sensor can capture images as the bracket rotates, and multiple images from multiple different perspectives can be obtained. For another example, a sweeping robot can also have a rotating base and a built-in camera, and capture images as the base rotates to obtain multiple images from different perspectives. It should be understood that the above examples are for illustration purposes only and are not specifically limited in this application.

[0009] In the above implementation, the image sensor changes the acquisition angle as the base rotates, and the width of the rotatable base where the image sensor is located can be smaller than the maximum width of the robot. In this way, when the robot enters a small space or a dead end environment, the robot is constrained by obstacles and cannot rotate. By rotating the base where the image sensor is located, the robot can obtain a field of view behind it, thereby avoiding the problem of navigation failure due to the robot being stuck in a blind spot.

[0010] In one possible implementation, the processing device inputs the first surrounding view and the first bird's-eye view into a navigation model to obtain first angle information and first motion information, wherein the navigation model includes a feature extraction network and an inference network, the feature extraction network includes a first network and a second network, the first network is used to generate a first feature based on the first surrounding view, the second network is used to generate a second feature based on the first surrounding view and the first bird's-eye view, and the inference network is used to obtain the first angle information and the first motion information according to the first feature, the second feature and the navigation target.

[0011] The above implementation method performs reasoning based on the first ring view and the first bird's-eye view, so that the navigation system can not only perceive the environmental information within the front field of view, but also perceive the environmental information in the rear, left, right and overhead perspectives, and fully perceive the environmental information around the equipment where the motion device is located (such as a robot), so that the equipment where the motion device is located can directly find the navigation path, avoiding repeated exploration, collision, and inefficiency.

[0012] In one possible implementation, the first network includes a first encoder of a first autoencoding network, the input data of the first encoder includes a first ring view, and the first feature is the output result of the first encoder; the second network includes a decoder of the first autoencoding network, an encoder of the second autoencoding network, and a decoder of the second autoencoding network, the second feature is the output result of the second decoder, and the input data of the second network includes the first feature and the first bird's-eye view.

[0013] In the above implementation, the first feature is the feature extracted by the first network from the ring view. Compared with the original ring view, the first feature can exclude noise or other redundant information. Extracting the visual features of the bird's-eye view based on the first feature can not only make the information contained in the second feature more comprehensive, but also improve the efficiency and accuracy of the second feature extraction.

[0014] In one possible implementation, in the second network, the output end of the first decoder is connected to the input end of the second encoder, and the output end of the second encoder is connected to the input end of the second decoder, wherein the input data of the first decoder includes the first feature, and the input data of the second encoder includes the output result of the first decoder and the first bird's-eye view.

[0015] In the above implementation, the surround view is input into the first encoder of the first network, the first encoder outputs the first feature, the first feature is input into the first decoder of the second network, the output result of the first decoder is obtained, and then the output result of the first decoder and the bird's-eye view are input into the second encoder of the second network to obtain the output result of the second encoder, and then the output result of the second encoder is input into the second decoder, and the second decoder outputs the second feature. This method uses the first feature and the bird's-eye view as the input data of the second network, so that the second feature finally extracted can have richer surround information and bird's-eye information, and the decision inferred based on this feature is more accurate.

[0016] In one possible implementation, the first decoder includes multiple first convolutional layers, the second encoder includes multiple second convolutional layers, and the output result of each first convolutional layer is sequentially input into each second convolutional layer in the order of the convolutional layers.

[0017] For example, the first decoder includes three convolutional layers, which are L11, L12 and L13 in order, and the second encoder includes three convolutional layers, which are L21, L22 and L23 in order. Then the output result of L11 is input into L21, the output result of L12 is input into L22, and the output result of L13 is input into L23. It should be understood that the above examples are for illustration only and this application does not make any specific limitations.

[0018] It should be understood that by inputting the output results of each first convolutional layer of the first decoder into each convolutional layer of the second encoder, the second encoder can fully extract the image features in the ring view and the bird's-eye view. Using multiple convolutional layers to refine the extracted feature information can better process noisy and deformed images with higher robustness, and the decisions inferred based on this feature are more accurate.

[0019] In one possible implementation, the processing device determines the first angle information and the first motion information based on the first bird's-eye view, the first surround view, the navigation target and the sensor information, where the sensor information includes information collected by sensors other than the image sensor, and the other sensors include one or more of a position sensor, an acceleration sensor, a gyroscope sensor, a geomagnetic sensor, an infrared sensor, an IMU, an odometer, an angular velocity sensor, and a linear velocity sensor.

[0020] Optionally, the above-mentioned multiple sensor information also includes information obtained from a third-party platform, such as a weather information service platform, which can provide relevant data for weather forecasts; a time information service platform, which can provide the current time, date, holiday information, etc. of each country; a medical institution service platform, which can provide notification information issued by each medical institution (such as hospital vacation notices, newly hired expert information), etc., and can also include other public data platforms that can provide information such as traffic planning data, road construction information, traffic flow data, parking lot locations, charging locations, census data, etc., which are not given examples here; a third-party network service platform can be a service platform that provides network data, such as an instant messaging service platform, an online audio and video service platform, an online shopping platform, a logistics tracking platform, a search engine, a map service platform, a takeaway service platform, etc. It should be understood that the above-mentioned government public data platform and third-party network service platform are only used for illustration, and can be determined according to the actual application scenario, and this application does not make specific limitations.

[0021] The above implementation combines the bird's-eye view, the surround view and the sensor information to determine the angle information and the motion information, and integrates various information to determine the movement direction of the device and the subsequent acquisition angle, so that the navigation information generated is more accurate.

[0022] In one possible implementation, the first motion information includes a motion speed. When the motion device adjusts the acquisition angle of the image sensor according to the first angle information, the motion device moves toward the navigation target using a preset speed. When the acquisition angle of the image sensor does not need to be adjusted, the motion device moves toward the navigation target using the motion speed.

[0023] In the above implementation, the preset speed may be a relatively low speed that can ensure the quality of image acquisition, so that the images acquired by the motion device may be high-quality images, thereby improving the accuracy of subsequent navigation decisions.

[0024] In one possible implementation, the motion device instructs the image sensor to capture multiple second images, where the multiple second images are multiple images obtained by photographing the second scene using one or more target capture angles indicated by the first angle information. The processing device determines a second bird's-eye view and a second surround view of the second scene based on the multiple second images. The processing device determines the second angle information and the second motion information based on the second bird's-eye view, the second surround view and the navigation target. The second angle information is used to instruct the image sensor to adjust the capture angle, and the second motion information is used to instruct the device where the motion device is located to move toward the navigation target.

[0025] The above implementation method periodically collects environmental information, periodically receives environmental information and infers it, obtains angle information and motion information, periodically receives angle information and motion information, and processes the angle information and motion information. Therefore, the generated motion information can be the angle information and motion information within the period. With the periodic collection of environmental information, the generated motion information can also be continuously changed. For example, the navigation route can be continuously adjusted to achieve the purpose of real-time navigation and improve the stability of the solution.

[0026] In one possible implementation, the motion device and the processing device are deployed on the same computing device, and the computing device includes any one of an autonomous driving vehicle, an unmanned aerial vehicle, a mobile robot, and a movable smart home appliance; or, the motion device is deployed on a first computing device, and the first computing device includes any one of an autonomous driving vehicle, an unmanned aerial vehicle, a mobile robot, and a movable smart home appliance; the processing device is deployed on a second computing device, and the second computing device includes any one of a virtual machine, a bare metal server, and a container.

[0027] For example, when the device is a vehicle, the vehicle can collect multiple first images at multiple angles by rotating the base of the on-board camera to generate a first bird's-eye view and a first surrounding view, and then determine first angle information and first motion information based on the first bird's-eye view, the first surrounding view, and the navigation target, and then adjust the motion direction and motion speed based on the first motion information to move toward the navigation target, and at the same time adjust the collection angle of the image sensor according to the first angle information, collect the second image to generate a second bird's-eye view and a second surrounding view, and then determine second angle information and second motion information based on the second bird's-eye view, the second surrounding view, and the navigation target, and then continue to move toward the navigation target based on the second motion information, and so on, until the vehicle reaches the destination indicated by the navigation target.

[0028] To give another example, when the device is a robot, the robot can capture multiple first images by rotating itself or rotating the base of the image sensor to generate a first bird's-eye view and a first surrounding view, and then determine first angle information and first motion information based on the first bird's-eye view, the first surrounding view, and the navigation target, and then adjust the motion direction and motion speed based on the first motion information to move toward the navigation target, and at the same time adjust the capture angle of the image sensor according to the first angle information, capture the second image to generate a second bird's-eye view and a second surrounding view, and then determine second angle information and second motion information based on the second bird's-eye view, the second surrounding view, and the navigation target, and then continue to move toward the navigation target based on the second motion information, and so on, until the robot reaches the destination indicated by the navigation target.

[0029] It should be understood that other types of devices such as drones and sweeping robots can also implement the solutions described in this application according to the above examples, and they are not illustrated one by one here.

[0030] In the above implementation, the motion device and the processing device can be deployed in the same or different devices, and can be flexibly deployed according to specific application scenarios, and is applicable to a wide range of scenarios.

[0031] In a possible implementation, the first feature, the second feature, the sensor information, the navigation target and the historical sensor information in the input data can be fused to obtain a fusion result, and then the fusion result is input into the inference network. In a specific implementation, since the feature dimensions of the first feature, the second feature, the sensor information, the navigation target and the historical sensor information may be different, the feature dimensions of the first feature, the second feature, the sensor information, the navigation target and the historical sensor information can be unified first, and the high-dimensional features can be reduced in dimension. For example, the second feature is a feature extracted based on a bird's-eye view. The bird's-eye view is a top view from a two-dimensional plane perspective, so the second feature may be a two-dimensional feature, and the sensor information is usually a one-dimensional feature, so the second feature needs to be compressed and reduced in dimension, and then spliced ​​and fused with the first feature. In a specific implementation, the second feature can be input into the third self-encoding network, and the second feature after compression and dimensionality reduction can be obtained according to the output result of the third encoder of the third self-encoding network. Of course, the second feature can also be compressed and reduced in dimension by other methods to make it consistent with the data dimension of other input data, so that feature fusion can be performed, and this application does not make specific limitations.

[0032] The above implementation method extracts and encodes features of the bird's-eye view and the surround view, so that images, features, and sensor information in different formats can be fused and inferred, avoiding problems such as poor model calculation efficiency and poor accuracy caused by data from multiple different sources.

[0033] In a second aspect, a motion device is provided, which is applied to a navigation system. The navigation system includes a motion device and a processing device. The motion device includes: an acquisition unit, which is used to obtain multiple first images, where the multiple first images are multiple images obtained by an image sensor after shooting a first scene from multiple acquisition angles; the acquisition unit is used to send the multiple first images to the processing device; the acquisition unit is used to receive first angle information sent by the processing device, and instruct the image sensor to adjust the acquisition angle according to the first angle information based on the first angle information; a motion unit is used to receive first motion information sent by the processing device, and instruct the device where the motion device is located to move toward the navigation target based on the first motion information.

[0034] The motion device described in the second aspect is implemented, and multiple images at different acquisition angles are obtained by rotating the image sensor, and then the images are sent to the processing device, so that the processing device can obtain a bird's-eye view and a surrounding view according to the multiple images, and then determine the angle information and motion information based on the bird's-eye view, the surrounding view and the navigation target. The motion device can adjust the next acquisition angle of the image sensor based on the angle information, and instruct the device where the navigation system is located to move toward the navigation target based on the motion information, so that the image sensor can continue to acquire new images based on the adjusted acquisition angle, obtain new acquisition angles and motion information, and so on, until the position indicated by the navigation target is reached, so that the entire navigation process does not need to load the map in advance, saves memory resources, and has high flexibility. In addition, the navigation system can perceive all-round environmental information such as the front, back, left, right and overhead perspective, and the device can directly find the navigation path to avoid repeated exploration, collision and low efficiency.

[0035] In a possible implementation, the multiple first images are multiple images obtained after the image sensor changes the acquisition angle, and the image sensor changes the acquisition angle as the device rotates, or the device includes a base, and the image sensor changes the acquisition angle as the base rotates.

[0036] In one possible implementation, the first motion information includes a motion speed, a motion unit for moving toward a navigation target using a preset speed when the acquisition angle of the image sensor is adjusted according to the first angle information, and a motion unit for moving toward the navigation target using the motion speed when the acquisition angle of the image sensor does not need to be adjusted.

[0037] In one possible implementation, the acquisition unit is further used to instruct the image sensor to acquire multiple second images, where the multiple second images are multiple images obtained by photographing the second scene at one or more target acquisition angles indicated by the first angle information. The acquisition unit is used to send the multiple second images to the processing device. The acquisition unit is used to receive the second angle information sent by the processing device, and to instruct the image sensor to adjust the acquisition angle according to the second angle information based on the second angle information. The motion unit is used to receive the second motion information sent by the processing device, and to instruct the device where the motion device is located to move toward the navigation target based on the second motion information.

[0038] In one possible implementation, the motion device and the processing device are deployed on the same computing device, and the computing device includes any one of an autonomous driving vehicle, an unmanned aerial vehicle, a mobile robot, and a movable smart home appliance; or, the motion device is deployed on a first computing device, and the first computing device includes any one of an autonomous driving vehicle, an unmanned aerial vehicle, a mobile robot, and a movable smart home appliance; the processing device is deployed on a second computing device, and the second computing device includes any one of a virtual machine, a bare metal server, and a container.

[0039] In a third aspect, a processing device is provided, and the processing device is applied to a navigation system. The navigation system includes a processing device and a motion device, and the processing device includes: an image processing unit, used to receive a plurality of first images sent by the motion device, the plurality of first images are a plurality of images obtained after the motion device instructs an image sensor to shoot a first scene from a plurality of acquisition angles, the image processing unit is further used to determine a first bird's-eye view and a first surrounding view of the first scene according to the plurality of first images, a navigation unit, used to determine first angle information and first motion information based on the first bird's-eye view, the first surrounding view and a navigation target, and send the first angle information and the first motion information to the motion device, wherein the first angle information is used to instruct the image sensor to adjust the acquisition angle, and the first motion information is used to instruct the device where the motion device is located to move toward the navigation target.

[0040] The processing device described in the third aspect is implemented. After receiving the bird's-eye view and the surrounding view sent by the moving device, the bird's-eye view and the surrounding view are obtained according to multiple images, and then the angle information and the motion information are determined based on the bird's-eye view, the surrounding view and the navigation target, and then the information is sent to the moving device, so that the moving device can adjust the next acquisition angle of the image sensor based on the angle information, and instruct the device where the navigation system is located to move toward the navigation target based on the motion information, so that the image sensor can continue to acquire new images based on the adjusted acquisition angle, obtain new acquisition angles and motion information, and so on, until the position indicated by the navigation target is reached, so that the entire navigation process does not need to load the map in advance, saves memory resources, and has high flexibility. In addition, the navigation system can perceive all-round environmental information such as the front, back, left, right and overhead perspective, and the device can directly find the navigation path to avoid repeated exploration, collision and low efficiency.

[0041] In a possible implementation, the multiple first images are multiple images obtained after the image sensor changes the acquisition angle, and the image sensor changes the acquisition angle as the device rotates, or the device includes a base, and the image sensor changes the acquisition angle as the base rotates.

[0042] In one possible implementation, a navigation unit is used to input a first surrounding view and a first bird's-eye view into a navigation model to obtain first angle information and first motion information, wherein the navigation model includes a feature extraction network and an inference network, the feature extraction network includes a first network and a second network, the first network is used to generate a first feature based on the first surrounding view, the second network is used to generate a second feature based on the first surrounding view and the first bird's-eye view, and the inference network is used to obtain the first angle information and the first motion information according to the first feature, the second feature and the navigation target.

[0043] In one possible implementation, the first network includes a first encoder of a first autoencoding network, the input data of the first encoder includes a first ring view, the first feature is the output result of the first encoder, the second network includes a decoder of the first autoencoding network, an encoder of the second autoencoding network and a decoder of the second autoencoding network, the second feature is the output result of the second decoder, and the input data of the second network includes the first feature and the first bird's-eye view.

[0044] In one possible implementation, in the second network, the output end of the first decoder is connected to the input end of the second encoder, and the output end of the second encoder is connected to the input end of the second decoder, wherein the input data of the first decoder includes the first feature, and the input data of the second encoder includes the output result of the first decoder and the first bird's-eye view.

[0045] In one possible implementation, the first decoder includes multiple first convolutional layers, the second encoder includes multiple second convolutional layers, and the output result of each first convolutional layer is sequentially input into each second convolutional layer in the order of the convolutional layers.

[0046] In one possible implementation, the navigation unit is used to determine the first angle information and the first motion information based on the first bird's-eye view, the first surrounding view, the navigation target and the sensor information, wherein the sensor information includes information collected by other sensors except the image sensor, and the other sensors include one or more of a position sensor, an acceleration sensor, a gyroscope sensor, a geomagnetic sensor, an infrared sensor, an IMU, an odometer, an angular velocity sensor and a linear velocity sensor.

[0047] In one possible implementation, an image processing unit is used to receive multiple second images sent by a moving device, where the multiple second images are multiple images obtained by photographing a second scene using one or more target acquisition angles indicated by the first angle information. The image processing unit is also used to determine a second bird's-eye view and a second surrounding view of the second scene based on the multiple second images. The navigation unit is used to determine second angle information and second motion information based on the second bird's-eye view, the second surrounding view and the navigation target, and send the second angle information and second motion information to the moving device, wherein the second angle information is used to instruct the image sensor to adjust the acquisition angle, and the second motion information is used to instruct the device where the moving device is located to move toward the navigation target.

[0048] In one possible implementation, the motion device and the processing device are deployed on the same computing device, and the computing device includes any one of an autonomous driving vehicle, an unmanned aerial vehicle, a mobile robot, and a movable smart home appliance; or, the motion device is deployed on a first computing device, and the first computing device includes any one of an autonomous driving vehicle, an unmanned aerial vehicle, a mobile robot, and a movable smart home appliance; the processing device is deployed on a second computing device, and the second computing device includes any one of a virtual machine, a bare metal server, and a container.

[0049] In a fourth aspect, a navigation system is provided, which includes a motion device and a processing device, wherein the motion device implements the functions implemented by the unit module described in the second aspect, and the processing device implements the functions implemented by the unit module described in the third aspect.

[0050] In a fifth aspect, a computing device is provided, including a processor and a memory, wherein the memory is used to store code, and the processor is used to execute the code to implement the method described in the first aspect.

[0051] In a sixth aspect, a computing device cluster is provided, comprising at least one first computing device and at least one second computing device, each first computing device comprising a first processor and a first memory, the first memory being used to store a first code, and the first processor being used to execute the first code to implement a unit module as described in the second aspect; each second computing device comprising a second processor and a second memory, the second memory being used to store a second code, and the second processor being used to execute the second code to implement a unit module as described in the third aspect.

[0052] In a seventh aspect, a computer-readable storage medium is provided, in which instructions are stored. When the instructions are executed by a computing device or a computing device cluster, the method described in the first aspect is implemented.

[0053] Based on the implementations provided in the above aspects, the present application can also be further combined to provide more implementations. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 It is a schematic diagram of the architecture of a navigation system provided by this application;

[0055] Figure 2 It is a deployment method of a navigation system provided by this application;

[0056] Figure 3 It is another deployment method of a navigation system provided by the present application;

[0057] Figure 4 It is a flowchart of steps of a navigation method provided by the present application;

[0058] Figure 5 is a schematic diagram of the navigation model structure in the navigation method provided by this application;

[0059] Figure 6 It is a schematic diagram of the steps of adjusting the acquisition viewing angle in the navigation method provided by the present application;

[0060] Figure 7 It is a structural schematic diagram of a computing device provided by the present application;

[0061] Figure 8 It is a structural diagram of a computing device cluster provided by this application;

[0062] Fig. 9 It is a structural diagram of another computing device cluster provided by this application. DETAILED DESCRIPTION

[0063] The technical solutions in the embodiments of the present invention will be described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0064] The rapid development and popularization of fields such as autonomous driving navigation, robot navigation, and drone navigation have prompted continuous exploration and innovation in navigation technology. Traditional navigation technology is usually map-based navigation, which requires the preparation and loading of map data in advance. When the environment changes, the map data needs to be updated again, which limits the flexibility of the navigation system. Mapless navigation technology is an emerging navigation method based on environmental perception, which can realize navigation functions without loading map information in advance. It uses various sensors (such as cameras, radars, lidars, etc.) to obtain environmental information, and realizes precise navigation and obstacle avoidance through real-time processing and analysis algorithms, thereby improving safety and efficiency. Compared with traditional map-based navigation technology, mapless navigation technology has higher flexibility, adaptability and personalization, and can better adapt to complex and changing environments.

[0065] However, the sensors of map-free navigation technology are usually fixed-angle sensors, which can only perceive environmental information within the field of view in front. As a result, robots, vehicles or drones need to run for a certain distance before finding a new navigation path. There are problems such as repeated exploration, easy collision and low efficiency. In complex obstacle environments, it is easy to get stuck in blind spots, resulting in navigation failure. For example, when a robot enters a narrow space under the guidance of a fixed perspective, it is constrained by obstacles and cannot switch the camera field of view. At this time, the robot cannot switch to a new navigation path under the constant perspective, and the robot gets stuck in a blind spot, resulting in navigation failure.

[0066] In summary, traditional map-based navigation requires offline map construction of the scene, which is cumbersome and memory-intensive. However, the environmental information obtained by sensors using map-free navigation technology has a lot of repetition and redundancy, and is prone to repeated exploration and falling into local blind spots, resulting in poor navigation accuracy and low efficiency.

[0067] In order to solve the above-mentioned problems of poor accuracy and low efficiency of map-free navigation, the present application provides a navigation system, which obtains multiple images at different acquisition angles by rotating the image sensor, then obtains a bird's-eye view and a surround view based on the multiple images, and then determines the angle information and motion information based on the bird's-eye view, the surround view and the navigation target, adjusts the next acquisition angle of the image sensor based on the angle information, and instructs the device where the navigation system is located to move toward the navigation target based on the motion information, so that the image sensor can continue to acquire new images based on the adjusted acquisition angle, obtain new acquisition angles and motion information, and so on, until it reaches the position indicated by the navigation target. The entire navigation process does not require the prior loading of a map, saves memory resources, and has high flexibility. In addition, the navigation system can perceive all-round environmental information such as the front, back, left, right, and overhead perspective. The device can directly find the navigation path to avoid repeated exploration, collision, and inefficiency.

[0068] Figure 1 This is a schematic diagram of the architecture of a navigation system provided by this application, such as Figure 1 As shown, the navigation system 100 may include a motion device 110 and a processing device 120, wherein there is a communication connection between the motion device 110 and the processing device 120, which may be a wired connection or a wireless connection, which is not specifically limited in this application. The number of the motion device 110 and the processing device 120 may be one or more, and one processing device 120 may establish a communication connection with one or more motion devices 110, which is not specifically limited in this application.

[0069] The motion device 110 can be deployed on a device with mobile function, which can respond to and execute navigation instructions issued by the processing device 120, such as a mobile robot, a vehicle, a drone, a smart appliance, etc., where a mobile robot is a robot that can move in an environment, and may include a mobile mechanical device with tracks / wheels, such as a logistics robot, an agricultural robot, an inspection robot, a voice guide robot, etc. The vehicle may include an autonomous vehicle, and the smart appliance may include a sweeping robot. Examples are not given here one by one.

[0070] The processing device 120 can be deployed on a device with computing capabilities, such as a computing device, a terminal device, a computing device cluster, a robot, a drone, an autonomous vehicle, etc. Among them, the computing device includes a bare metal server (Bare Metal Server, BMS), a virtual machine, a container or an edge computing device. Among them, BMS refers to a general physical server, such as an ARM server or an X86 server; a virtual machine refers to a complete computer system with complete hardware system functions implemented by network function virtualization (NFV) technology, simulated by software, and running in a completely isolated environment; a container refers to a group of processes that are resource-constrained and isolated from each other; an edge computing device refers to a device that is closer to the data source and the end user, and has low latency and high bandwidth characteristics, such as an intelligent router, an edge server, etc. The processing device 120 can also be deployed in a computing device cluster, which may include multiple of the above computing devices. Terminal devices include personal computers, smart phones, handheld processing devices, tablet computers, mobile notebooks, augmented reality (AR) devices, virtual reality (VR) devices, integrated handheld game consoles, wearable devices, vehicle-mounted devices, smart conference equipment, smart advertising equipment, smart home appliances, etc. Smart home appliances can be sweeping robots, mopping robots, etc., which are not specifically limited here.

[0071] Optionally, the motion device 110 and the processing device 120 may be deployed on different devices, and the motion device 110 and the processing device 120 are connected via a network 130. For example, Figure 2 As shown, Figure 2 It is a deployment method of a navigation system provided by this application. The processing device 120 can be an edge computing device, a computing device cluster, a terminal device, etc., and the motion device 110 can be a robot, a transportation device, a drone, etc. This application does not make specific limitations. For example, the processing device 120 is a computing device cluster, such as a cloud data center, and the motion device 110 is a transportation device, such as an autonomous driving vehicle. For another example, the processing device 120 is a terminal device, such as a user's smartphone, and the motion device 110 is a robot, which has a mobile device. For another example, the processing device 120 is an edge computing device, and the motion device 110 is a drone. The edge computing device can be deployed on both sides of the road. It should be understood that the above examples are for illustration and are not specifically limited in this application.

[0072] Optionally, the motion device 110 and the processing device 120 may also be deployed on the same device, and the motion device 110 and the processing device 120 are connected via a bus, which may be an internal bus or an external bus, and this application does not specifically limit this. Figure 3 This is another deployment method of a navigation system provided by the present application, such as Figure 3 As shown, the processing device 120A and the motion device 110A can be deployed on the robot 200 , the processing device 120B and the motion device 110B can be deployed on the same transportation device 300 , and the processing device 120C and the motion device 110C can be deployed on the same drone 400 .

[0073] It should be understood that Figure 2 and Figure 3 For illustration, the motion device 110 and the processing device 120 may also be deployed on other types of equipment, which is not specifically limited in this application.

[0074] In a possible implementation, the navigation system can be implemented in the form of software, specifically, an independent navigation software tool dedicated to navigation, and the user can download the navigation software tool to implement it. Figure 1 The functions of the navigation system shown in the figure can also be built into other software tools as plug-ins, such as secondary open function modules, solutions, or cloud services of robots, autonomous driving systems, or drones. Users can use the navigation system by upgrading robots, autonomous driving systems, drone systems, or purchasing cloud platform services.

[0075] In one possible implementation, the above-mentioned navigation system can be deployed in a robot, an autonomous vehicle or an unmanned aerial vehicle in the form of hardware. As a part of the robot, the autonomous vehicle or the unmanned aerial vehicle, the user can use the navigation system by purchasing a robot, an autonomous vehicle or an unmanned aerial vehicle.

[0076] Furthermore, the motion device 110 and the processing device 120 may be further divided into a plurality of unit modules, for example, Figure 1 As shown, the motion device 110 may include a motion unit 111 and a collection unit 112, and the processing device 120 may include an image processing unit 121, a feature extraction unit 122, an inference unit 123 and a training unit 124. It should be understood that Figure 1 This is an exemplary division method. The motion device 110 and the processing device 120 may also include more or fewer units. For example, the feature extraction unit 122 and the reasoning unit 123 may be combined into a navigation unit. For example, the image processing unit 121 may be further divided into a bird's-eye view generation unit and a surround view generation unit. This application does not limit this.

[0077] Among them, the image processing unit 121, the feature extraction unit 122, the reasoning unit 123 and the training unit 124 in the processing device 120 can be implemented by software or by hardware. Exemplarily, the implementation of the image processing unit 121 in the processing device 120 is described below by taking the image processing unit 121 as an example. Similarly, the implementation of the feature extraction unit 122, the reasoning unit 123 and the training unit 124 can refer to the implementation of the image processing unit 121.

[0078] The image processing unit 121 is taken as an example of a software functional unit. The image processing unit 121 may include code running on a computing instance. Among them, the computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Further, the above-mentioned computing instance may be one or more. For example, the image processing unit 121 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one data center or multiple data centers with close geographical locations. Among them, usually a region may include multiple AZs.

[0079] The image processing unit 121 is an example of a hardware functional unit, and the image processing unit 121 may include at least one computing device, such as a server, etc. Alternatively, the image processing unit 121 may also be a device implemented by an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0080] The functions of each unit module in the motion device 110 and the processing device 120 are described below.

[0081] The collection unit 112 in the motion device 110 is used to collect multiple environmental information and send the multiple environmental information to the processing device 120. The multiple environmental information is information collected by the same or different sensors on the same scene.

[0082] In a specific implementation, the acquisition unit 112 in the motion device 110 can collect multiple environmental information through sensors on the device where the motion device 110 is located, and then send the multiple environmental information to the information feature extraction unit 122 in the processing device 120. The sensor includes but is not limited to image sensors, radar sensors, voice sensors, position sensors, acceleration sensors, gyroscope sensors, geomagnetic sensors, infrared sensors, inertial measurement units (IMU), odometers, angular velocity sensors, linear velocity sensors, and the like.

[0083] Among them, the image sensor is a device for capturing images, including cameras, webcams, etc.; the voice sensor is a sensor for capturing human voice, such as microphones, etc.; the position sensor is a sensor for measuring the position or moving distance of an object, such as the global positioning system (GPS) or trajectory sensor; the acceleration sensor is a sensor that can measure the acceleration of an object, and it is usually used to detect the motion state of an object. For example, in self-driving cars, accelerometers can be used to detect vehicle acceleration, braking, steering and other operations to help the vehicle achieve accurate control and navigation. In mobile robots, accelerometers can be used to detect the robot's shaking, vibration and tilt to support the robot's stability and safety. A gyroscope sensor is a sensor that can measure the angular velocity of an object. It can be used to detect information such as the rotation and direction change of the device. A geomagnetic sensor is a sensor that can measure the strength and direction of the earth's magnetic field. It is widely used in fields such as navigation and location recognition. An infrared sensor is a sensor that can sense infrared radiation from surrounding objects. It is often used in applications such as measuring temperature and detecting obstacles. An IMU is a device that integrates sensors such as accelerometers and gyroscopes, which can provide information about acceleration, angular velocity and direction. An odometer is a sensor that measures the distance and direction of a robot's movement. The robot's position and direction are usually determined by the rotation of wheels or tracks. An angular velocity sensor is a sensor that can measure the rotation speed of an object around an axis. A linear velocity sensor is a sensor that can measure the speed of an object as it moves in a certain direction.

[0084] It should be understood that the examples of the above sensors are for illustration purposes only, and the equipment where the motion device 110 is located may also include other information acquisition sensors that can sense specified measured quantities (physical quantities, chemical quantities, state quantities, and mechanical quantities) and convert them into usable output signals according to certain rules. Examples are not given here one by one.

[0085] In a possible implementation, the multiple environmental information may include multiple images, and the multiple images include multiple images captured from multiple different perspectives of the same scene, and the multiple images may be obtained by continuously changing the capture angle of the image sensor. Optionally, the scene contents captured by the multiple images are adjacent, and the scene contents captured by each image are adjacent in the horizontal direction or vertical direction in spatial position. The edges and features of the multiple images can be spliced ​​to form a complete ring view or panoramic view, or a complete ring view or panoramic view can be formed by filling gaps or discontinuous gaps through image processing technology.

[0086] The sensor on the device where the motion device 110 is located includes an image sensor that can change the acquisition angle. The image sensor can change the acquisition angle by rotating the device where the motion device 110 is located, or by rotating the base where the image sensor is located. For example, assuming that the motion device 110 is deployed on a robot, the image sensor (such as a camera) can be installed on the head of the robot, and multiple images at different perspectives can be obtained by rotating the robot head, or the image sensor is installed on a rotatable base at the bottom of the robot. When the robot does not rotate, the image sensor can also be rotated along the base to rotate and shoot images, and multiple images at multiple different perspectives can be obtained. For another example, assuming that the motion device 110 is deployed on an autonomous driving vehicle, the image sensor (such as a camera) can be installed on a rotatable bracket in the autonomous driving vehicle, so that the image sensor can shoot images as the bracket rotates, and multiple images at multiple different perspectives can be obtained. For another example, the sweeping robot can also have a rotating base and a built-in camera, and the images can be shot as the base rotates to obtain multiple images at different perspectives. It should be understood that the above examples are for illustration and are not specifically limited in this application.

[0087] Optionally, the width of the rotatable base where the image sensor is located may be smaller than the maximum width of the robot. In this way, when the robot enters a small space or a dead end environment, the robot is constrained by obstacles and cannot rotate. By rotating the base where the image sensor is located, the robot can obtain a field of view behind it, thus avoiding navigation failure due to the robot being stuck in a blind spot.

[0088] For example, when the device is a vehicle, the vehicle can collect multiple first images at multiple angles by rotating the base of the on-board camera to generate a first bird's-eye view and a first surrounding view, and then determine first angle information and first motion information based on the first bird's-eye view, the first surrounding view, and the navigation target, and then adjust the motion direction and motion speed based on the first motion information to move toward the navigation target, and at the same time adjust the collection angle of the image sensor according to the first angle information, collect the second image to generate a second bird's-eye view and a second surrounding view, and then determine second angle information and second motion information based on the second bird's-eye view, the second surrounding view, and the navigation target, and then continue to move toward the navigation target based on the second motion information, and so on, until the vehicle reaches the destination indicated by the navigation target.

[0089] To give another example, when the device is a robot, the robot can capture multiple first images by rotating itself or rotating the base of the image sensor to generate a first bird's-eye view and a first surrounding view, and then determine first angle information and first motion information based on the first bird's-eye view, the first surrounding view, and the navigation target, and then adjust the motion direction and motion speed based on the first motion information to move toward the navigation target, and at the same time adjust the capture angle of the image sensor according to the first angle information, capture the second image to generate a second bird's-eye view and a second surrounding view, and then determine second angle information and second motion information based on the second bird's-eye view, the second surrounding view, and the navigation target, and then continue to move toward the navigation target based on the second motion information, and so on, until the robot reaches the destination indicated by the navigation target.

[0090] It should be understood that other types of devices such as drones and sweeping robots can also implement the solutions described in this application according to the above examples, and they are not illustrated one by one here.

[0091] It should be noted that the above-mentioned multiple images can also be obtained by capturing the same scene through multiple image sensors with different capturing angles. That is to say, multiple image sensors with different capturing angles are deployed on the device where the motion device 110 is located. However, this will increase the hardware cost. Therefore, it can be selected according to the actual application scenario, and this application does not make any specific limitations.

[0092] In a possible implementation, the multiple environmental information may also include multiple sensor information, and the multiple sensor information includes information collected by other sensors except image sensors, such as the above-mentioned voice sensor, position sensor, acceleration sensor, gyroscope sensor, geomagnetic sensor, infrared sensor, IMU, odometer, angular velocity sensor, linear velocity sensor and other sensors, which are not specifically limited in this application.

[0093] Furthermore, the above-mentioned multiple sensor information also includes information obtained from a third-party platform, and the third-party platform may include a government public data platform and a third-party network service platform, wherein the government public data platform may provide various public data, such as a weather information service platform, which may provide weather forecast related data; a time information service platform, which may provide the current time, date, holiday information, etc. of each country; a medical institution service platform, which may provide notification information issued by each medical institution (such as hospital vacation notification, newly hired expert information), etc., and may also include other public data platforms that can provide information such as traffic planning data, road construction information, traffic flow data, parking lot location, charging location, census data, etc., which are not illustrated one by one here; a third-party network service platform may be a service platform that provides network data, such as an instant messaging service platform, an online audio and video service platform, an online shopping platform, a logistics tracking platform, a search engine, a map service platform, a takeaway service platform, etc. It should be understood that the above-mentioned government public data platform and third-party network service platform are only used for illustration, which may be determined according to the actual application scenario, and this application does not make specific limitations.

[0094] In one possible implementation, multiple environmental information may also include a navigation target, which may include a destination. The destination may be a specific address, such as a certain address, a certain store, a certain restaurant, my home, a living room, a kitchen, etc., or it may be specific coordinates, such as global positioning system (GPS) coordinates, or it may be a directional feature of the destination, such as 100 meters east, 100 meters left, etc.; the navigation target may also include navigation conditions, such as the shortest path, the shortest time, avoiding congestion, avoiding obstacles, etc. The navigation target may also include richer content, which will not be illustrated one by one here.

[0095] The image processing unit 121 of the processing device 120 is used to receive the multiple environmental information sent by the acquisition unit 112, and generate a surround view and a bird's-eye view according to the multiple images in the multiple environmental information.

[0096] Optionally, the surround view may be obtained by aligning and stitching multiple images in the horizontal or vertical direction. The surround view is usually an image with a wide-angle field of view. The wide-angle field of view here may be a field of view that exceeds what a camera or human eye can capture, such as a field of view exceeding 160 degrees in the horizontal direction or exceeding 175 degrees in the vertical direction. It should be understood that the wider the field of view of the surround view, the richer and more comprehensive the visual information it provides. In some embodiments, the surround view may display the scene from a 360-degree perspective.

[0097] It should be understood that if there is a gap or discontinuous gap between two images, it can be filled by image processing technology. The image processing technology here may include but is not limited to pixel interpolation, Poisson reconstruction, texture synthesis, image restoration, deep learning and other methods. For example, linear interpolation or quadratic interpolation methods can be used to calculate the value of the missing pixel based on the pixels around the missing pixel, and then fill it into the missing area, or an AI model can be trained based on a deep learning method to enable it to have the ability to predict the missing area. The above examples are for illustration only and are not specifically limited in this application.

[0098] Optionally, a bird's-eye view refers to an image that shows the whole picture of a scene or area from a bird's-eye view, similar to observing the ground from a bird's eye view. This perspective can provide a wider and more comprehensive field of view. In a specific implementation, a bird's-eye view can be an image obtained by projecting pictures of multiple perspectives onto a plane using inverse perspective mapping (IPM) technology, or it can be obtained by other technical means, such as using three-dimensional reconstruction technology to build a three-dimensional model and then rendering a bird's-eye view from a bird's-eye view. It should be understood that the above examples are for illustration and are not specifically limited in this application. Among them, a bird's-eye view can be generated based on multiple images, or it can be generated based on a surround view. In other words, a bird's-eye view and a surround view can be generated based on multiple images in parallel, or a surround view can be generated first and then a bird's-eye view can be generated. This application does not make specific limitations.

[0099] It can be understood that, based on the bird's-eye view and the surround view, the visual features of the device where the motion device 110 is located under an all-round perspective can be obtained, so that the visual features obtained by the motion device 110 are more comprehensive, and the navigation information generated thereby is more accurate.

[0100] The feature extraction unit 122 of the processing device 120 is used to extract features from the surround view and the bird's-eye view, obtain a first feature and a second feature, and then send the first feature and the second feature to the reasoning unit 123 .

[0101] In a possible implementation, the surrounding view and the bird's-eye view may be input into a feature extraction network to obtain a first feature and a second feature. The feature extraction network includes a first network and a second network, the input data of the first network includes the surrounding view, and the output data includes the first feature, and the input data of the second network includes the bird's-eye view, and the output data includes the second feature.

[0102] Optionally, the input data of the second network may also include a surround view, that is, the input data of the second network includes a bird's-eye view and a surround view, and the output data includes a second feature. It should be understood that extracting the visual features of the bird's-eye view in combination with the surround view can make the information contained in the second feature more comprehensive, and the navigation information generated thereby is more accurate.

[0103] In one possible implementation, the input data of the second network may include the first feature and the bird's-eye view, and the output data includes the second feature. It should be understood that the first feature is the feature extracted by the first network from the ring view. Compared with the original ring view, the first feature can exclude noise or other redundant information. Extracting the visual features of the bird's-eye view based on the first feature can not only make the information contained in the second feature more comprehensive, but also improve the efficiency and accuracy of extracting the second feature.

[0104] Optionally, the first network and the second network can be determined based on an autoencoder network (AE), wherein the autoencoder network can extract features from the input data, and these features can be used for subsequent tasks, such as classification, clustering, etc., and can also represent high-dimensional input data as low-dimensional codes to achieve data compression and dimensionality reduction. The autoencoder network can also have other functions, such as the data reconstruction autoencoder network can achieve data reconstruction and denoising, and the data generation autoencoder network can achieve image style conversion and other applications, which are not specifically limited in this application.

[0105] In a specific implementation, the self-encoding network may include an encoder and a decoder, wherein the encoder is used to convert the input data into a low-dimensional vector representation, and the output result of the encoder has the function set by the self-encoding network, such as dimensionality reduction, feature extraction, denoising, style conversion, etc. in the aforementioned content. The decoder is used to receive the low-dimensional vector output by the encoder, and map the low-dimensional vector back to the space of the input data to obtain the output result. If the output result of the decoder is very different from the input data, it means that the output result of the encoder is wrong and needs to be further optimized. In some cases, the input data may lose some information due to noise, distortion, etc. during transmission or storage. At this time, the decoder can be allowed to try to reconstruct the original data from the encoding result of the input data of these loss information. In addition, in some application scenarios, the output result of the decoder is the result of noise reduction or data compression, which can be understood as higher-quality input data, and data in the corresponding format can be generated according to subsequent processing requirements. Therefore, the output of the decoder and the output of the encoder have their own applications. When the input data needs to be encoded and its vector representation is obtained, the encoder output should be used. When a specific format or higher-quality input data is required, the decoder output should be used. The above examples are for illustration, and this application is not specifically limited.

[0106] Optionally, the first network may include a first encoder of a first autoencoding network, and the second network may include a first decoder of the first autoencoding network, a second encoder of the second autoencoding network, and a decoder of the second autoencoding network. The input data of the first network includes a ring view, and the output data includes a first feature, which is the output result of the first encoder.

[0107] In a specific implementation, the output end of the first decoder in the second network is connected to the input end of the second encoder, and the output end of the second encoder is connected to the input end of the second decoder. The input data of the first decoder includes the first feature, the input data of the second encoder includes the output result of the first decoder and the bird's-eye view, the input data of the second decoder includes the output result of the second encoder, and the output of the second decoder is the second feature.

[0108] In simple terms, the surround view is input into the first encoder of the first network, the first encoder outputs the first feature, the first feature is input into the first decoder of the second network, the output result of the first decoder is obtained, and then the output result of the first decoder and the bird's-eye view are input into the second encoder of the second network to obtain the output result of the second encoder, and then the output result of the second encoder is input into the second decoder, and the second decoder outputs the second feature. It should be understood that using the first feature and the bird's-eye view as input data for the second network enables the second feature finally extracted to have richer surround view information and bird's-eye view information, and the decision inferred based on this feature is more accurate.

[0109] Optionally, the autoencoding network can be multiple convolutional layers, the first decoder may include multiple first convolutional layers, and the second encoder may include multiple second convolutional layers. In one possible implementation, the output result of each first convolutional layer is sequentially input into multiple second convolutional layers, and there is a corresponding relationship between each first convolutional layer and the second convolutional layer. For example, the first decoder includes 3 convolutional layers, which are L11, L12 and L13 in order, and the second encoder includes 3 convolutional layers, which are L21, L22 and L23 in order. Then the output result of L11 is input into L21, the output result of L12 is input into L22, and the output result of L13 is input into L23. It should be understood that the above examples are for illustration and are not specifically limited in this application.

[0110] It should be noted that in the above implementation, the first number of the first convolutional layers of the first decoder is the same as the second number of the second convolutional layers of the second encoder. When the first number is greater than the second number, the input data of each second convolutional layer can be the output results of one or more first convolutional layers. When the first number is less than the second number, a part of the input data of the second convolutional layer is the output data of the first convolutional layer, and another part of the input data of the second convolutional layer can be the output data of the previous second convolutional layer.

[0111] It should be understood that the first feature is the output result of the first encoder, the first feature can be a feature vector, and the second feature is the output result of the second decoder. The function of the decoder is to map the vector output by the encoder back to the space of the input data. Therefore, the second feature can be a two-dimensional depth bird's-eye view. Since the input data of the second encoder includes not only the bird's-eye view, but also the output result of the first decoder, the two-dimensional depth bird's-eye view additionally includes the image features of the surrounding view compared to the bird's-eye view, and the decision inferred based on this feature is more accurate.

[0112] The inference unit 123 can obtain an inference result based on the first feature, the second feature and the navigation target, and the inference result includes angle information and motion information, wherein the acquisition angle is used for the acquisition unit 112 of the motion device 110 to adjust the acquisition angle of the image sensor to obtain multiple new images, and the motion information is used for the motion unit 111 of the motion device 110 to navigate and move to the target position.

[0113] It can be understood that the reasoning unit 123 performs reasoning based on the first feature and the second feature, and the first feature and the second feature are obtained from the bird's-eye view and the surround view, so that the navigation system can not only perceive the environmental information within the front field of view, but also perceive the environmental information in the rear, left, right and overhead perspectives, and fully perceive the environmental information around the device (such as a robot) where the motion device is located, so that the device where the motion device is located can directly find the navigation path, avoiding repeated exploration, collision, and inefficiency.

[0114] In a possible implementation, the reasoning unit 123 can obtain a reasoning result based on the first feature, the second feature, the sensor information, and the navigation target. It should be understood that the bird's-eye view and the surround view can provide multi-directional environmental information, and the sensor information can provide the kinematic information of the device and other auxiliary reference information, so that the navigation system can obtain more accurate reasoning results and improve the accuracy and efficiency of navigation.

[0115] In a specific implementation, the reasoning unit 123 can input the first feature, the second feature, the sensor information and the navigation target into the reasoning network to obtain the reasoning result, wherein the reasoning network can include a convolutional neural network (CNN), a recurrent neural network (RNN), a support vector machine (SVM), a decision tree (decision tree), a deep learning neural network (DNN), a long short-term memory (LSTM) network, etc., which is not specifically limited in this application.

[0116] Optionally, when the inference network is an LSTM network, the input data of the inference network also includes historical sensor information, that is, the inference unit 123 can input the first feature, the second feature, the sensor information, the navigation target and the historical sensor information into the inference network to obtain the inference result, wherein the collection time of the historical sensor information is earlier than the collection time of the above-mentioned sensor information. It should be understood that the LSTM network is a network based on historical data to capture the long-term dependencies in the time series. Using the LSTM network as the inference network can take into account the past historical sensor information to obtain more accurate inference results, and at the same time can solve the gradient disappearance problem and improve the accuracy of the model.

[0117] Optionally, the reasoning unit 123 can fuse the first feature, the second feature, the sensor information, the navigation target and the historical sensor information in the input data to obtain a fusion result, and then input the fusion result into the reasoning network. In the specific implementation, since the feature dimensions of the first feature, the second feature, the sensor information, the navigation target and the historical sensor information may be different, the feature dimensions of the first feature, the second feature, the sensor information, the navigation target and the historical sensor information can be unified first, and the high-dimensional features can be reduced in dimension. For example, the second feature is a feature extracted based on a bird's-eye view. The bird's-eye view is a top view from a two-dimensional plane perspective, so the second feature may be a two-dimensional feature, and the sensor information is usually a one-dimensional feature, so the second feature needs to be compressed and reduced in dimension, and then spliced ​​and fused with the first feature. In the specific implementation, the second feature can be input into the third self-encoding network, and the second feature after compression and dimensionality reduction can be obtained according to the output result of the third encoder of the third self-encoding network. Of course, the second feature can also be compressed and reduced in dimension in other ways to make it consistent with the data dimension of other input data, so that feature fusion can be performed, and this application does not make specific limitations.

[0118] It should be understood that by extracting and encoding features from the bird's-eye view and the panoramic view, images, features, and sensor information in different formats can be fused and inferred, thus avoiding problems such as poor model calculation efficiency and accuracy caused by data from multiple different sources.

[0119] In one embodiment, the angle information includes one or more target acquisition angles, and the target acquisition angles are used for the acquisition unit 112 to adjust the acquisition angle of the image sensor. In a specific implementation, the target acquisition angle in the above angle information can be a specific value, such as the acquisition angle of the positive viewing angle is 0°, the acquisition angle is +120° when rotated 120 degrees to the left based on the positive viewing angle, and the acquisition angle is -120° when rotated 120 degrees to the right based on the positive viewing angle. The acquisition angle can also be replaced by a noun, for example, the acquisition angle can be a left viewing angle, which means rotating 120 degrees to the left based on the positive viewing angle, and the acquisition angle can be a right viewing angle, which means rotating 120 degrees to the right based on the positive viewing angle. It should be understood that the above examples are for illustration, and the present application does not limit the specific expression of the acquisition angle.

[0120] Optionally, the angle information also includes the correspondence between the acquisition sequence and the target acquisition angle. Referring to the foregoing content, it can be known that the image sensor can continuously change the acquisition angle to obtain multiple images. Therefore, the one or more target acquisition angles output here may be multiple acquisition angles continuously changed by the image sensor, or one or more of the multiple acquisition angles continuously changed above. Therefore, the angle information may also include the correspondence between the acquisition sequence and the target acquisition angle. For example, the angle information may include the first acquisition angle of 0°, the second acquisition angle of +120°, and the third acquisition angle of -120°.

[0121] It should be noted that the target acquisition angle in the angle information can be the acquisition angle after the image sensor is adjusted, or it can be the angle that needs to be adjusted based on the current acquisition angle. For example, if the current acquisition angle is -120° and the acquisition angle needs to be adjusted to -180°, then the target acquisition angle can be -180°, or the target acquisition angle can be -60°. It should be understood that the above examples are for illustration only and this application does not make any specific limitations.

[0122] In one embodiment, the motion information may include the driving route, motion direction, motion speed, motion acceleration, motion angular velocity, etc., and may also include other information required for indicating the movement of the device. The motion information required by different devices may be different, and this application does not make specific limitations. For example, when the device is a vehicle, the vehicle can adjust the motion direction, motion speed, motion acceleration and other information according to the motion information to move toward the navigation target; when the device is a robot, the robot can adjust the motion direction, motion mode and speed and other information according to the motion information to move toward the navigation target; when the device is a drone, the drone can adjust the motion direction, motion speed, motion height, motion posture and so on according to the motion information, and this application does not make specific limitations.

[0123] The training unit 124 of the processing device 120 is used to train the AI ​​network using the sample set to obtain a trained feature extraction network and inference network.

[0124] In one embodiment, the feature extraction network and the inference network may be different network layers in the same navigation model, the feature extraction network is the feature extraction layer of the navigation model, and the inference network is the fully connected layer of the navigation model. The navigation model is obtained after training using the sample set.

[0125] Optionally, the sample set may be a labeled sample set, the sample set includes input data and labels, the input data includes a known bird's-eye view, a known surrounding view, known sensor information, and a known navigation target, and the labels of the input data include known angle information and known motion information. After the input data is input into the navigation model, an output result is obtained, and the network parameters are adjusted according to the loss value between the output result and the label until the model converges to obtain a trained navigation model.

[0126] Optionally, the sample set can be an unlabeled sample set, which includes a known bird's-eye view, a known surround view, known sensor information and a known navigation target. The navigation model can be implemented by a reinforcement learning algorithm. The output results of the navigation model include angle information, motion information and decision values. The decision values ​​are used to evaluate the quality of the output results. The model parameters are adjusted according to the decision values ​​and loss until the model converges to obtain a trained navigation model.

[0127] It should be noted that if the reasoning network in the navigation model is an LSTM network, the input data of the sample set may also include known historical sensor information.

[0128] It should be understood that the feature extraction network and the inference network can also be trained using other machine learning methods, which are not illustrated one by one here, and this application does not limit the training methods of the feature extraction network and the inference network.

[0129] The motion unit 111 of the motion device 110 is used to receive the motion information of the reasoning unit 123, and the acquisition unit 112 is used to receive the angle information of the reasoning unit, wherein the motion unit 111 can instruct the device where the motion device 110 is located to move according to the motion information, and the acquisition unit 112 can adjust the acquisition angle of the image sensor according to the angle information.

[0130] It should be noted that when the acquisition unit 112 acquires multiple images for the first time, it may acquire images at a preset acquisition angle, and then may acquire images at an acquisition angle adjusted according to the angle information sent by the inference unit 123 .

[0131] Optionally, when the angle information shows that the acquisition angle of the image sensor needs to be adjusted, the motion unit 111 can move toward the navigation target at a preset speed, and after the acquisition scheduling is adjusted, it moves at the motion speed in the motion information. The preset speed can be a relatively low speed that can ensure the image acquisition quality, thereby improving the quality of multiple image acquisitions.

[0132] Optionally, when the angle information shows that there is no need to adjust the acquisition angle of the image sensor, the motion unit 111 may instruct the device where the motion device 110 is located to move toward the navigation target according to the motion information.

[0133] Optionally, the acquisition unit 112 can periodically collect environmental information, the processing device 120 can periodically receive the environmental information and infer it to obtain angle information and motion information, the acquisition unit and the motion unit 111 periodically receive the angle information and motion information, and process the angle information and motion information. Therefore, the motion information generated by the inference unit can be the angle information and motion information within the period. With the periodic collection of environmental information, the generated motion information can also be continuously changed. For example, the navigation route can be continuously adjusted to achieve the purpose of real-time navigation and improve the stability of the solution.

[0134] Exemplarily, if the image collected in the first period is the first image, and the image collected in the second period is the second image, then the acquisition unit 112 may first collect multiple first images in the first period, where the multiple first images are multiple images obtained after the image sensor shoots the first scene from different acquisition angles, and then the image processing unit 121 determines the first bird's-eye view and the first surrounding view of the first scene based on the multiple first images, and the feature extraction unit 122 and the reasoning unit 123 then determine the first angle information and the first motion information based on the first bird's-eye view, the first surrounding view and the navigation target.

[0135] Next, the acquisition unit 112 may acquire multiple second images in the second cycle, the multiple second images being multiple images obtained by photographing the second scene at the acquisition angle indicated by the first angle information, and then the image processing unit 121 determines the second bird's-eye view and the second surround view of the second scene based on the multiple second images, and the feature extraction unit 122 and the reasoning unit 123 determine the second angle information and the second motion information based on the second bird's-eye view, the second surround view and the navigation target. This is done by analogy until the device where the motion device 110 is located reaches the position indicated by the navigation target.

[0136] In summary, the navigation system provided by the present application obtains multiple images at different acquisition angles by rotating the image sensor, and then obtains a bird's-eye view and a surround view based on the multiple images, and then determines the angle information and motion information based on the bird's-eye view, the surround view and the navigation target, and adjusts the next acquisition angle of the image sensor based on the angle information, and instructs the device where the navigation system is located to move toward the navigation target based on the motion information, so that the image sensor can continue to acquire new images based on the adjusted acquisition angle, obtain new acquisition angles and motion information, and so on, until it reaches the position indicated by the navigation target. The entire navigation process does not require the prior loading of maps, saves memory resources, and has high flexibility. In addition, the navigation system can perceive all-round environmental information such as the front, back, left, right, and overhead perspectives. The device can directly find the navigation path to avoid repeated exploration, collisions, and inefficiency.

[0137] Figure 4 This is a flowchart of a navigation method provided by the present application. The method can be applied to Figure 1 to Figure 3 In the navigation system shown, Figure 4 As shown, the method may include the following steps:

[0138] Step S410: The sports device 110 obtains multiple environmental information. This step can be performed by Figure 1 The acquisition unit 112 in the embodiment is implemented.

[0139] Optionally, the environmental information may include multiple images, and the above multiple images may be obtained after the image sensor (such as a camera) on the motion device 110 changes the collection angle. The image sensor can change the collection angle by rotating the device where the motion device 110 is located, or by rotating the base where the image sensor is located.

[0140] Furthermore, the width of the rotatable base where the image sensor is located may be smaller than the maximum width of the robot. In this way, when the robot enters a small space or a dead end environment, the robot is constrained by obstacles and cannot rotate. By rotating the base where the image sensor is located, the robot can obtain a field of view behind it, thus avoiding the problem of navigation failure due to the robot being stuck in a blind spot.

[0141] Optionally, the environmental information may also include navigation targets and sensor information. The navigation target may include a destination and navigation conditions. The destination may be a specific address, specific coordinates, or a location feature of the destination. The navigation conditions may be the shortest path, the shortest time, congestion avoidance, obstacle avoidance, etc. Sensor line information may include information collected by other sensors except image sensors, and may also include information obtained from third-party platforms, such as government public data platforms and third-party network service platforms.

[0142] It should be noted that the detailed description of the specific implementation of the motion device 110 and the environmental information can be found in Figure 1 The relevant description of the embodiments will not be repeated here.

[0143] Step S420: The motion device 110 sends a plurality of environmental information to the processing device 120. This step can be performed by Figure 1 The acquisition unit 112 in the embodiment is implemented.

[0144] Step S430: The processing device 120 generates a bird's-eye view and a surround view according to the multiple images. Figure 1 The image processing unit 121 in the embodiment is implemented.

[0145] Optionally, the surround view may be obtained by aligning and stitching multiple images in the horizontal or vertical direction. The surround view is usually an image with a wide-angle field of view, where the wide-angle field of view may exceed the field of view that can be captured by a camera or human eye. In some embodiments, the surround view may show a scene with a 360-degree viewing angle. In a specific implementation, if there is a gap or discontinuous gap between two images, it can be filled by image processing technology. The image processing technology here may include but is not limited to pixel interpolation, Poisson reconstruction, texture synthesis, image restoration, deep learning and other methods, which are not specifically limited in this application.

[0146] Optionally, a bird's-eye view refers to an image that shows the whole scene or area from a bird's-eye view, similar to observing the ground from a bird's eye view. This perspective can provide a wider and more comprehensive field of view. In a specific implementation, a bird's-eye view can be an image obtained by projecting pictures from multiple perspectives onto a plane using IPM technology, or it can be obtained by other technical means. Among them, a bird's-eye view can be generated based on multiple images or based on a panoramic view, and this application does not make specific limitations.

[0147] It can be understood that, based on the bird's-eye view and the surround view, the visual features of the device where the motion device 110 is located under an all-round perspective can be obtained, so that the visual features obtained by the motion device 110 are more comprehensive, and the navigation information generated thereby is more accurate.

[0148] It should be noted that the specific implementation of the processing device 120 and the specific description of the bird's-eye view and the surround view can be referred to Figure 1 The relevant descriptions in the embodiments are not repeated here.

[0149] Step S440: The processing device 120 determines the angle information and motion information based on the bird's-eye view, the surrounding view, and the navigation target. This step can be performed by Figure 1 The feature extraction unit 122 and the inference unit 123 in the embodiment are implemented.

[0150] Optionally, the processing device 120 may input the first surrounding view and the first bird's-eye view into a navigation model to obtain first angle information and first motion information, wherein the navigation model includes a feature extraction network and an inference network, the feature extraction network includes a first network and a second network, the first network is used to generate a first feature based on the first surrounding view, the second network is used to generate a second feature based on the first surrounding view and the first bird's-eye view, and the inference network is used to obtain the first angle information and the first motion information according to the first feature, the second feature and the navigation target.

[0151] It can be understood that by reasoning based on the first ring view and the first bird's-eye view, the navigation system can not only perceive the environmental information within the front field of view, but also perceive the environmental information in the rear, left, right and overhead perspectives, and all-roundly perceive the environmental information around the equipment where the motion device is located (such as a robot), so that the equipment where the motion device is located can directly find the navigation path, avoiding repeated exploration, collisions, and inefficiency.

[0152] Optionally, the input data of the second network may also include a surround view, that is, the input data of the second network includes a bird's-eye view and a surround view, and the output data includes a second feature. It should be understood that extracting the visual features of the bird's-eye view in combination with the surround view can make the information contained in the second feature more comprehensive, and the navigation information generated thereby is more accurate.

[0153] Optionally, the input data of the second network may include the first feature and the bird's-eye view, and the output data includes the second feature. It should be understood that the first feature is the feature extracted by the first network from the ring view. Compared with the original ring view, the first feature can exclude noise or other redundant information. Extracting the visual features of the bird's-eye view based on the first feature can not only make the information contained in the second feature more comprehensive, but also improve the efficiency and accuracy of extracting the second feature.

[0154] In a specific implementation, the first network and the second network may be determined based on an autoencoding network. The autoencoding network may include an encoder and a decoder. The description of the autoencoding network may refer to Figure 1 The relevant description in the embodiment is not repeated here. Optionally, the first network may include a first encoder of a first self-encoding network, and the second network may include a first decoder of the first self-encoding network, a second encoder of the second self-encoding network, and a decoder of the second self-encoding network. Wherein, the input data of the first network includes a ring view, and the output data includes a first feature, which is the output result of the first encoder.

[0155] Optionally, the output of the first decoder in the second network is connected to the input of the second encoder, and the output of the second encoder is connected to the input of the second decoder. The input data of the first decoder includes the first feature, the input data of the second encoder includes the output result of the first decoder and the bird's-eye view, the input data of the second decoder includes the output result of the second encoder, and the output of the second decoder is the second feature. It should be understood that the first feature and the bird's-eye view are used as the input data of the second network, so that the second feature finally extracted can have richer surround information and bird's-eye information, and the decision inferred based on this feature is more accurate.

[0156] Optionally, the autoencoding network can be multiple convolutional layers, the first decoder may include multiple first convolutional layers, and the second encoder may include multiple second convolutional layers. In one possible implementation, the output result of each first convolutional layer is sequentially input into multiple second convolutional layers, and there is a corresponding relationship between each first convolutional layer and the second convolutional layer.

[0157] It should be noted that in the above implementation, the first number of the first convolutional layers of the first decoder is the same as the second number of the second convolutional layers of the second encoder. When the first number is greater than the second number, the input data of each second convolutional layer can be the output results of one or more first convolutional layers. When the first number is less than the second number, a part of the input data of the second convolutional layer is the output data of the first convolutional layer, and another part of the input data of the second convolutional layer can be the output data of the previous second convolutional layer.

[0158] It should be understood that the first feature is the output result of the first encoder, the first feature can be a feature vector, and the second feature is the output result of the second decoder. The function of the decoder is to map the vector output by the encoder back to the space of the input data. Therefore, the second feature can be a two-dimensional depth bird's-eye view. Since the input data of the second encoder includes not only the bird's-eye view, but also the output result of the first decoder, the two-dimensional depth bird's-eye view additionally includes the image features of the surrounding view compared to the bird's-eye view, and the decision inferred based on this feature is more accurate.

[0159] In a possible implementation, the reasoning result can be obtained based on the first feature, the second feature, the sensor information and the navigation target. It should be understood that the bird's-eye view and the surround view can provide multi-directional environmental information, and the sensor information can provide the kinematic information of the device and other auxiliary reference information, so that the navigation system can obtain more accurate reasoning results and improve the accuracy and efficiency of navigation.

[0160] In a specific implementation, the first feature, the second feature, the sensor information and the navigation target can be input into the inference network to obtain the inference result. Optionally, when the inference network is an LSTM network, the input data of the inference network also includes historical sensor information, wherein the collection time of the historical sensor information is earlier than the collection time of the above sensor information.

[0161] Optionally, the first feature, the second feature, the sensor information, the navigation target and the historical sensor information in the input data can be fused to obtain a fusion result, and then the fusion result can be input into the inference network. In the specific implementation, since the feature dimensions of the first feature, the second feature, the sensor information, the navigation target and the historical sensor information may be different, the feature dimensions of the first feature, the second feature, the sensor information, the navigation target and the historical sensor information can be unified first, and the high-dimensional features can be reduced in dimension. For example, the second feature is a feature extracted based on a bird's-eye view. The bird's-eye view is a top view from a two-dimensional plane perspective, so the second feature may be a two-dimensional feature, and the sensor information is usually a one-dimensional feature, so the second feature needs to be compressed and reduced in dimension, and then spliced ​​and fused with the first feature. In the specific implementation, the second feature can be input into the third self-encoding network, and the second feature after compression and dimensionality reduction can be obtained according to the output result of the third encoder of the third self-encoding network. Of course, the second feature can also be compressed and reduced in dimension by other means to make it consistent with the data dimension of other input data, so that feature fusion can be performed, and this application does not make specific limitations.

[0162] It should be understood that by extracting and encoding features from the bird's-eye view and the panoramic view, images, features, and sensor information in different formats can be fused and inferred, thus avoiding problems such as poor model calculation efficiency and accuracy caused by data from multiple different sources.

[0163] In an embodiment of the present application, the above-mentioned navigation model may be obtained by training using a sample set. Specifically, before step S430, the AI ​​network may be trained using the sample set to obtain a trained navigation model.

[0164] Optionally, the feature extraction network and the inference network may be different network layers in the same navigation model, the feature extraction network is the feature extraction layer of the navigation model, and the inference network is the fully connected layer of the navigation model. The navigation model is obtained after training using the sample set.

[0165] Optionally, the sample set may be a labeled sample set, the sample set includes input data and labels, the input data includes a known bird's-eye view, a known surrounding view, known sensor information, and a known navigation target, and the labels of the input data include known angle information and known motion information. After the input data is input into the navigation model, an output result is obtained, and the network parameters are adjusted according to the loss value between the output result and the label until the model converges to obtain a trained navigation model.

[0166] Optionally, the sample set can be an unlabeled sample set, which includes a known bird's-eye view, a known surround view, known sensor information and a known navigation target. The navigation model can be implemented by a reinforcement learning algorithm. The output results of the navigation model include angle information, motion information and decision values. The decision values ​​are used to evaluate the quality of the output results. The model parameters are adjusted according to the decision values ​​and loss until the model converges to obtain a trained navigation model.

[0167] It should be noted that if the reasoning network in the navigation model is an LSTM network, the input data of the sample set may also include known historical sensor information.

[0168] It should be understood that the feature extraction network and the inference network can also be trained using other machine learning methods, which are not illustrated one by one here, and this application does not limit the training methods of the feature extraction network and the inference network.

[0169] For example, Figure 5 is a schematic diagram of the navigation model structure in the navigation method provided by this application, such as Figure 5 As shown, the navigation model may include a feature extraction network and a reasoning network.

[0170] exist Figure 5 In the example shown, the surround view is input into the first encoder, the first encoder outputs the first feature, the output result of the first decoder and the bird's-eye view are input into the second encoder, the second decoder outputs the second feature, and then the second feature is input into the third encoder, the third encoder outputs the reduced second feature, and then the first feature, the reduced second feature, the sensor information and the navigation target are input into the inference network to obtain angle information and motion information.

[0171] It should be noted that if the inference network is an LSTM, the input data of the inference network also includes historical sensor information. During the training process, if the inference network is implemented by a reinforcement learning algorithm, the output result of the inference network also includes a reward value, and the model parameters are adjusted based on the reward value. However, in the inference stage, the output result of the inference network does not include a reward value. For details, please refer to the description in the aforementioned embodiment, which will not be repeated here.

[0172] The output of each convolutional layer in the first decoder can be used as input data to each convolutional layer in the second encoder, for example Figure 5 In the example shown, the output result of the first convolutional layer of the first decoder can be used as input data of the second convolutional layer of the second encoder, and the output result of the second convolutional layer of the first decoder can be used as input data of the third convolutional layer of the second encoder, and so on. It should be understood that Figure 5 For illustration, the present application does not limit the number of convolutional layers in each encoder and each decoder, nor does it limit the output of each convolutional layer of the first decoder and the input of each convolutional layer of the second encoder.

[0173] It should be understood that Figure 5 The network structure shown is used for illustration. The second encoder can also directly take the ring view and the bird's-eye view as input, and replace the ring view itself with the output result of the first decoder. This is because in some cases, the ring view may lose some information due to noise, distortion and other problems during transmission or storage. At this time, the decoder can try to reconstruct the original data. In addition, in some application scenarios, the output result of the decoder is the result of noise reduction or data compression, which can be understood as a higher quality ring view, and the format required by the second decoder can be generated. Therefore, compared with the output result of the ring view and the first decoder, using the output result of the first decoder can improve the extraction efficiency and accuracy of the second feature.

[0174] It should be noted that the first feature is the output result of the first encoder, the first feature can be a feature vector, and the second feature is the output result of the second decoder. The function of the decoder is to map the vector output by the encoder back to the space of the input data. Therefore, the second feature can be a two-dimensional depth bird's-eye view. Since the input data of the second encoder includes not only the bird's-eye view, but also the output result of the first decoder, the two-dimensional depth bird's-eye view additionally includes the image features of the surrounding view compared to the bird's-eye view, and the decision inferred based on this feature is more accurate.

[0175] However, the two-dimensional depth bird's-eye view as the second feature is high-dimensional data, while the first feature, navigation target and sensor information are low-dimensional data. Therefore, it is necessary to reduce the dimension of the two-dimensional depth bird's-eye view. Specifically, Figure 5 As shown, the second feature is input into the third encoder, and the output result of the third encoder is the second feature after dimensionality reduction, and this feature can be spliced ​​and fused with the first feature, the navigation target and the sensor information to obtain a fusion result, and reasoning is completed based on the fusion result to obtain motion information and angle information. By extracting and encoding features of the bird's-eye view and the surround view, images, features and sensor information in different formats can be fused and inferred, avoiding problems such as poor model calculation efficiency and poor accuracy caused by data from multiple different sources.

[0176] It should be understood that a connection can also be established between each convolution layer in the first encoder and each convolution layer in the second encoder, for example, the second convolution layer of the first encoder can be connected to the second-to-last convolution layer of the second encoder, the third convolution layer of the first encoder can be connected to the third-to-last convolution layer of the second encoder, and so on. Similarly, a connection can be established between each convolution layer in the second encoder and each convolution layer in the second decoder, a connection can be established between each convolution layer in the first encoder and each convolution layer in the first decoder, and a connection can be established between each convolution layer in the third encoder and each convolution layer in the third decoder. The network structure of each encoder and decoder can be determined according to the actual application scenario, and this application does not limit this.

[0177] It should be noted that the structures of the first encoder, the second encoder, and the third encoder may be the same or different, and the structures of the first decoder, the second decoder, and the third decoder may be the same or different. Figure 5 For ease of understanding, three encoders and decoders with the same structure are drawn, but this application is not limited to this.

[0178] Step S450: The processing device 120 sends the angle information and the motion information to the motion device 110. This step can be performed by Figure 1 The reasoning unit 123 in the embodiment implements.

[0179] In one embodiment, the angle information includes one or more target acquisition angles. The target acquisition angle in the angle information may be a specific value or may be replaced by a noun. Figure 1 In the related examples in the embodiments, this application does not limit the specific form of the acquisition angle.

[0180] Optionally, the angle information also includes the correspondence between the acquisition sequence and the target acquisition angle. Referring to the foregoing content, it can be known that the image sensor can continuously change the acquisition angle to obtain multiple images. Therefore, the one or more target acquisition angles output here can be multiple acquisition angles continuously changed by the image sensor, or one or more of the multiple acquisition angles continuously changed above. Therefore, the angle information can also include the correspondence between the acquisition sequence and the target acquisition angle.

[0181] In one embodiment, the motion information may include navigation route, motion direction, motion speed, motion acceleration, motion angular velocity, etc., and may also include other information required to indicate the movement of the device, which is not specifically limited in this application.

[0182] Step S460: The motion device 110 adjusts the acquisition angle according to the angle information and moves toward the navigation target according to the motion information. Figure 1 The acquisition unit 112 and the motion unit 111 in the embodiment are implemented.

[0183] It should be noted that when acquiring multiple images for the first time, the images can be acquired according to a preset acquisition angle, and then the images can be acquired according to the acquisition angle adjusted by the angle information. Referring to the above content, it can be known that the image sensor acquires multiple images by rotating the base to change the acquisition angle, so the acquisition angle adjusted here can be the target acquisition angle indicated in the recording angle information, so that when the image sensor acquires multiple images in the future, it acquires multiple images according to the multiple target acquisition angles indicated by the angle information.

[0184] Optionally, when the angle information shows that the acquisition angle of the image sensor needs to be adjusted, the camera can move toward the navigation target at a preset speed, and after the acquisition scheduling is adjusted, the camera can move at the movement speed in the movement information. The preset speed can be a relatively low speed that can ensure the image acquisition quality, thereby improving the quality of multiple image acquisitions.

[0185] Optionally, environmental information can be collected periodically, environmental information can be received periodically and inferred to obtain angle information and motion information, angle information and motion information can be received periodically, and the angle information and motion information can be processed. Therefore, the generated motion information can be the angle information and motion information within the period. As the environmental information is collected periodically, the generated motion information can also be continuously changed. For example, the navigation route can be continuously adjusted to achieve the purpose of real-time navigation and improve the stability of the solution.

[0186] Exemplarily, if the image collected in the first period is the first image and the image collected in the second period is the second image, then multiple first images can be collected in the first period, where the multiple first images are multiple images obtained after the image sensor shoots the first scene from different collection angles, and then a first bird's-eye view and a first surrounding view of the first scene are determined based on the multiple first images, and then the first angle information and the first motion information are determined based on the first bird's-eye view, the first surrounding view and the navigation target.

[0187] Next, multiple second images are collected in a second cycle, where the multiple second images are obtained by photographing the second scene at the collection angle indicated by the first angle information, and then a second bird's-eye view and a second surround view of the second scene are determined based on the multiple second images, and second angle information and second motion information are determined based on the second bird's-eye view, the second surround view, and the navigation target. This process is repeated until the device reaches the location indicated by the navigation target.

[0188] Step S470: Acquire multiple environmental information based on the adjusted acquisition angle. This step can be performed by Figure 1 The acquisition unit 112 in the embodiment is implemented.

[0189] It should be noted that the adjusted acquisition angle can be multiple angles, so as to obtain multiple images under multiple viewing angles. Further, after executing step S470, step S420 can be executed to send the newly acquired multiple environmental information to the processing device 120 for processing. The processing device 120 can execute steps S430 to S450 to generate angle information and motion information and feed it back to the motion device 110. The motion device 110 can adjust the acquisition angle of the image sensor according to the new angle information, and then use the new acquisition angle to obtain multiple environmental information. Steps S420 to S470 are repeated until the device where the motion device 110 is located reaches the position indicated by the navigation target.

[0190] For example, Figure 6 As shown, Figure 6 This is a schematic flow chart of the steps for adjusting the acquisition angle of view in the navigation method provided by the present application. After the motion device 110 receives the angle information and the motion information, it can execute step S460 to change the acquisition angle of the image sensor according to the angle information. Among them, the angle of view 1, the angle of view 3 and the angle of view 4 do not need to be adjusted, and the angle of view 2 needs to be rotated 15 degrees to the right, so the acquisition angles of the angles 1 to 4 are recorded according to the angle information.

[0191] When executing step S470, the image sensor is rotated to capture multiple images according to the multiple capture angles indicated by the angle information. For example, Figure 6 Four viewing angles are used as an example for illustration, wherein the image captured by viewing angle 1 is the left front image, the image captured by viewing angle 2 is the right front image, the image captured by viewing angle 3 is the left rear image, and the image captured by viewing angle 4 is the right rear image. Figure 6 For the purpose of illustration, this application does not limit the number of multiple images and the range of viewing angle scanning.

[0192] After executing step S470, steps S420 to S430 can be executed to generate a panoramic view and a bird's-eye view based on multiple images, and then generate new angle information and motion information based on the panoramic view, the bird's-eye view and the navigation target. Then, use the acquisition angle indicated by the new angle information to acquire multiple new images, and so on, until the device where the motion device 110 is located reaches the position indicated by the navigation target.

[0193] In summary, the navigation method provided by the present application obtains multiple images at different acquisition angles by rotating the image sensor, then obtains a bird's-eye view and a surround view based on the multiple images, and then determines the angle information and motion information based on the bird's-eye view, the surround view and the navigation target, adjusts the next acquisition angle of the image sensor based on the angle information, and instructs the device where the navigation system is located to move toward the navigation target based on the motion information, so that the image sensor can continue to acquire new images based on the adjusted acquisition angle, obtain new acquisition angles and motion information, and so on, until the position indicated by the navigation target is reached. The entire navigation process does not require the prior loading of a map, saves memory resources, and has high flexibility. In addition, the navigation system can perceive all-round environmental information such as the front, back, left, right, and overhead perspectives. The device can directly find the navigation path to avoid repeated exploration, collisions, and inefficiency.

[0194] Figure 7 is a schematic diagram of the structure of a computing device provided by the present application, and the computing device 700 may be the navigation system in the aforementioned content. The navigation system may be deployed on computing devices with computing capabilities and mobility requirements, such as mobile robots, vehicles, drones, smart home appliances, etc., wherein mobile robots are robots that can move in an environment, and may include mobile mechanical devices with tracks / wheels, such as logistics robots, agricultural robots, inspection robots, voice guide robots, etc., vehicles may include self-driving cars, and smart home appliances may include sweeping robots, which are not given examples one by one here.

[0195] Furthermore, the computing device 700 includes a processor 701, a storage unit 702, a storage medium 703 and a communication interface 704, wherein the processor 701, the storage unit 702, the storage medium 703 and the communication interface 704 communicate through a bus 705, and also communicate through other means such as wireless transmission.

[0196] The processor 701 may include any one or more processors such as a central processing unit (CPU), a microprocessor (MP), or a digital signal processor (DSP).

[0197] For example, a CPU, an NPU, or a combination of a CPU and a hardware chip. The hardware chip is an Application-Specific Integrated Circuit (ASIC), a Programmable Logic Device (PLD), or a combination thereof. The PLD is a Complex Programmable Logic Device (CPLD), a Field-Programmable Gate Array (FPGA), a Generic Array Logic (GAL), or any combination thereof. The processor 701 executes various types of digital storage instructions, such as software or firmware programs stored in the storage unit 702, which enables the computing device 700 to provide a wide variety of services.

[0198] In a specific implementation, as an embodiment, the processor 701 includes one or more CPUs, such as Figure 7 CPU0 and CPU1 are shown in the figure.

[0199] In a specific implementation, as an embodiment, the computing device 700 also includes multiple processors, such as Figure 7 1 and 10. Each of these processors may be a single-CPU or a multi-CPU. A processor herein refers to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0200] The storage unit 702 is used to store program codes, and is controlled by the processor 701 to execute the above Figure 1-Figure 6 The processing steps of the navigation system in any embodiment of the present invention. The program code includes one or more software units. The one or more software units are Figure 1 The acquisition unit, motion unit, image processing unit, feature extraction unit and reasoning unit in the embodiment, wherein the acquisition unit is used to acquire multiple first images, the image processing unit is used to determine a first bird's-eye view and a first surround view according to the multiple first images, the feature extraction unit is used to generate a first feature and a second feature based on the first bird's-eye view and the first surround view, the reasoning unit is used to determine first angle information and first motion information according to the first feature, the second feature and the navigation target, the acquisition unit is used to acquire multiple second images according to multiple target acquisition angles indicated by the first angle information, the motion unit is used to move toward the navigation target according to the first motion information, and specifically, the acquisition unit is used to perform Figure 4In the step S410 and step S420 of the embodiment and the optional steps thereof, the image processing unit is used to perform Figure 4 In step S430 and its optional steps in the embodiment, the feature extraction unit and the reasoning unit are used to perform Figure 4 In step S440 and its optional steps in the embodiment, the reasoning unit is also used to execute Figure 4 In step S450 and its optional steps in the embodiment, the acquisition unit and the motion unit are also used to perform Figure 4 Step S460 and step S470 and optional steps in the embodiment are not described in detail here.

[0201] The storage unit 702 includes a read-only memory and a random access memory, and provides instructions and data to the processor 701. The storage unit 702 also includes a non-volatile random access memory. The storage unit 702 is a volatile memory or a non-volatile memory, or includes both volatile and non-volatile memories. Among them, the non-volatile memory is a read-only memory (ROM), a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory is a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are used, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct rambus RAM (DRRAM). It can also be a hard disk, a universal serial bus (USB), a flash memory, a secure digital memory card (SD card), a memory stick, etc. The hard disk is a hard disk drive (HDD), a solid state disk (SSD), a mechanical hard disk (HDD), etc., and this application does not make specific limitations.

[0202] The storage medium 703 is a carrier for storing data, such as a hard disk, a USB flash drive (universal serialbus), a flash memory, a secure digital memory card (SD card), a memory stick, etc. The hard disk can be a hard disk drive (HDD), a solid state disk (SSD), a mechanical hard disk (HDD), etc., and this application does not make specific limitations.

[0203] The communication interface 704 can be used to provide information input or output for the processor 701, or alternatively, the communication interface 704 can be used to receive data sent externally and / or send data to the outside, and can be a wired interface including an Ethernet cable, or a wireless interface (such as Wi-Fi, Bluetooth, general wireless transmission, etc.), or alternatively, the communication interface 704 can also include a transmitter coupled to the interface (such as a radio frequency transmitter, antenna, etc.), or a receiver, etc.

[0204] The bus 705 is a Peripheral Component Interconnect Express (PCIe) bus, or an extended industry standard architecture (EISA) bus, a unified bus (Ubus or UB), a compute express link (CXL), a cache coherent interconnect for accelerators (CCIX), etc. The bus 705 is divided into an address bus, a data bus, a control bus, etc. In addition to the data bus, the bus 705 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, various buses are labeled as bus 705 in the figure.

[0205] Need to explain, Figure 7 This is only a possible implementation of the embodiment of the present application. In practical applications, the computing device 700 may also include more or fewer components, which is not limited here. For the contents not shown or described in the embodiment of the present application, please refer to the aforementioned Figure 1-Figure 6 The relevant explanations in the embodiments will not be repeated here.

[0206] Figure 8It is a structural diagram of a computing device cluster provided by the present application, and the computing device cluster includes one or more first computing devices 800 and one or more second computing devices 900. The first computing device 800 and the second computing device 900 establish a communication connection through a network 1000, wherein the first computing device 800 may be deployed with the motion device 110 in the aforementioned content, and the second computing device 900 may be deployed with the processing device 120 in the aforementioned content. The first computing device 800 may be a computing device with mobile requirements, such as a mobile robot, a vehicle, a drone, a smart appliance, etc., and the second computing device 900 may be a computing device with computing functions, such as a BMS, a virtual machine, a container, an edge server, a terminal device, etc. The network 1000 may be a wired network or a wireless network, which is not specifically limited in the present application.

[0207] like Figure 8 As shown, the memory 803 in each first computing device 800 may store the same instructions required to be executed by the motion device 110 in executing the navigation method. The instructions may implement one or more software units. The one or more software units are Figure 1 The acquisition unit and the motion unit in the embodiment, specifically, the acquisition unit is used to acquire a plurality of first images, the image processing unit is used to determine a first bird's-eye view and a first surround view according to the plurality of first images, and send them to the second computing device 900, and is also used to receive the first angle information and the first motion information sent by the second computing device 900, and acquire a plurality of second images according to the plurality of target acquisition angles indicated by the first angle information, and the motion unit is used to move toward the navigation target according to the first motion information, specifically, the acquisition unit is used to perform Figure 4 In the embodiment, in step S410 and step S420 and optional steps thereof, the acquisition unit and the motion unit are further configured to perform Figure 4 Step S460 and step S470 and optional steps in the embodiment are not described in detail here.

[0208] The memory 903 in each second computing device 900 may store the same instructions required to be executed by the processing device 120 in the navigation method. The instructions may implement one or more software units, and the one or more software units are Figure 1 The image processing unit, feature extraction unit, reasoning unit and training unit in the embodiment, wherein the image processing unit is used to determine the first bird's-eye view and the first surround view according to the plurality of first images, the feature extraction unit is used to generate the first feature and the second feature based on the first bird's-eye view and the first surround view, the reasoning unit is used to determine the first angle information and the first motion information according to the first feature, the second feature and the navigation target, and the training unit is used to train the network model required for the feature extraction unit and the reasoning unit to work. Specifically, the image processing unit is used to execute Figure 4In step S430 and its optional steps in the embodiment, the feature extraction unit and the reasoning unit are used to perform Figure 4 In step S440 and its optional steps in the embodiment, the reasoning unit is also used to execute Figure 4 Step S450 and its optional steps in the embodiment are not described in detail here.

[0209] The first computing device 800 includes a processor 801, a communication interface 802, a memory 803, and a bus 804, and the second computing device 900 includes a processor 901, a communication interface 902, a memory 803, and a bus 804, wherein the processor 801, the communication interface 802, the memory 803, the bus 804, the processor 901, the communication interface 902, the memory 903, the bus 904 are further described in Figure 7 The descriptions of the processor 701 , the processor 701 , the storage unit 702 , the storage medium 703 , the communication interface 704 , and the bus 705 in the embodiment are not repeated here.

[0210] exist Figure 7 In the implementation shown, each first computing device 800 in the computing cluster can establish a communication connection with one second computing device 900, that is, the multiple images collected by each first computing device 800 are handed over to the corresponding second computing device 900 for processing. In another possible implementation, each first computing device 800 can establish a connection with multiple second computing devices 900, that is, the multiple images collected by each first computing device 800 are handed over to the corresponding multiple second computing devices 900 for processing.

[0211] like Figure 8 As shown, Figure 8 is another structural diagram of a computing device cluster provided by the present application. Figure 8 In the implementation shown, the memory 903 in different second computing devices 900 may also store different instructions, which are respectively used to execute part of the functions of the processing device 120 in the navigation system. That is, the instructions stored in the memory 903 in different second computing devices 900 may be combined to implement the image processing unit, the feature extraction unit, the reasoning unit and the training unit, and one or more second computing devices 900 may establish a communication connection through an external network or an internal network.

[0212] like Fig. 9As shown, the second computing device 900A and the second computing device 900B are connected via a network. Specifically, the network is connected via the communication interface in each computing device. In this type of possible implementation, the memory 903 in the second computing device 900A stores instructions for executing the functions of the image processing unit, the feature extraction unit, and the reasoning unit. At the same time, the memory 903 in the second computing device 900B stores instructions for executing the functions of the training unit. The functional description of each unit can be referred to Figure 8 The embodiments are not described in detail here.

[0213] Need to explain, Fig. 9 The implementation method shown may be an implementation method for a case where the processing capability of the second computing device 900A is insufficient, or the storage space of the second computing device 900A is insufficient, or an implementation method for other business scenarios, and this application does not make any specific limitation.

[0214] The embodiment of the present application also provides a computer program product including instructions. The computer program product may be a software or program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on a computing device or a computing device cluster, the computing device or computing device cluster executes the navigation method.

[0215] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by the computing device or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk). The computer-readable storage medium includes instructions that instruct the computing device to execute the navigation method, or instructs a computing device cluster to execute the navigation method.

[0216] The above embodiments may be implemented in whole or in part by software, hardware, firmware or any other combination thereof. When implemented using software, the above embodiments may be implemented in whole or in part in the form of a computer program product. The computer program product includes a plurality of computer instructions. When the computer program instructions are loaded or executed on a computer, the process or function according to the embodiment of the present invention is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium.

[0217] The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent repairs or replacements within the technical scope disclosed by the present invention, and these repairs or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A navigation method, characterized in that: The method is applied to a navigation system, the navigation system comprising a motion device and a processing device, and the method comprises: The motion device acquires a plurality of first images, where the plurality of first images are a plurality of images obtained after the motion device instructs an image sensor to photograph a first scene from a plurality of acquisition angles; The processing device determines a first bird's-eye view and a first surround view of the first scene based on the plurality of first images; The processing device determines first angle information and first motion information based on the first bird's-eye view, the first surrounding view and the navigation target, wherein the first angle information is used to instruct the image sensor to adjust the acquisition angle, and the first motion information is used to instruct the device where the motion device is located to move toward the navigation target.

2. The method according to claim 1, characterized in that The multiple first images are multiple images obtained after the image sensor changes its acquisition angle, and the image sensor changes its acquisition angle as the device rotates, or the device includes a base, and the image sensor changes its acquisition angle as the base rotates.

3. The method according to claim 1 or 2, characterized in that: The processing device determines the first angle information and the first motion information based on the first bird's-eye view, the first surrounding view, and the navigation target, including: The processing device inputs the first surrounding view and the first bird's-eye view into a navigation model to obtain the first angle information and the first motion information, wherein the navigation model includes a feature extraction network and an inference network, the feature extraction network includes a first network and a second network, the first network is used to generate a first feature based on the first surrounding view, the second network is used to generate a second feature based on the first surrounding view and the first bird's-eye view, and the inference network is used to obtain the first angle information and the first motion information according to the first feature, the second feature and the navigation target.

4. The method according to claim 3, characterized in that The first network includes a first encoder of a first autoencoding network, the input data of the first encoder includes the first ring view, and the first feature is the output result of the first encoder; the second network includes a decoder of the first autoencoding network, an encoder of the second autoencoding network and a decoder of the second autoencoding network, the second feature is the output result of the second decoder, and the input data of the second network includes the first feature and the first bird's-eye view.

5. The method according to claim 4, characterized in that In the second network, the output end of the first decoder is connected to the input end of the second encoder, and the output end of the second encoder is connected to the input end of the second decoder, wherein the input data of the first decoder includes the first feature, and the input data of the second encoder includes the output result of the first decoder and the first bird's-eye view.

6. The method according to claim 5, characterized in that The first decoder includes multiple first convolutional layers, the second encoder includes multiple second convolutional layers, and the output result of each first convolutional layer is sequentially input into each second convolutional layer in the order of the convolutional layers.

7. The method according to any one of claims 1 to 6, characterized in that: The processing device determines the first angle information and the first motion information based on the first bird's-eye view, the first surrounding view, and the navigation target, including: The processing device determines first angle information and first motion information based on the first bird's-eye view, the first surround view, the navigation target and sensor information, wherein the sensor information includes information collected by other sensors except the image sensor, and the other sensors include one or more of a position sensor, an acceleration sensor, a gyroscope sensor, a geomagnetic sensor, an infrared sensor, an IMU, an odometer, an angular velocity sensor and a linear velocity sensor.

8. The method according to any one of claims 1 to 7, characterized in that: The first motion information includes a motion speed, and the method further includes: The motion device moves toward the navigation target at a preset speed when adjusting the acquisition angle of the image sensor according to the first angle information; When the acquisition angle of the image sensor does not need to be adjusted, the motion device moves toward the navigation target using the motion speed.

9. The method according to any one of claims 1 to 8, characterized in that: The method further comprises: The motion device instructs the image sensor to capture a plurality of second images, where the plurality of second images are a plurality of images obtained after photographing a second scene using one or more target capture angles indicated by the first angle information; The processing device determines a second bird's-eye view and a second surrounding view of the second scene based on the plurality of second images; The processing device determines second angle information and second motion information based on the second bird's-eye view, the second surrounding view and the navigation target, the second angle information is used to instruct the image sensor to adjust the acquisition angle, and the second motion information is used to instruct the device where the motion device is located to move toward the navigation target.

10. The method according to any one of claims 1 to 9, characterized in that: The motion device and the processing device are deployed on the same computing device, and the computing device includes any one of an autonomous driving vehicle, an unmanned aerial vehicle, a mobile robot, and a movable smart home appliance; or, The motion device is deployed on a first computing device, and the first computing device includes any one of an autonomous driving vehicle, a drone, a mobile robot, and a movable smart home appliance; The processing device is deployed on a second computing device, and the second computing device includes any one of a virtual machine, a bare metal server, and a container.

11. A sports device, characterized in that: The motion device is applied to a navigation system, the navigation system comprises the motion device and a processing device, and the motion device comprises: A capture unit, used to acquire a plurality of first images, where the plurality of first images are a plurality of images obtained after an image sensor captures a first scene from a plurality of capture angles; The acquisition unit is used to send the multiple first images to the processing device; The acquisition unit is used to receive the first angle information sent by the processing device, and instruct the image sensor to adjust the acquisition angle according to the first angle information based on the first angle information; The motion unit is used to receive the first motion information sent by the processing device, and instruct the device where the motion device is located to move towards the navigation target based on the first motion information.

12. The exercise device according to claim 11, characterized in that The multiple first images are multiple images obtained after the image sensor changes its acquisition angle, and the image sensor changes its acquisition angle as the device rotates, or the device includes a base, and the image sensor changes its acquisition angle as the base rotates.

13. The exercise device according to claim 11 or 12, characterized in that: The first motion information includes motion speed; The motion unit is configured to move toward the navigation target at a preset speed when adjusting the acquisition angle of the image sensor according to the first angle information; The motion unit is used to move toward the navigation target using the motion speed when the acquisition angle of the image sensor does not need to be adjusted.

14. The sports device according to any one of claims 11 to 13, characterized in that: The acquisition unit is further used to instruct the image sensor to acquire a plurality of second images, where the plurality of second images are a plurality of images obtained after photographing the second scene using one or more target acquisition angles indicated by the first angle information; The acquisition unit is used to send the plurality of second images to the processing device The acquisition unit is configured to receive the second angle information sent by the processing device, and instruct the image sensor to adjust the acquisition angle according to the second angle information based on the second angle information; The motion unit is used to receive the second motion information sent by the processing device, and instruct the device where the motion device is located to move towards the navigation target based on the second motion information.

15. The sports device according to any one of claims 11 to 14, characterized in that: The motion device and the processing device are deployed on the same computing device, and the computing device includes any one of an autonomous driving vehicle, an unmanned aerial vehicle, a mobile robot, and a movable smart home appliance; or, The motion device is deployed on a first computing device, and the first computing device includes any one of an autonomous driving vehicle, a drone, a mobile robot, and a movable smart home appliance; The processing device is deployed on a second computing device, and the second computing device includes any one of a virtual machine, a bare metal server, and a container.

16. A processing device, characterized in that: The processing device is applied to a navigation system, the navigation system comprises the processing device and a motion device, and the processing device comprises: An image processing unit, configured to receive a plurality of first images sent by the motion device, wherein the plurality of first images are a plurality of images obtained after the motion device instructs an image sensor to photograph a first scene from a plurality of acquisition angles; The image processing unit is further used to determine a first bird's-eye view and a first surround view of the first scene based on the multiple first images; A navigation unit is used to determine first angle information and first motion information based on the first bird's-eye view, the first surrounding view and the navigation target, and send the first angle information and the first motion information to the motion device, wherein the first angle information is used to instruct the image sensor to adjust the acquisition angle, and the first motion information is used to instruct the device where the motion device is located to move toward the navigation target.

17. The processing device according to claim 16, characterized in that The multiple first images are multiple images obtained after the image sensor changes its acquisition angle, and the image sensor changes its acquisition angle as the device rotates, or the device includes a base, and the image sensor changes its acquisition angle as the base rotates.

18. The processing device according to claim 16 or 17, characterized in that The navigation unit is used to input the first surrounding view and the first bird's-eye view into a navigation model to obtain the first angle information and the first motion information, wherein the navigation model includes a feature extraction network and an inference network, the feature extraction network includes a first network and a second network, the first network is used to generate a first feature based on the first surrounding view, the second network is used to generate a second feature based on the first surrounding view and the first bird's-eye view, and the inference network is used to obtain the first angle information and the first motion information according to the first feature, the second feature and the navigation target.

19. The processing device according to claim 18, characterized in that The first network includes a first encoder of a first autoencoding network, the input data of the first encoder includes the first ring view, and the first feature is the output result of the first encoder; the second network includes a decoder of the first autoencoding network, an encoder of the second autoencoding network and a decoder of the second autoencoding network, the second feature is the output result of the second decoder, and the input data of the second network includes the first feature and the first bird's-eye view.

20. The processing device according to claim 19, characterized in that In the second network, the output end of the first decoder is connected to the input end of the second encoder, and the output end of the second encoder is connected to the input end of the second decoder, wherein the input data of the first decoder includes the first feature, and the input data of the second encoder includes the output result of the first decoder and the first bird's-eye view.

21. The processing device according to claim 20, characterized in that The first decoder includes multiple first convolutional layers, the second encoder includes multiple second convolutional layers, and the output result of each first convolutional layer is sequentially input into each second convolutional layer in the order of the convolutional layers.

22. The processing device according to any one of claims 16 to 21, characterized in that: The navigation unit is used to determine first angle information and first motion information based on the first bird's-eye view, the first surround view, the navigation target and sensor information, wherein the sensor information includes information collected by other sensors except the image sensor, and the other sensors include one or more of a position sensor, an acceleration sensor, a gyroscope sensor, a geomagnetic sensor, an infrared sensor, an IMU, an odometer, an angular velocity sensor, and a linear velocity sensor.

23. The processing device according to any one of claims 16 to 22, characterized in that: an image processing unit, configured to receive a plurality of second images sent by the motion device, wherein the plurality of second images are a plurality of images obtained by photographing a second scene using one or more target acquisition angles indicated by the first angle information; The image processing unit is further used to determine a second bird's-eye view and a second surrounding view of the second scene based on the multiple second images; A navigation unit is used to determine second angle information and second motion information based on the second bird's-eye view, the second surrounding view and the navigation target, and send the second angle information and the second motion information to the motion device, wherein the second angle information is used to instruct the image sensor to adjust the acquisition angle, and the second motion information is used to instruct the device where the motion device is located to move toward the navigation target.

24. The processing device according to any one of claims 16 to 23, characterized in that: The motion device and the processing device are deployed on the same computing device, and the computing device includes any one of an autonomous driving vehicle, an unmanned aerial vehicle, a mobile robot, and a movable smart home appliance; or, The motion device is deployed on a first computing device, and the first computing device includes any one of an autonomous driving vehicle, a drone, a mobile robot, and a movable smart home appliance; The processing device is deployed on a second computing device, and the second computing device includes any one of a virtual machine, a bare metal server, and a container.

25. A navigation system, characterized in that: The system includes a motion device and a processing device, wherein the motion device implements the function implemented by the unit module described in any one of claims 11 to 15, and the processing device implements the function implemented by the unit module described in any one of claims 16 to 24.

26. A computing device, characterized in that The method comprises a processor and a memory, wherein the memory is used to store codes, and the processor is used to execute the codes to implement the method according to any one of claims 1 to 10.

27. A computing device cluster, characterized in that: comprising at least one first computing device and at least one second computing device; Each of the first computing devices includes a first processor and a first memory, wherein the first memory is used to store a first code, and the first processor is used to execute the first code to implement the unit module described in any one of claims 11 to 15; Each of the second computing devices includes a second processor and a second memory, wherein the second memory is used to store a second code, and the second processor is used to execute the second code to implement the unit module described in any one of claims 16 to 24.

28. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and the instructions are executed by a computing device or a computing device cluster to implement the method according to any one of claims 1 to 10.

Citation Information

Cited By

  • Navigation method and system, and related device

    EP4660582A1

  • Autonomous vehicle contorl in unmarked lane intersection

    US20260167218A1