Intelligent agent navigation method and intelligent agent

Abstract

The present application discloses a method and an agent for intelligent agent navigation, and belongs to the field of navigation technology. The method comprises: determining the first reachable navigation point corresponding to the current navigation point, obtaining the first image corresponding to the first reachable navigation point; determining the predicted navigation end point according to the navigation instructions of this navigation, the position information corresponding to the historical navigation point, the posture information of the intelligent agent at the historical navigation point, the topological information of the historical navigation point and the first image, and determining the navigation point to be driven among the candidate navigation points, wherein the candidate navigation point includes the historical navigation point and the first reachable navigation point; and controlling the intelligent agent to drive to the navigation point to be driven. The present application can realize the navigation of the intelligent agent.

Description

Technical Field

[0001] The present application relates to the field of navigation technology, and in particular to a method, device, intelligent agent and storage medium for intelligent agent navigation. Background Art

[0002] With the development of science and technology, intelligent agents have gradually entered people's field of vision. Intelligent agents are used in all walks of life. At present, intelligent agents include household service robots, delivery robots, public place guidance robots, etc.

[0003] Navigating according to the navigation instructions input by the user is one of the keys to intelligent agents. How to achieve intelligent agent navigation is a problem that needs to be solved urgently. Summary of the invention

[0004] The embodiment of the present application provides a method and agent for intelligent agent navigation. The technical solution is as follows:

[0005] In a first aspect, a method for intelligent agent navigation is provided, the method comprising:

[0006] Determine a first reachable navigation point corresponding to the current navigation point, and obtain a first image corresponding to the first reachable navigation point;

[0007] Determine a predicted navigation end point according to the navigation instruction of this navigation, the location information corresponding to the historical navigation point, the posture information of the intelligent body at the historical navigation point, the topological information of the historical navigation point and the first image, and determine a navigation point to be driven from candidate navigation points, wherein the candidate navigation points include the historical navigation point and the first reachable navigation point;

[0008] Control the intelligent agent to travel toward the navigation point to be traveled.

[0009] In a possible implementation, the method further includes:

[0010] Determine a second reachable navigation point corresponding to the navigation starting point, obtain a second image corresponding to the second reachable navigation point, and determine an image feature vector corresponding to the second reachable navigation point based on the second image;

[0011] Determine a language feature vector corresponding to each word in the navigation instruction and an overall feature vector corresponding to the navigation instruction;

[0012] Determine a navigation point marking vector corresponding to the navigation starting point according to the image feature vector corresponding to the second reachable navigation point, the position information corresponding to the navigation starting point, the posture information of the agent corresponding to the navigation starting point, and the timing information corresponding to the navigation starting point;

[0013] Selecting a plurality of initial candidate navigation endpoints from among the candidate navigation endpoints whose distance from the navigation starting point is a preset distance, and determining navigation endpoint mark vectors respectively corresponding to the plurality of initial candidate navigation endpoints;

[0014] Based on the image feature vector corresponding to the second reachable navigation point, the language feature vector and the overall feature vector, the navigation point mark vector corresponding to the navigation starting point and the navigation end point mark vector corresponding to each initial candidate navigation end point, a navigation point to be traveled is determined among the second reachable navigation points, and a predicted navigation end point is determined among the multiple initial candidate navigation end points.

[0015] In a possible implementation, determining a navigation point to be traveled among the second reachable navigation points, and determining a predicted navigation endpoint among the multiple initial candidate navigation endpoints according to the image feature vector corresponding to the second reachable navigation point, the language feature vector and the overall feature vector, the navigation point mark vector corresponding to the navigation starting point and the navigation endpoint mark vector corresponding to each initial candidate navigation endpoint, includes:

[0016] Taking the overall feature vector as a global tag vector, the language feature vector, the global tag vector, the navigation point tag vector corresponding to the navigation starting point, the image feature vector corresponding to the second reachable navigation point, and the navigation endpoint tag vector corresponding to each initial candidate navigation endpoint are input into a transformer model to obtain a first updated global tag vector, a first updated navigation point tag vector corresponding to the navigation starting point, a first updated image feature vector corresponding to the second reachable navigation point, and first updated navigation endpoint tag vectors corresponding to each of the multiple initial candidate navigation endpoints;

[0017] Determine a navigation point to be driven among the second reachable navigation points according to the first updated image feature vector and the first updated global marking vector;

[0018] A predicted navigation endpoint is determined from among the multiple initial candidate navigation endpoints according to the first updated navigation endpoint mark vector.

[0019] In a possible implementation, the method of determining a predicted navigation end point according to the navigation instruction of the current navigation, the position information corresponding to the historical navigation point, the posture information of the agent at the historical navigation point, the topological information of the historical navigation point and the first image, and determining the navigation point to be driven from the candidate navigation points includes:

[0020] Obtaining a second updated global marking vector obtained by the transformer model at a previous historical navigation point, second updated navigation point marking vectors corresponding to historical navigation points other than the previous historical navigation point, and a plurality of second updated navigation endpoint marking vectors;

[0021] Determine a navigation point marking vector corresponding to the previous navigation point according to an image feature vector corresponding to a third reachable navigation point corresponding to the previous navigation point, position information corresponding to the previous navigation point, posture information of the agent at the previous navigation point, and timing information corresponding to the previous navigation point;

[0022] Determining, according to the first image, an image feature vector corresponding to the first reachable navigation point;

[0023] According to the second updated global mark vector, the language feature vector corresponding to each word in the navigation instruction, the second updated navigation point mark vector, the navigation point mark vector corresponding to the previous historical navigation point, the topological information of the historical navigation point, the image feature vector corresponding to the first reachable navigation point and the multiple second updated navigation endpoint mark vectors, the navigation point to be traveled is determined among the candidate navigation points, and the predicted navigation endpoint is determined.

[0024] In a possible implementation manner, determining, according to the first image, an image feature vector corresponding to the first reachable navigation point includes:

[0025] Determining a location feature vector corresponding to the first reachable navigation point according to the location information corresponding to the first reachable navigation point;

[0026] Inputting the first image into an image feature extractor to obtain an initial feature vector of the first image;

[0027] The initial feature vector and the position feature vector corresponding to the first reachable navigation point are added to obtain an image feature vector corresponding to the first reachable navigation point.

[0028] In a possible implementation, determining a navigation point to be driven among candidate navigation points and determining a predicted navigation endpoint according to the second updated global mark vector, the language feature vector corresponding to each word in the navigation instruction, the second updated navigation point mark vector, the navigation point mark vector corresponding to the previous historical navigation point, the topological information of the historical navigation point, the image feature vector corresponding to the first reachable navigation point, and the plurality of second updated navigation endpoint mark vectors, includes:

[0029] Determine an adjacency matrix corresponding to the historical navigation point according to the topological information of the historical navigation point, and use the adjacency matrix as an attention matrix of the second updated navigation point mark vector and the navigation point mark vector corresponding to the previous historical navigation point;

[0030] Input the second updated global mark vector, the language feature vector corresponding to each word in the navigation instruction, the second updated navigation point mark vector, the navigation point mark vector corresponding to the previous historical navigation point, the attention matrix, the image feature vector corresponding to the first reachable navigation point, and the multiple second updated navigation endpoint mark vectors into the transformer model to obtain a third updated global mark vector, a third updated navigation point mark vector corresponding to each historical navigation point, a second updated image feature vector corresponding to the first reachable navigation point, and multiple third updated navigation endpoint mark vectors;

[0031] Determine a navigation point to be driven among candidate navigation points according to the third updated global marking vector, the third updated navigation point marking vector and the second updated image feature vector;

[0032] Determine the updated candidate navigation endpoints to which the plurality of third updated navigation endpoint mark vectors respectively correspond;

[0033] According to the plurality of third updated navigation endpoint mark vectors, a predicted navigation endpoint is determined from the updated candidate navigation endpoints respectively corresponding to the plurality of third updated navigation endpoint mark vectors.

[0034] In a possible implementation, determining the navigation point to be driven from among the candidate navigation points according to the third updated global marking vector, the third updated navigation point marking vector and the second updated image feature vector includes:

[0035] Calculating the similarity between the second updated image feature vector corresponding to each first reachable navigation point and the third updated global label vector as the confidence corresponding to each first reachable navigation point;

[0036] Calculate the similarity between the third updated navigation point mark vector corresponding to each historical navigation point and the third updated global mark vector as the confidence corresponding to each historical navigation point;

[0037] Among the first reachable navigation points and the historical navigation points, a navigation point corresponding to the maximum confidence is determined as the navigation point to be driven.

[0038] In a possible implementation, determining, according to the multiple third updated navigation endpoint mark vectors, a predicted navigation endpoint from updated candidate navigation endpoints respectively corresponding to the multiple third updated navigation endpoint mark vectors, includes:

[0039] Inputting the plurality of third updated navigation endpoint mark vectors into a navigation endpoint prediction model to obtain confidences of the updated candidate navigation endpoints respectively corresponding to the plurality of third updated navigation endpoint mark vectors;

[0040] The updated candidate navigation endpoint corresponding to the maximum confidence is used as the predicted navigation endpoint.

[0041] In a possible implementation, the navigation endpoint prediction model is composed of a multi-layer perceptron MLP and a normalization function.

[0042] In a second aspect, a device for intelligent agent navigation is provided, the device comprising:

[0043] A determination module, used to determine a first reachable navigation point corresponding to the current navigation point, and obtain a first image corresponding to the first reachable navigation point;

[0044] a prediction module, configured to determine a predicted navigation end point according to the navigation instruction of this navigation, the location information corresponding to the historical navigation point, the posture information of the agent at the historical navigation point, the topological information of the historical navigation point and the first image, and determine a navigation point to be driven from candidate navigation points, wherein the candidate navigation points include the historical navigation point and the first reachable navigation point;

[0045] The navigation module is used to control the intelligent body to travel to the navigation point to be traveled.

[0046] In a possible implementation, the prediction module is further used to:

[0047] Determine a second reachable navigation point corresponding to the navigation starting point, obtain a second image corresponding to the second reachable navigation point, and determine an image feature vector corresponding to the second reachable navigation point based on the second image;

[0048] Determine a language feature vector corresponding to each word in the navigation instruction and an overall feature vector corresponding to the navigation instruction;

[0049] Determine a navigation point marking vector corresponding to the navigation starting point according to the image feature vector corresponding to the second reachable navigation point, the position information corresponding to the navigation starting point, the posture information of the agent corresponding to the navigation starting point, and the timing information corresponding to the navigation starting point;

[0050] Selecting a plurality of initial candidate navigation endpoints from among the candidate navigation endpoints whose distance from the navigation starting point is a preset distance, and determining navigation endpoint mark vectors respectively corresponding to the plurality of initial candidate navigation endpoints;

[0051] Based on the image feature vector corresponding to the second reachable navigation point, the language feature vector and the overall feature vector, the navigation point mark vector corresponding to the navigation starting point and the navigation end point mark vector corresponding to each initial candidate navigation end point, a navigation point to be traveled is determined among the second reachable navigation points, and a predicted navigation end point is determined among the multiple initial candidate navigation end points.

[0052] In a possible implementation, the prediction module is used to:

[0053] Taking the overall feature vector as a global tag vector, the language feature vector, the global tag vector, the navigation point tag vector corresponding to the navigation starting point, the image feature vector corresponding to the second reachable navigation point, and the navigation endpoint tag vector corresponding to each initial candidate navigation endpoint are input into a transformer model to obtain a first updated global tag vector, a first updated navigation point tag vector corresponding to the navigation starting point, a first updated image feature vector corresponding to the second reachable navigation point, and first updated navigation endpoint tag vectors corresponding to each of the multiple initial candidate navigation endpoints;

[0054] Determine a navigation point to be driven among the second reachable navigation points according to the first updated image feature vector and the first updated global marking vector;

[0055] A predicted navigation endpoint is determined from among the multiple initial candidate navigation endpoints according to the first updated navigation endpoint mark vector.

[0056] In a possible implementation, the prediction module is used to:

[0057] Obtaining a second updated global marking vector obtained by the transformer model at a previous historical navigation point, second updated navigation point marking vectors corresponding to historical navigation points other than the previous historical navigation point, and a plurality of second updated navigation endpoint marking vectors;

[0058] Determine a navigation point marking vector corresponding to the previous navigation point according to an image feature vector corresponding to a third reachable navigation point corresponding to the previous navigation point, position information corresponding to the previous navigation point, posture information of the agent at the previous navigation point, and timing information corresponding to the previous navigation point;

[0059] Determining, according to the first image, an image feature vector corresponding to the first reachable navigation point;

[0060] According to the second updated global mark vector, the language feature vector corresponding to each word in the navigation instruction, the second updated navigation point mark vector, the navigation point mark vector corresponding to the previous historical navigation point, the topological information of the historical navigation point, the image feature vector corresponding to the first reachable navigation point and the multiple second updated navigation endpoint mark vectors, the navigation point to be traveled is determined among the candidate navigation points, and the predicted navigation endpoint is determined.

[0061] In a possible implementation, the prediction module is used to:

[0062] Determining a location feature vector corresponding to the first reachable navigation point according to the location information corresponding to the first reachable navigation point;

[0063] Inputting the first image into an image feature extractor to obtain an initial feature vector of the first image;

[0064] The initial feature vector and the position feature vector corresponding to the first reachable navigation point are added to obtain an image feature vector corresponding to the first reachable navigation point.

[0065] In a possible implementation, the prediction module is used to:

[0066] Determine an adjacency matrix corresponding to the historical navigation point according to the topological information of the historical navigation point, and use the adjacency matrix as an attention matrix of the second updated navigation point mark vector and the navigation point mark vector corresponding to the previous historical navigation point;

[0067] Input the second updated global mark vector, the language feature vector corresponding to each word in the navigation instruction, the second updated navigation point mark vector, the navigation point mark vector corresponding to the previous historical navigation point, the attention matrix, the image feature vector corresponding to the first reachable navigation point, and the multiple second updated navigation endpoint mark vectors into the transformer model to obtain a third updated global mark vector, a third updated navigation point mark vector corresponding to each historical navigation point, a second updated image feature vector corresponding to the first reachable navigation point, and multiple third updated navigation endpoint mark vectors;

[0068] Determine a navigation point to be driven among candidate navigation points according to the third updated global marking vector, the third updated navigation point marking vector and the second updated image feature vector;

[0069] Determine the updated candidate navigation endpoints to which the plurality of third updated navigation endpoint mark vectors respectively correspond;

[0070] According to the plurality of third updated navigation endpoint mark vectors, a predicted navigation endpoint is determined from the updated candidate navigation endpoints respectively corresponding to the plurality of third updated navigation endpoint mark vectors.

[0071] In a possible implementation, the prediction module is used to:

[0072] Calculating the similarity between the second updated image feature vector corresponding to each first reachable navigation point and the third updated global label vector as the confidence corresponding to each first reachable navigation point;

[0073] Calculate the similarity between the third updated navigation point mark vector corresponding to each historical navigation point and the third updated global mark vector as the confidence corresponding to each historical navigation point;

[0074] Among the first reachable navigation points and the historical navigation points, a navigation point corresponding to the maximum confidence is determined as the navigation point to be driven.

[0075] In a possible implementation, the prediction module is used to:

[0076] Inputting the plurality of third updated navigation endpoint mark vectors into a navigation endpoint prediction model to obtain confidences of the updated candidate navigation endpoints respectively corresponding to the plurality of third updated navigation endpoint mark vectors;

[0077] The updated candidate navigation endpoint corresponding to the maximum confidence is used as the predicted navigation endpoint.

[0078] In a possible implementation, the navigation endpoint prediction model is composed of a multi-layer perceptron and a normalization function.

[0079] According to a third aspect, an intelligent agent is provided, comprising a processor and a memory, wherein the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the method for intelligent agent navigation as described in the first aspect above.

[0080] In a fourth aspect, a computer-readable storage medium is provided, wherein the storage medium stores at least one instruction, and the at least one instruction is loaded and executed by a processor to implement the method for intelligent body navigation as described in the first aspect above.

[0081] In a fifth aspect, a computer-readable storage medium is provided, wherein the storage medium stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the method for intelligent body navigation as described in the first aspect above.

[0082] The beneficial effects brought by the technical solution provided by the embodiment of the present application include at least:

[0083] In the embodiment of the present application, the intelligent agent first determines the first reachable navigation point corresponding to the current navigation point, and obtains the first image corresponding to the first reachable navigation point. Then, the predicted navigation end point is determined in combination with the navigation instructions of this navigation, the position information corresponding to the historical navigation points, the posture information of the intelligent agent at the historical navigation points, the topological information of the historical navigation points and the first image, and the navigation point to be traveled is determined among the candidate navigation points. In the example of the present application, the candidate navigation points include not only the reachable navigation points corresponding to the current navigation point, but also the historical navigation points. In this way, even if the intelligent agent navigates incorrectly, it can return to the historical navigation points in the subsequent navigation process, making the navigation accuracy higher. In addition, in the embodiment of the present application, the topological information of the historical navigation points is also introduced when determining the navigation point to be traveled, which can also improve the navigation accuracy to a certain extent. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0085] Figure 1 It is a schematic diagram of an implementation scenario provided by an embodiment of the present application;

[0086] Figure 2 is a flow chart of a method for intelligent agent navigation provided by an embodiment of the present application;

[0087] Figure 3 is a schematic diagram of a candidate navigation destination provided in an embodiment of the present application;

[0088] Figure 4 is a flow chart of a method for intelligent agent navigation provided by an embodiment of the present application;

[0089] Figure 5 is a flow chart of a method for intelligent agent navigation provided by an embodiment of the present application;

[0090] Figure 6 is a schematic diagram of the structure of an intelligent body navigation device provided in an embodiment of the present application;

[0091] Figure 7 It is a schematic diagram of an intelligent agent structure provided in an embodiment of the present application. DETAILED DESCRIPTION

[0092] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0093] The embodiment of the present application provides a method for intelligent navigation, which can be implemented by an intelligent agent. The intelligent agent can be a household service robot, a delivery robot, a public place guidance robot, etc. The method can be applied in a variety of scenarios, for example, in public places such as shopping malls, museums, and restaurants, in personal residences, and in designated outdoor delivery areas. Figure 1 , showing an implementation scenario. Figure 1 The scene shown is a private residence. Figure 1 In the example, the agent is in a bedroom, and the user can send navigation instructions to the agent to instruct it to move from the current position (which can be called the navigation starting point) to the specified position (which can be called the navigation end point).

[0094] In the intelligent agent navigation method provided in the embodiment of the present application, the intelligent agent can determine the reachable navigation points based on the scene images around the current location. Then, the intelligent agent obtains the navigation instructions, the images corresponding to the reachable navigation points, the location information of the historical navigation points, the posture information of the historical navigation points, and the topological information of the historical navigation points according to the user input, predicts the navigation end point, and determines the navigation point to be traveled among the historical navigation points and the reachable navigation points. Then, it is possible to navigate to the navigation point to be traveled. It can be seen that in the navigation scheme provided in the embodiment of the present application, the topological information of the historical navigation points is introduced, and the historical navigation points are added to the candidate navigation points, so that the navigation accuracy is higher.

[0095] The following is a description of the method for intelligent navigation provided by the embodiment of the present application in conjunction with the accompanying drawings. Figure 2 As shown, the processing flow of the method may include the following steps:

[0096] Step 201: Get navigation instructions.

[0097] The navigation instructions may be natural language instructions, such as Chinese instructions, English instructions, etc.

[0098] In practice, when a user wants to instruct an agent to go to a certain location, the user can issue navigation instructions to the agent. There are many ways to issue navigation instructions, and several of them are listed below for explanation.

[0099] Method 1: Send navigation instructions to the intelligent agent through voice.

[0100] For example, in Figure 1In the scenario shown, the agent is at the bedside, and the user can speak navigation instructions, such as "Go out of the room, then turn right, continue to walk straight ahead to the end of the hallway", or "Go out of the room, then turn right and continue to the end of the hallway". Furthermore, the agent can obtain the navigation instructions spoken by the user through the audio collection device.

[0101] Method 2: Send navigation instructions to the agent through the terminal

[0102] In the second method, the terminal can establish a communication connection with the intelligent agent, and the communication connection can be a wired connection or a wireless connection, and the wireless connection can be WiFi (Wireless Fidelity), Bluetooth, etc. The terminal can be a mobile terminal such as a mobile phone or a tablet computer, or a computer device.

[0103] For example, the user can enter navigation instructions in the terminal, such as "Go out of the room, then turn right, continue to walk straight ahead to the end of the hallway", or "Go out of the room, then turn right and continue to the end of the hallway", and the terminal can send the navigation instructions to the agent through the communication connection between the terminal and the agent. Then, the agent can receive the navigation instructions sent by the terminal.

[0104] Method 3: Input navigation instructions through the touch screen of the intelligent agent

[0105] In the third method, the intelligent agent may be equipped with a touch screen, and the user may find a navigation instruction input interface on the touch screen of the intelligent agent and input navigation instructions on the interface, such as inputting "go out of the room, then turn right, continue to walk straight ahead to the end of the hallway", or "go out of the room, then turn right and continue to the end of the hallway". Then, the intelligent agent may obtain the navigation instructions input by the user.

[0106] In addition, relevant personnel can configure a work area map in the agent in advance so that the agent can know the maximum range of its movement. Figure 1For example, if a personal residence is shown, the intelligent agent can be configured with a map of the personal residence. Whenever a navigation instruction is obtained, the intelligent agent can generate a minimum rectangle including the working area based on the known working area, and then divide the minimum rectangle into multiple squares of equal area according to a preset scale, and the center of each square is a candidate navigation destination.

[0107] For example, Figure 3 As shown, in Figure 1 Based on this, a Figure 1 The smallest rectangle including the individual residence is obtained, and then the smallest rectangle is divided into multiple squares of equal area, and the center of each square is a candidate navigation end point.

[0108] Step 202: Acquire a scene image.

[0109] In implementation, at least one image acquisition device, such as a camera, is installed on the intelligent body, and the intelligent body can continuously acquire images of surrounding scenes through the at least one image acquisition device.

[0110] Step 203: Determine reachable navigation points based on the scene image, and obtain images corresponding to the reachable navigation points.

[0111] The reachable navigation points are navigation points reachable from the current location, and the reachable navigation points do not include historical navigation points of this navigation.

[0112] In implementation, the agent can discretize the working area into multiple points, each of which is a possible navigation point. At the navigation starting point or whenever the agent reaches a navigation point, the agent can perform obstacle analysis on the scene image collected at the current navigation point to determine the reachable navigation point. Then, the agent can obtain an image including the reachable navigation point in the collected scene image as the image corresponding to the reachable navigation point.

[0113] Step 204: determine the predicted navigation end point based on the navigation instructions, the location information corresponding to the historical navigation points, the posture information of the intelligent agent at the historical navigation points, the topological information of the historical navigation points and the images corresponding to the reachable navigation points, and determine the navigation point to be driven among the candidate navigation points.

[0114] Among them, candidate navigation points include historical navigation points and reachable navigation points.

[0115] The following first describes how the agent determines the navigation point to be driven and predicts the navigation end point at the navigation starting point (i.e., the first navigation point). Figure 4 , it can be determined that the process may include the following steps:

[0116] S11, inputting the navigation instruction into a language model to obtain a language feature vector corresponding to each word in the navigation instruction and an overall feature vector corresponding to the navigation instruction.

[0117] Among them, the language model can be a Bidirectional Encoder Representation from Transformers (BERT) model.

[0118] The navigation instruction is input into the BERT model to obtain the language feature vector corresponding to each character (or word) in the navigation instruction and the overall feature vector corresponding to the navigation instruction. For the convenience of description, the language feature vector corresponding to each character (or word) in the navigation instruction is recorded as: x1, x2…x m , where m is the number of characters (or words) in the navigation instruction, and the overall feature vector corresponding to the navigation instruction is recorded as x0.

[0119] S12. Obtain a position feature vector corresponding to a reachable navigation point corresponding to the navigation starting point.

[0120] In the embodiment of the present application, the coordinates of the navigation starting point may be preset to (0, 0), that is, the navigation starting point is used as the origin of the navigation coordinate system.

[0121] The location information of the reachable navigation point may be the coordinates of the reachable navigation point in the navigation coordinate system. For ease of description, the location information of the reachable navigation point is recorded as: where i∈{1, 2, ..., k t}, k t Indicates the number of reachable navigation points corresponding to the t-th moment (also called time sequence t). The t-th moment is also the moment (or time sequence information) corresponding to the current navigation point. The moment can be an increasing positive integer, such as: the moment corresponding to the first navigation point (navigation starting point) is 1, the moment corresponding to the second navigation point is 2, and so on.

[0122] The position information of each reachable navigation point is input into the position encoder to obtain the position feature vector corresponding to each reachable navigation point. For the convenience of description, the position feature vector corresponding to the reachable navigation point is recorded as: Among them, f p (·) indicates a position encoder.

[0123] S13. Obtain an image feature vector corresponding to a reachable navigation point corresponding to the navigation starting point.

[0124] The image corresponding to each reachable navigation point is input into the image feature extractor to obtain the initial feature vector of the image corresponding to each reachable navigation point. For the convenience of description, the initial feature vector is recorded as: v it , where i∈{1, 2, ..., k t}, k t It represents the number of reachable navigation points corresponding to the t-th moment. If the agent is at the navigation starting point, t=1.

[0125] Add the initial feature vector and the position feature vector corresponding to the reachable navigation point to obtain the image feature vector corresponding to the reachable navigation point. The formula is as follows:

[0126]

[0127] in, is the image feature vector corresponding to the i-th reachable navigation point at the t-th time. If the agent is currently at the navigation starting point, t=1.

[0128] S14. According to the image feature vector corresponding to the reachable point navigation point, the panoramic view feature vector corresponding to the starting point navigation starting point is obtained.

[0129] The image feature vectors corresponding to each reachable navigation point corresponding to the navigation starting point are input into the panoramic view feature extractor to obtain the panoramic view feature vector corresponding to the navigation starting point. For the convenience of description, the panoramic view feature vector is recorded as: Among them, f V (·) denotes the panoramic view feature extractor.

[0130] S15. Obtain a time series feature vector corresponding to the moment of the navigation starting point.

[0131] The time corresponding to the navigation starting point is input into the time series feature extractor to obtain the time series feature vector corresponding to the time of the navigation starting point. For the convenience of description, the time series feature vector is recorded as: f T (t), where f T (·) represents the temporal feature extractor, if the agent is at the navigation starting point, t = 1.

[0132] S16. Obtain the posture feature vector corresponding to the navigation starting point.

[0133] The posture information of the agent at the navigation starting point is input into the action feature extractor to obtain the posture feature vector corresponding to the agent during navigation. The posture information is the direction of the agent, which can be expressed as r t =(sinθ t , cosoθ t , sinφ t , cosφ t ) represents, where θ t is the horizontal orientation angle of the agent at time t, φ tis the vertical orientation angle of the agent at time t. If the agent is at the navigation starting point, t = 1. For ease of description, the posture feature vector is recorded as: A (r t ), where f A (·) represents the action feature extractor.

[0134] S17. Add the panoramic view feature vector, the time sequence feature vector, the position feature vector and the posture feature vector corresponding to the navigation starting point to obtain a navigation point marking vector corresponding to the navigation starting point.

[0135] The formula can be expressed as follows:

[0136]

[0137] Among them, h t is the navigation point marking vector corresponding to the t-th navigation point. If the agent is at the navigation starting point, t=1.

[0138] S18. Select a plurality of initial candidate navigation endpoints from among the candidate navigation endpoints whose distances from the navigation starting point are a preset distance.

[0139] Among the candidate navigation destinations obtained in step 201, q candidate navigation points whose distances from the navigation starting point are preset distances are selected, where q is a preset value and is a positive integer greater than 1.

[0140] S19. For each of the multiple selected initial candidate navigation endpoints, obtain a position feature vector corresponding to the initial candidate navigation endpoint, multiply the position feature vector corresponding to the initial candidate navigation endpoint by the overall feature vector corresponding to the navigation instruction, and obtain a navigation endpoint mark vector corresponding to the initial candidate navigation endpoint.

[0141] The position information of each selected initial candidate navigation endpoint is input into the position encoder to obtain the position feature vector corresponding to each initial candidate navigation endpoint. For the convenience of description, the position information of the initial candidate navigation endpoint is recorded as: i , where i∈{1, 2, ..., q}, q is the number of candidate navigation endpoints selected in S18.

[0142] Then, for each selected initial candidate navigation endpoint, the position feature vector corresponding to the initial navigation endpoint is multiplied by the overall feature vector corresponding to the navigation instruction to obtain the navigation endpoint mark vector corresponding to the initial candidate navigation endpoint. The formula is as follows:

[0143] c i =f p (l i )*x0

[0144] Among them, c i is the navigation endpoint mark vector corresponding to the i-th candidate navigation endpoint.

[0145] S20, using the overall feature vector corresponding to the navigation instruction as a global tag vector. The global tag vector, the language feature vector corresponding to each word in the navigation instruction, the navigation point tag vector corresponding to the navigation starting point, the image feature vector corresponding to the reachable navigation point, and the navigation endpoint tag vector corresponding to the initial candidate navigation endpoint are input into a pre-trained Transformer model to obtain an updated global tag vector, an updated navigation point tag vector corresponding to the navigation starting point, an updated image feature vector corresponding to the reachable navigation point, and multiple updated navigation endpoint tag vectors.

[0146] Among them, the Transformer model is a transformer-based cross-modal structured (Cross-modal Structured Transformer) model. In S20, the number of updated navigation endpoint mark vectors output by the model is the same as the number of candidate navigation endpoints selected in S18, and each updated navigation endpoint mark vector corresponds to a candidate navigation endpoint.

[0147] S21. Determine a navigation point to be driven among the reachable navigation points according to the updated image feature vector and the updated global marking vector corresponding to the reachable navigation points.

[0148] For each updated image feature vector corresponding to the output reachable navigation point, calculate the similarity between the updated image feature vector corresponding to the reachable navigation point and the updated global label vector. The reachable navigation point corresponding to the highest similarity is used as the navigation point to be driven. The similarity can be cosine distance, Euclidean distance, etc.

[0149] S22: Determine a predicted navigation endpoint from among a plurality of initial candidate navigation endpoints according to the updated navigation endpoint mark vectors respectively corresponding to the plurality of candidate navigation endpoints.

[0150] The output multiple updated navigation endpoint mark vectors are input into the navigation endpoint prediction model to obtain the confidence of the initial candidate navigation endpoints corresponding to the multiple updated navigation endpoint mark vectors, and the initial candidate navigation endpoint corresponding to the maximum confidence is used as the predicted navigation endpoint. The navigation endpoint prediction model consists of a multilayer perceptron (MLP) and a normalization function.

[0151] The following describes how to determine the navigation point to be driven and predict the navigation end point when the agent is at a navigation point other than the navigation starting point. Figure 5 , it can be determined that the process may include the following steps:

[0152] S30, obtaining a first updated global mark vector output by the Transformer model at the previous historical navigation point, a first updated navigation point mark vector corresponding to historical navigation points other than the previous historical navigation point, and a plurality of first updated navigation endpoint mark vectors.

[0153] Among them, the navigation points (including the navigation starting point) that the agent has traveled in this navigation can be called historical navigation points, and the navigation point where the agent is currently located is not counted as a historical navigation point. Assuming that the current navigation point is the t-th navigation point, the previous historical navigation point is the t-1-th navigation point.

[0154] For ease of description, the first updated global tag vector output by the Transformer model at the previous historical navigation point is denoted as: g t-1 . The first updated navigation point marker vector output by the transformer model at the previous historical navigation point is recorded as: in, It represents the updated navigation point mark vector corresponding to the first navigation point (navigation starting point) in this navigation output by the transformer model at the t-1th moment (the moment of the current navigation point is taken as the tth moment, and the t-1th moment is the moment of the previous historical navigation point). It represents the updated navigation point mark vector corresponding to the second navigation point in this navigation outputted by the transformer model at time t-1, and so on. The multiple first updated navigation end point mark vectors outputted by the transformer model at the previous historical navigation point are recorded as: Wherein, q is the number of candidate navigation endpoints selected at the navigation starting point.

[0155] S31. Obtain a location feature vector corresponding to a previous historical navigation point.

[0156] For ease of description, the position feature vector corresponding to the previous historical navigation point is recorded as: f p (l t-1 ), where l t -1 The location information of the previous historical navigation point.

[0157] S32. Obtain an image feature vector corresponding to a reachable navigation point corresponding to a previous historical navigation point.

[0158] For ease of description, the image feature vector corresponding to the reachable navigation point corresponding to the previous historical navigation point is recorded as where i∈{1, 2, ..., k t-1}, k t-1It indicates the number of reachable navigation points corresponding to the t-1th moment, that is, the number of reachable navigation points corresponding to the previous historical navigation point.

[0159] S33. Calculate a panoramic view feature vector corresponding to the previous historical navigation point based on the image feature vector corresponding to the reachable navigation point corresponding to the previous historical navigation point.

[0160] The image feature vectors corresponding to each reachable navigation point corresponding to the previous historical navigation point are input into the panoramic view feature extractor to obtain the panoramic view feature vector corresponding to the navigation starting point. For ease of description, the panoramic view feature vector is recorded as:

[0161] S34. Obtain the time series feature vector of the moment corresponding to the previous historical navigation point.

[0162] The time corresponding to the previous historical navigation point is input into the time series feature extractor to obtain the time series feature vector corresponding to the previous historical navigation point. For the convenience of description, the time series feature vector is recorded as: T (t-1).

[0163] S35. Obtain the posture feature vector corresponding to the previous historical navigation point.

[0164] The posture information of the agent at the previous historical navigation point is input into the action feature extractor to obtain the posture feature vector corresponding to the navigation of the agent. The posture information of the agent at the previous historical navigation point can be expressed as: t -1 =(sinθ t-1 , cosθ t-1 , sinφ t-1 , cosφ t-1 ) represents, where θ t-1 is the horizontal heading angle of the agent at time t-1 (i.e. the orientation of the agent when it reaches the previous historical navigation point), φ t is the vertical orientation angle of the agent at the t-1th moment. For the convenience of description, the posture feature vector corresponding to the previous historical navigation point is recorded as: A (r t-1 ).

[0165] S36. Add the panoramic view feature vector, the time series feature vector, the position feature vector and the posture feature vector corresponding to the previous historical navigation point to obtain a navigation point mark vector corresponding to the previous historical navigation point.

[0166] The formula can be expressed as follows:

[0167]

[0168] S37. Obtain the image feature vector corresponding to the reachable navigation point corresponding to the current navigation point.

[0169] The specific acquisition method is the same as the above S13, which will not be described in detail here. For the convenience of description, the image feature vector corresponding to the reachable navigation point corresponding to the current navigation point is recorded as:

[0170] S38. Determine the adjacency matrix corresponding to the historical navigation point based on the topological information of the historical navigation point, and use the adjacency matrix as the attention matrix of the first updated navigation point mark vector corresponding to the historical navigation points other than the previous historical navigation point and the navigation point mark vector corresponding to the previous historical navigation point.

[0171] According to the topological information of the historical navigation points, an undirected graph corresponding to the historical navigation points is generated, and according to the undirected graph, an adjacency matrix corresponding to the historical navigation points is generated. Assuming that the current navigation point is the fourth navigation point, the intelligent agent travels from the first navigation point to the second navigation point, and then from the second navigation point to the third navigation point, in the undirected graph corresponding to the historical navigation points, there is a connection relationship between the first navigation point and the second navigation point, and there is a connection relationship between the second navigation point and the third navigation point, then the generated adjacency matrix can be as follows:

[0172]

[0173] S39. Input the first updated global mark vector, the language feature vector corresponding to each word in the navigation instruction, the first updated navigation point mark vector, the navigation point mark vector corresponding to the previous historical navigation point, the attention matrix, the image feature vector corresponding to the reachable navigation point corresponding to the current navigation point, and multiple first updated navigation endpoint mark vectors into the transformer model to obtain the second updated global mark vector, the second updated navigation point mark vector corresponding to each historical navigation point, the second updated image feature vector corresponding to the reachable navigation before, and multiple second updated navigation endpoint mark vectors.

[0174] For ease of description, the second updated global marking vector is denoted as: g t , the second updated navigation point mark vector is recorded as: The second updated image feature vector is recorded as: The second updated navigation endpoint mark vector is recorded as:

[0175] S40. Determine a navigation point to be driven from the reachable navigation points and the historical navigation points according to the second updated global marking vector, the second updated navigation point marking vector and the second updated image feature vector.

[0176] Calculate the similarity between each second updated navigation point mark vector and the second updated global mark vector, and calculate the similarity between each second updated image feature vector and the second updated global mark vector. Determine the navigation point corresponding to the maximum similarity (which may be a reachable navigation point or a historical navigation point) as the navigation point to be driven.

[0177] S41, determining the update candidate navigation endpoints to which the plurality of second update navigation endpoint mark vectors respectively correspond.

[0178] For each second update of the output navigation end point marker vector (i.e. ), according to the second updated navigation end point mark vector and the overall feature vector corresponding to the navigation instruction, determine the updated position feature vector corresponding to each updated navigation point. The formula can be expressed as follows:

[0179]

[0180] Among them, f p (l′ i ) is the updated position feature vector corresponding to the i-th updated candidate navigation endpoint, It is the second updated navigation endpoint mark vector corresponding to the i-th updated candidate navigation endpoint. and x0 are known.

[0181] Then, each determined updated position feature vector is input into the position decoder to obtain the position information of each updated candidate navigation endpoint.

[0182] The position decoder corresponds to the position encoder, and the position information is input into the position encoder to obtain the corresponding position feature information, and the position feature information is input into the position decoder to obtain the corresponding position information.

[0183] S42: Determine a predicted navigation endpoint from among multiple updated candidate navigation endpoints according to multiple second updated navigation endpoint mark vectors.

[0184] The specific processing of step S42 is the same as that of step S22, and will not be described in detail here.

[0185] Step 205: Control the intelligent agent to travel to the navigation point to be traveled.

[0186] In implementation, after determining the navigation point to be driven and the predicted navigation end point, the agent drives toward the navigation point to be driven. If the navigation point to be driven and the predicted navigation end point meet the close distance condition, the agent stops driving after reaching the navigation point to be driven. The close distance condition can be that the distance between the navigation point to be driven and the predicted navigation end point is less than a preset value, that is, when the distance between the navigation point to be driven and the predicted navigation end point is less than the preset value, it can be approximately considered that the navigation point to be driven and the predicted navigation end point are the same point.

[0187] The preset value may be configured accordingly according to the requirement for navigation accuracy. For example, if high navigation accuracy is required, the preset value may be configured to be smaller, such as 5 cm.

[0188] Based on the same technical concept, the embodiment of the present application also provides a device for intelligent body navigation, such as Figure 6 As shown, the device includes a determination module 610, a prediction module 620, and a navigation module 630, wherein:

[0189] A determination module 610 is used to determine a first reachable navigation point corresponding to a current navigation point, and obtain a first image corresponding to the first reachable navigation point;

[0190] The prediction module 620 is used to determine the predicted navigation end point according to the navigation instruction of this navigation, the position information corresponding to the historical navigation point, the posture information of the intelligent agent at the historical navigation point, the topological information of the historical navigation point and the first image, and determine the navigation point to be driven from the candidate navigation points, wherein the candidate navigation points include the historical navigation point and the first reachable navigation point;

[0191] The navigation module 630 is used to control the intelligent agent to travel to the navigation point to be traveled.

[0192] In a possible implementation, the prediction module 620 is further configured to:

[0193] Determine a second reachable navigation point corresponding to the navigation starting point, obtain a second image corresponding to the second reachable navigation point, and determine an image feature vector corresponding to the second reachable navigation point based on the second image;

[0194] Determine a language feature vector corresponding to each word in the navigation instruction and an overall feature vector corresponding to the navigation instruction;

[0195] Determine a navigation point marking vector corresponding to the navigation starting point according to the image feature vector corresponding to the second reachable navigation point, the position information corresponding to the navigation starting point, the posture information of the agent corresponding to the navigation starting point, and the timing information corresponding to the navigation starting point;

[0196] Selecting a plurality of initial candidate navigation endpoints from among the candidate navigation endpoints whose distance from the navigation starting point is a preset distance, and determining navigation endpoint mark vectors respectively corresponding to the plurality of initial candidate navigation endpoints;

[0197] Based on the image feature vector corresponding to the second reachable navigation point, the language feature vector and the overall feature vector, the navigation point mark vector corresponding to the navigation starting point and the navigation end point mark vector corresponding to each initial candidate navigation end point, a navigation point to be traveled is determined among the second reachable navigation points, and a predicted navigation end point is determined among the multiple initial candidate navigation end points.

[0198] In a possible implementation, the prediction module 620 is used to:

[0199] Taking the overall feature vector as a global tag vector, the language feature vector, the global tag vector, the navigation point tag vector corresponding to the navigation starting point, the image feature vector corresponding to the second reachable navigation point, and the navigation endpoint tag vector corresponding to each initial candidate navigation endpoint are input into a transformer model to obtain a first updated global tag vector, a first updated navigation point tag vector corresponding to the navigation starting point, a first updated image feature vector corresponding to the second reachable navigation point, and first updated navigation endpoint tag vectors corresponding to each of the multiple initial candidate navigation endpoints;

[0200] Determine a navigation point to be driven among the second reachable navigation points according to the first updated image feature vector and the first updated global marking vector;

[0201] A predicted navigation endpoint is determined from among the multiple initial candidate navigation endpoints according to the first updated navigation endpoint mark vector.

[0202] In a possible implementation, the prediction module 620 is used to:

[0203] Obtaining a second updated global marking vector obtained by the transformer model at a previous historical navigation point, second updated navigation point marking vectors corresponding to historical navigation points other than the previous historical navigation point, and a plurality of second updated navigation endpoint marking vectors;

[0204] Determine a navigation point marking vector corresponding to the previous navigation point according to an image feature vector corresponding to a third reachable navigation point corresponding to the previous navigation point, position information corresponding to the previous navigation point, posture information of the agent at the previous navigation point, and timing information corresponding to the previous navigation point;

[0205] Determining, according to the first image, an image feature vector corresponding to the first reachable navigation point;

[0206] According to the second updated global mark vector, the language feature vector corresponding to each word in the navigation instruction, the second updated navigation point mark vector, the navigation point mark vector corresponding to the previous historical navigation point, the topological information of the historical navigation point, the image feature vector corresponding to the first reachable navigation point and the multiple second updated navigation endpoint mark vectors, the navigation point to be traveled is determined among the candidate navigation points, and the predicted navigation endpoint is determined.

[0207] In a possible implementation, the prediction module 620 is used to:

[0208] Determining a location feature vector corresponding to the first reachable navigation point according to the location information corresponding to the first reachable navigation point;

[0209] Inputting the first image into an image feature extractor to obtain an initial feature vector of the first image;

[0210] The initial feature vector and the position feature vector corresponding to the first reachable navigation point are added to obtain an image feature vector corresponding to the first reachable navigation point.

[0211] In a possible implementation, the prediction module 620 is used to:

[0212] Determine an adjacency matrix corresponding to the historical navigation point according to the topological information of the historical navigation point, and use the adjacency matrix as an attention matrix of the second updated navigation point mark vector and the navigation point mark vector corresponding to the previous historical navigation point;

[0213] Input the second updated global mark vector, the language feature vector corresponding to each word in the navigation instruction, the second updated navigation point mark vector, the navigation point mark vector corresponding to the previous historical navigation point, the attention matrix, the image feature vector corresponding to the first reachable navigation point, and the multiple second updated navigation endpoint mark vectors into the transformer model to obtain a third updated global mark vector, a third updated navigation point mark vector corresponding to each historical navigation point, a second updated image feature vector corresponding to the first reachable navigation point, and multiple third updated navigation endpoint mark vectors;

[0214] Determine a navigation point to be driven among candidate navigation points according to the third updated global marking vector, the third updated navigation point marking vector and the second updated image feature vector;

[0215] Determine the updated candidate navigation endpoints to which the plurality of third updated navigation endpoint mark vectors respectively correspond;

[0216] According to the plurality of third updated navigation endpoint mark vectors, a predicted navigation endpoint is determined from the updated candidate navigation endpoints respectively corresponding to the plurality of third updated navigation endpoint mark vectors.

[0217] In a possible implementation, the prediction module 620 is used to:

[0218] Calculating the similarity between the second updated image feature vector corresponding to each first reachable navigation point and the third updated global label vector as the confidence corresponding to each first reachable navigation point;

[0219] Calculate the similarity between the third updated navigation point mark vector corresponding to each historical navigation point and the third updated global mark vector as the confidence corresponding to each historical navigation point;

[0220] Among the first reachable navigation points and the historical navigation points, a navigation point corresponding to the maximum confidence is determined as the navigation point to be driven.

[0221] In a possible implementation, the prediction module 620 is used to:

[0222] Inputting the plurality of third updated navigation endpoint mark vectors into a navigation endpoint prediction model to obtain confidences of the updated candidate navigation endpoints respectively corresponding to the plurality of third updated navigation endpoint mark vectors;

[0223] The updated candidate navigation endpoint corresponding to the maximum confidence is used as the predicted navigation endpoint.

[0224] In a possible implementation, the navigation endpoint prediction model is composed of a multi-layer perceptron and a normalization function.

[0225] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0226] It should be noted that: the intelligent body navigation device provided in the above embodiment only uses the division of the above functional modules as an example for intelligent body navigation. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the intelligent body navigation device provided in the above embodiment and the method embodiment of intelligent body navigation belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0227] Figure 7 The structure block diagram of an intelligent agent 500 provided by an exemplary embodiment of the present application is shown. The intelligent agent 500 may be a drone, an unmanned delivery vehicle, etc.

[0228] Typically, the agent 500 includes a processor 501 and a memory 502 .

[0229] The processor 501 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 501 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 501 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 501 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 501 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0230] The memory 502 may include one or more computer-readable storage media, which may be non-transitory. The memory 502 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 502 is used to store at least one instruction, which is used to be executed by the processor 501 to implement the global positioning method provided in the method embodiment of the present application.

[0231] In some embodiments, the agent 500 may also optionally include: a peripheral device interface 503 and at least one peripheral device. The processor 501, the memory 502 and the peripheral device interface 503 may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface 503 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 504, a display screen 505, a camera assembly 506, an audio circuit 507, a positioning assembly 508 and a power supply 509.

[0232] The peripheral device interface 503 may be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 501 and the memory 502. In some embodiments, the processor 501, the memory 502, and the peripheral device interface 503 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 501, the memory 502, and the peripheral device interface 503 may be implemented on a separate chip or circuit board, which is not limited in this embodiment.

[0233] The radio frequency circuit 504 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 504 communicates with the communication network and other communication devices through electromagnetic signals. The radio frequency circuit 504 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the radio frequency circuit 504 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The radio frequency circuit 504 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes but is not limited to: the World Wide Web, a metropolitan area network, an intranet, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 504 may also include circuits related to NFC (Near Field Communication), which is not limited in this application.

[0234] The display screen 505 is used to display the UI (User Interface). The UI may include graphics, text, icons, videos, and any combination thereof. When the display screen 505 is a touch display screen, the display screen 505 also has the ability to collect touch signals on the surface or above the surface of the display screen 505. The touch signal can be input to the processor 501 as a control signal for processing. At this time, the display screen 505 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 505 can be one, arranged on the front panel of the intelligent body 500; in other embodiments, the display screen 505 can be at least two, respectively arranged on different surfaces of the intelligent body 500 or in a folding design; in other embodiments, the display screen 505 can be a flexible display screen, arranged on the outer surface of the intelligent body 500. Even, the display screen 505 can also be set to a non-rectangular irregular shape, that is, a special-shaped screen. The display screen 505 can be made of materials such as LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode, organic light-emitting diode).

[0235] The camera assembly 506 is used to capture images or videos. Optionally, the camera assembly 506 includes a front camera and a rear camera. Typically, the front camera is arranged on the front panel of the terminal, and the rear camera is arranged on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize the panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 506 may also include a flash. The flash can be a monochrome temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0236] The audio circuit 507 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals and input them into the processor 501 for processing, or input them into the radio frequency circuit 504 to achieve voice communication. For the purpose of stereo acquisition or noise reduction, there can be multiple microphones, which are respectively arranged at different parts of the intelligent body 500. The microphone can also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signal from the processor 501 or the radio frequency circuit 504 into sound waves. The speaker can be a traditional film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signal into sound waves audible to humans, but also convert the electrical signal into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 507 may also include a headphone jack.

[0237] The positioning component 508 is used to locate the current geographical location of the intelligent body 500 to achieve navigation or LBS (Location Based Service). The positioning component 508 can be a positioning component based on the GPS (Global Positioning System), Beidou system or Galileo system.

[0238] The power supply 509 is used to power various components in the intelligent body 500. The power supply 509 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 509 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged through a wired line, and a wireless rechargeable battery is a battery that is charged through a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0239] In some embodiments, the intelligent agent 500 further includes one or more sensors 510 . The one or more sensors 510 include, but are not limited to, an acceleration sensor 511 , a gyroscope sensor 512 , and a proximity sensor 513 .

[0240] The acceleration sensor 511 can detect the magnitude of acceleration on the three coordinate axes of the coordinate system established by the agent 500. For example, the acceleration sensor 511 can be used to detect the components of gravity acceleration on the three coordinate axes. The processor 501 can control the display screen 505 to display the user interface in a horizontal view or a vertical view according to the gravity acceleration signal collected by the acceleration sensor 511. The acceleration sensor 511 can also be used to collect motion data of games or users.

[0241] The gyro sensor 512 can detect the body direction and rotation angle of the intelligent body 500, and the gyro sensor 512 can cooperate with the acceleration sensor 511 to collect the user's 3D actions on the intelligent body 500. The processor 501 can realize the following functions based on the data collected by the gyro sensor 512: motion sensing, image stabilization during shooting, and inertial navigation.

[0242] The proximity sensor 513, also called a distance sensor, is usually disposed on the front panel of the intelligent body 500. The proximity sensor 513 is used to collect the distance between an external object and the front face of the intelligent body 500.

[0243] Those skilled in the art will understand that Figure 7 The structure shown in the figure does not constitute a limitation on the intelligent agent 500, and it may include more or less components than those shown in the figure, or combine certain components, or adopt a different arrangement of components.

[0244] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including instructions, which can be executed by a processor in an agent to complete the method of agent navigation in the above embodiment. The computer-readable storage medium can be non-transitory. For example, the computer-readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), a magnetic tape, a floppy disk, and an optical data storage device.

[0245] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0246] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

[0247] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals (including but not limited to signals transmitted between user terminals and other devices, etc.) involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions. For example, the images and location information involved in this application are all obtained with full authorization.

Claims

1. A method for intelligent agent navigation, characterized in that: The method comprises: Determine a first reachable navigation point corresponding to the current navigation point, and obtain a first image corresponding to the first reachable navigation point; According to the navigation instruction of this navigation, the location information corresponding to the historical navigation point, the posture information of the intelligent body at the historical navigation point, the topological information of the historical navigation point and the first image, the predicted navigation end point is determined, and the navigation point to be driven is determined among the candidate navigation points, wherein the candidate navigation points include the historical navigation point and the first reachable navigation point; the intelligent body is controlled to drive to the navigation point to be driven; Wherein, determining the navigation point to be traveled and determining the predicted navigation end point further comprises: Determine a second reachable navigation point corresponding to the navigation starting point, obtain a second image corresponding to the second reachable navigation point, and determine an image feature vector corresponding to the second reachable navigation point based on the second image; Determine a language feature vector corresponding to each word in the navigation instruction and an overall feature vector corresponding to the navigation instruction; Determine a navigation point marking vector corresponding to the navigation starting point according to the image feature vector corresponding to the second reachable navigation point, the position information corresponding to the navigation starting point, the posture information of the agent corresponding to the navigation starting point, and the timing information corresponding to the navigation starting point; Selecting a plurality of initial candidate navigation endpoints from among the candidate navigation endpoints whose distance from the navigation starting point is a preset distance, and determining navigation endpoint mark vectors respectively corresponding to the plurality of initial candidate navigation endpoints; Based on the image feature vector corresponding to the second reachable navigation point, the language feature vector and the overall feature vector, the navigation point mark vector corresponding to the navigation starting point and the navigation end point mark vector corresponding to each initial candidate navigation end point, the navigation point to be traveled is determined among the second reachable navigation points, and the predicted navigation end point is determined among the multiple initial candidate navigation end points.

2. The method according to claim 1, characterized in that: The method of determining a navigation point to be driven from among the second reachable navigation points, and determining a predicted navigation endpoint from among the multiple initial candidate navigation endpoints according to the image feature vector corresponding to the second reachable navigation point, the language feature vector and the overall feature vector, the navigation point mark vector corresponding to the navigation starting point and the navigation endpoint mark vector corresponding to each initial candidate navigation endpoint, comprises: Taking the overall feature vector as a global tag vector, the language feature vector, the global tag vector, the navigation point tag vector corresponding to the navigation starting point, the image feature vector corresponding to the second reachable navigation point, and the navigation endpoint tag vector corresponding to each initial candidate navigation endpoint are input into a transformer model to obtain a first updated global tag vector, a first updated navigation point tag vector corresponding to the navigation starting point, a first updated image feature vector corresponding to the second reachable navigation point, and first updated navigation endpoint tag vectors corresponding to each of the multiple initial candidate navigation endpoints; Determine a navigation point to be driven among the second reachable navigation points according to the first updated image feature vector and the first updated global marking vector; A predicted navigation endpoint is determined from among the multiple initial candidate navigation endpoints according to the first updated navigation endpoint mark vector.

3. The method according to claim 2, characterized in that The method of determining a predicted navigation end point according to the navigation instruction of the current navigation, the position information corresponding to the historical navigation point, the posture information of the intelligent body at the historical navigation point, the topological information of the historical navigation point and the first image, and determining the navigation point to be driven from the candidate navigation points includes: Obtaining a second updated global marking vector obtained by the transformer model at a previous historical navigation point, second updated navigation point marking vectors corresponding to historical navigation points other than the previous historical navigation point, and a plurality of second updated navigation endpoint marking vectors; Determine a navigation point marking vector corresponding to the previous navigation point according to an image feature vector corresponding to a third reachable navigation point corresponding to the previous navigation point, position information corresponding to the previous navigation point, posture information of the agent at the previous navigation point, and timing information corresponding to the previous navigation point; Determining, according to the first image, an image feature vector corresponding to the first reachable navigation point; According to the second updated global mark vector, the language feature vector corresponding to each word in the navigation instruction, the second updated navigation point mark vector, the navigation point mark vector corresponding to the previous historical navigation point, the topological information of the historical navigation point, the image feature vector corresponding to the first reachable navigation point and the multiple second updated navigation endpoint mark vectors, the navigation point to be traveled is determined among the candidate navigation points, and the predicted navigation endpoint is determined.

4. The method according to claim 3, characterized in that: The determining, according to the first image, an image feature vector corresponding to the first reachable navigation point includes: Determine, according to the location information corresponding to the first reachable navigation point, a location feature vector corresponding to the first reachable navigation point; input the first image into an image feature extractor to obtain an initial feature vector of the first image; The initial feature vector and the position feature vector corresponding to the first reachable navigation point are added to obtain an image feature vector corresponding to the first reachable navigation point.

5. The method according to claim 3, characterized in that: The method further comprises: determining a navigation point to be driven from among candidate navigation points according to the second updated global mark vector, a language feature vector corresponding to each word in the navigation instruction, the second updated navigation point mark vector, a navigation point mark vector corresponding to the previous historical navigation point, topological information of the historical navigation point, an image feature vector corresponding to the first reachable navigation point, and the plurality of second updated navigation endpoint mark vectors, and determining a predicted navigation endpoint, including: Determine an adjacency matrix corresponding to the historical navigation point according to the topological information of the historical navigation point, and use the adjacency matrix as an attention matrix of the second updated navigation point mark vector and the navigation point mark vector corresponding to the previous historical navigation point; Input the second updated global mark vector, the language feature vector corresponding to each word in the navigation instruction, the second updated navigation point mark vector, the navigation point mark vector corresponding to the previous historical navigation point, the attention matrix, the image feature vector corresponding to the first reachable navigation point, and the multiple second updated navigation endpoint mark vectors into the transformer model to obtain a third updated global mark vector, a third updated navigation point mark vector corresponding to each historical navigation point, a second updated image feature vector corresponding to the first reachable navigation point, and multiple third updated navigation endpoint mark vectors; Determine a navigation point to be driven among candidate navigation points according to the third updated global marking vector, the third updated navigation point marking vector and the second updated image feature vector; Determine the updated candidate navigation endpoints to which the plurality of third updated navigation endpoint mark vectors respectively correspond; According to the plurality of third updated navigation endpoint mark vectors, a predicted navigation endpoint is determined from the updated candidate navigation endpoints respectively corresponding to the plurality of third updated navigation endpoint mark vectors.

6. The method according to claim 5, characterized in that The method of determining a navigation point to be driven from among candidate navigation points according to the third updated global mark vector, the third updated navigation point mark vector and the second updated image feature vector comprises: Calculating the similarity between the second updated image feature vector corresponding to each first reachable navigation point and the third updated global label vector as the confidence corresponding to each first reachable navigation point; Calculate the similarity between the third updated navigation point mark vector corresponding to each historical navigation point and the third updated global mark vector as the confidence corresponding to each historical navigation point; Among the first reachable navigation points and the historical navigation points, a navigation point corresponding to the maximum confidence is determined as the navigation point to be driven.

7. The method according to claim 5 or 6, characterized in that: The step of determining, according to the plurality of third updated navigation endpoint mark vectors, a predicted navigation endpoint from updated candidate navigation endpoints respectively corresponding to the plurality of third updated navigation endpoint mark vectors, comprises: Inputting the plurality of third updated navigation endpoint mark vectors into a navigation endpoint prediction model to obtain confidences of the updated candidate navigation endpoints respectively corresponding to the plurality of third updated navigation endpoint mark vectors; The updated candidate navigation endpoint corresponding to the maximum confidence is used as the predicted navigation endpoint.

8. The method according to claim 7, characterized in that The navigation endpoint prediction model consists of a multi-layer perceptron MLP and a normalization function.

9. An intelligent agent, characterized in that: The agent includes a processor and a memory, wherein the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the agent navigation method according to any one of claims 1-8.

Images