Natural language instruction processing method and device, equipment and storage medium

Abstract

The invention discloses a natural language instruction processing method and device, equipment and a storage medium. The generalization ability of a model and the accuracy of generating a navigation track can be improved. The method comprises the steps of performing feature processing on a natural language instruction and at least one image frame, and determining an instruction semantic feature and at least one visual feature; the at least one image frame comprises an image frame collected from at least one view angle at at least one moment; the task type of the natural language instruction comprises a navigation task; determining at least one indicator based on the at least one image frame; the indicator is used for identifying at least one of visual angle and time information of the image frame; and reasoning based on the instruction semantic feature, the at least one visual feature and the at least one indicator through a large language model, and determining a navigation trajectory corresponding to the natural language instruction of the navigation task.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a natural language instruction processing method and device, equipment and a storage medium. BACKGROUND

[0002] At present, the related technology in the field of intelligent navigation combines a large language model with a robot navigation system, parses a natural language instruction using the large language model, performs multi-modal fusion on visual features extracted from an image sequence collected by a camera, and then generates a navigation path. However, most of the methods of the related technology are trained under a single type of robot platform or a fixed camera configuration, and lack the ability to universally adapt to hardware platforms of different shapes, sizes and camera configurations. Therefore, when the trained model is applied to other hardware platforms, the model may have ambiguity in understanding different forms of multi-modal information input, thereby causing the navigation path predicted by the model to be inaccurate and reducing the generalization ability of the model on different hardware platforms. SUMMARY

[0003] The present application aims to provide a natural language instruction processing method, device, equipment and storage medium, which can improve the generalization ability of the model and the accuracy of generating a navigation trajectory.

[0004] The technical solution of the present application is as follows:

[0005] In a first aspect, the present application provides a natural language instruction processing method, which comprises:

[0006] performing feature processing on a natural language instruction and at least one image frame to determine instruction semantic features and at least one visual feature; the at least one image frame comprises an image frame collected from at least one view angle at at least one time; the task type of the natural language instruction includes a navigation task;

[0007] determining at least one indicator based on the at least one image frame; the indicator is used to identify at least one of the view angle and the time information of the image frame;

[0008] determining a navigation trajectory corresponding to the natural language instruction of the navigation task by a large language model based on the instruction semantic features, the at least one visual feature and the at least one indicator.

[0009] In a second aspect, the present application provides a natural language instruction processing device, which comprises:

[0010] The feature processing module is configured to perform feature processing on the natural language instruction and the at least one image frame, and determine instruction semantic features and at least one visual feature; the at least one image frame includes an image frame collected from at least one view angle at at least one time point; and the task type of the natural language instruction includes a navigation task.

[0011] The indicator generation module is configured to determine at least one indicator based on the at least one image frame; and the indicator is used to identify at least one of the view angle and the time information of the image frame.

[0012] The reasoning module is configured to perform reasoning on the instruction semantic features, the at least one visual feature, and the at least one indicator based on a large language model, and determine a navigation track corresponding to the natural language instruction.

[0013] In a third aspect, the present application provides an electronic device, comprising a memory and a processor; wherein,

[0014] The memory is configured to store executable instructions.

[0015] The processor is configured to implement the natural language instruction processing method provided by the embodiments of the present application when executing the executable instructions stored in the memory.

[0016] In a fourth aspect, the present application provides a computer readable storage medium storing executable instructions for causing a processor to implement the natural language instruction processing method provided by the embodiments of the present application.

[0017] In a fifth aspect, the embodiments of the present application provide a computer program product comprising a computer program or instructions, which, when executed by a processor, implement the natural language instruction processing method provided by the embodiments of the present application.

[0018] This application provides a natural language instruction processing method, apparatus, device, and storage medium. It performs feature processing on a natural language instruction and at least one image frame to determine semantic features and at least one visual feature of the instruction. The at least one image frame includes images acquired from at least one viewpoint at at least one time. The task type of the natural language instruction includes navigation tasks. Based on the at least one image frame, at least one indicator is determined. The indicator is used to identify at least one of the viewpoint and temporal information of the image frame. Using a large language model, reasoning is performed based on the instruction semantic features, at least one visual feature, and at least one indicator to determine the navigation trajectory corresponding to the natural language instruction for the navigation task. By using the above method to explicitly identify the viewpoint and / or temporal information of the image frame using the indicator, the large language model can distinguish different viewpoints and / or temporal sequences of image frames when applied on different hardware platforms, thus achieving unified observation on different hardware platforms. This allows the model to uniformly process input multi-view images and / or the temporal relationships between images without depending on the specific body shape and camera configuration of the hardware platform, thereby effectively fusing multi-view and / or temporal dynamics to generate accurate navigation trajectories. This improves the model's generalization ability on different hardware platforms and the accuracy of the generated navigation trajectories. Attached Figure Description

[0019] Figure 1 A schematic flowchart of an optional natural language instruction processing method provided in an embodiment of this application;

[0020] Figure 2 A schematic flowchart of an optional natural language instruction processing method provided in an embodiment of this application;

[0021] Figure 3 A schematic flowchart of an optional natural language instruction processing method provided in an embodiment of this application;

[0022] Figure 4 This is a schematic diagram of the structure of an intelligent model of the natural language instruction processing method based on an embodiment of this application;

[0023] Figure 5 A schematic diagram of the structure of a natural language instruction processing device provided in an embodiment of this application;

[0024] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0025] In order to make the purposes, technical solutions and advantages of the present application clearer, the following will further describe the present application in detail with reference to the drawings, and the described embodiments should not be regarded as limiting the present application, and all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0026] In the following description, “some embodiments” are referred to, which describe a subset of all possible embodiments, but it can be understood that “some embodiments” can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0027] In the following description, the terms “first\second\third” are only to distinguish similar objects, and do not represent a specific order of the objects, and it can be understood that “first\second\third” can be interchanged in a specific order or sequence as allowed, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0028] Navigation technology is an important research direction in embodied intelligence, and is applied in robots, autonomous driving and other scenarios. The navigation task usually requires a mobile body to complete path planning and movement in a physical environment according to language instructions or other forms of input. To adapt to different task requirements, the mobile body may be equipped with different types and quantities of cameras, and different camera configurations correspond to different data types and processing methods. Related research covers visual and language navigation, target following, autonomous driving and other tasks. In recent years, with the development of vision-language models, using multi-modal information for navigation tasks has become a research hotspot and promoted cross-scene and multi-task data collection and model training work.

[0029] Currently, the method of the related technology combines a large language model with a robot navigation system, uses the language model to analyze natural language instructions, and generates navigation targets or intermediate planning steps. The system usually includes a visual perception module, a language understanding module and a path planning module: the visual perception module collects image sequences from monocular or multi-camera cameras and extracts environmental semantic information; the language understanding module performs semantic analysis on the instructions based on a pre-trained language model and performs multi-modal fusion with visual features; the path planning module generates a trajectory or control instruction based on the fusion result and executes it through a motion control module. In addition, in cross-platform applications, some existing technologies train a unified Transformer strategy model on a large-scale dataset containing various robot forms and camera configurations, realizing navigation reasoning and control across devices.

[0030] However, the accuracy of the predicted navigation path of the current related technology trained multi-modal large language model is usually low when applied to the navigation task of multi-hardware platforms. The applicant found through research and experiments that the reason is that some large language models in the related technology are trained on a single view or a single camera hardware platform (such as robots or vehicles, etc.), so when they are deployed on a multi-camera hardware platform, the large language model cannot distinguish the images collected by the multi-angle cameras. Some large language models in the related technology are trained on a single type of hardware platform or a fixed camera configuration, and lack the ability to adapt to different shapes, sizes and sensor configurations. This is because for different hardware platforms, the positions, numbers, and angle ranges of the image sensors deployed are different, so the differences in the angles of the images collected by the image sensors on different hardware platforms will cause differences in geometric meaning, which is easy to cause feature matching ambiguity, and thus affect the model generalization ability and the accuracy of model reasoning.

[0031] In addition, the frame rates of the image sensors on different hardware platforms are different, the clock accuracy, frequency, reference or synchronization methods of the hardware platforms are different, and the moving speeds of different hardware platforms, such as the moving speeds of robots or vehicles, also differ. When a model trained on a certain type of hardware platform is applied to another type of hardware platform, the model is likely to have ambiguity in understanding the inherent and continuous time flow information in each image frame in the input video stream, which leads to deviations in the information such as event dynamics, action causality, and instruction execution timing perceived by the model, and thus causes errors in the path planning of the model.

[0032] In addition, in the cross-hardware platform scenario, due to the lack of unified observation and action space modeling, it is difficult to fully utilize the existing large-scale heterogeneous navigation sample data for model training, which also leads to high training cost and limited model generalization ability.

[0033] The embodiment of the application provides a natural language instruction processing method, device and equipment and a storage medium, which can uniformly process multi-view video input and natural language instructions without depending on specific body forms and camera configurations, and predict a trajectory meeting task requirements, thereby realizing efficient generalization and real-time navigation capability across different embodied platforms and task scenarios, and improving model generalization capability and accuracy of generated navigation trajectories. The natural language instruction processing method provided by the embodiment of the application is applied to an electronic device. The electronic device of the embodiment of the application can include a terminal and a server. In some embodiments, the terminal can include, but is not limited to, a robot, a drone, a vehicle, a mobile terminal and various hardware platform devices in motion modes and physical forms. In some embodiments, the server can be a physical server, a server cluster composed of multiple physical servers or a distributed system, and can also be a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDNs and basic cloud computing services such as big data and artificial intelligence platforms. The terminal and the server can be directly or indirectly connected through wired or wireless communication, which is not limited in the embodiment of the application.

[0034] Exemplarily, when the electronic device is implemented as a terminal, the terminal can obtain a natural language instruction through an interactive device such as a microphone or an interactive screen configured by the terminal, obtain at least one image frame through an image acquisition device such as at least one camera with a view angle configured by the terminal, determine at least one indicator based on the at least one image frame, and determine a navigation trajectory corresponding to the natural language instruction of the navigation task through a large language model configured on the terminal based on instruction semantic features, at least one visual feature and the at least one indicator. When the electronic device is implemented as a server, the terminal can send the acquired natural language instruction and at least one image frame to the server, and determine a navigation trajectory corresponding to the natural language instruction of the navigation task based on instruction semantic features, at least one visual feature and the at least one indicator through a large language model configured on the server. The specific selection is made according to actual conditions, which is not limited in the application.

[0035] The natural language instruction processing method provided by the embodiment of the application can be as shown in Figure 1

[0036] S101, performing feature processing on a natural language instruction and at least one image frame to determine instruction semantic features and at least one visual feature.

[0037] ​This application applies to scenarios where natural language commands are processed through an artificial intelligence-based network model on various hardware platforms, such as mobile terminals like smartphones and tablets, robots with various motion modes and physical forms, drones, and intelligent transportation tools like vehicles, to achieve tasks such as navigation and question answering.

[0038] In some embodiments, when applying this application to navigation tasks of robots (such as humanoid quadruped robots, wheeled / tracked robots, robot dogs, etc.), images or videos can be captured using image acquisition devices such as cameras deployed on the robot, thereby obtaining at least one image frame. Human-computer interaction devices deployed on the robot, such as microphones or touchscreens, can receive natural language commands issued by the user in natural language. Based on the acquired natural language commands and at least one image frame, intelligent navigation is achieved using the method of this application embodiment.

[0039] In this embodiment of the application, natural language instructions can be obtained through user voice input or text input.

[0040] In this embodiment, the task type of the natural language instruction may include navigation tasks. That is, the intent of the natural language instruction is related to navigation. For example, natural language instructions for navigation tasks may include instructions in natural language form that require the model to perform inference tasks such as path planning and trajectory generation, such as "Take me to the kitchen," "Put the cup in the kitchen," and "Take me to the nearest supermarket."

[0041] In this embodiment, natural language instructions can be feature-extracted to obtain semantic features corresponding to the natural language instructions, which serve as the instruction semantic features. For example, the natural language instructions can be segmented into words, mapped to embedding vectors, and the embedding vectors can be processed to obtain the semantic features corresponding to the natural language instructions. This embodiment does not impose specific limitations on the method of extracting instruction semantic features.

[0042] In this embodiment, at least one image frame includes an image frame acquired from at least one viewpoint at at least one moment. For example, at least one image sensor is deployed at different locations on a robot or vehicle, and each image sensor acquires video streams or images from its own viewpoint, thereby obtaining an image frame acquired from at least one viewpoint at at least one moment. For example, for navigation tasks, at least one moment may include the current moment, or it may include the current moment and at least one historical moment. At least one image frame may include an image frame corresponding to at least one viewpoint at each of the at least one moments obtained by sampling the video stream acquired by the image sensor at at least one viewpoint.

[0043] In this embodiment of the application, feature extraction is performed on each image frame in at least one image frame to obtain the visual features corresponding to each image frame, thereby determining at least one visual feature. It is understood that instruction semantic features can characterize the intent of the instruction and navigation requirements, while visual features can characterize perceptual information about the current scene, environment, or target object.

[0044] S102. Based on at least one image frame, determine at least one indicator; the indicator is used to identify at least one of the viewpoint and time information of the image frame.

[0045] In this embodiment, since at least one image frame originates from at least one viewpoint and is acquired at at least one moment, the viewpoint and / or temporal information differs between different image frames. Furthermore, due to the different shapes of different hardware platforms and the different viewpoints of image sensors, the visual information acquired by different hardware platforms differs in geometric meaning. Therefore, when a trained model is applied to a new hardware platform, the model may misunderstand the images acquired by the new hardware platform, leading to deviations in model processing such as feature matching, ultimately affecting the accuracy of the inference results. To enable the model to better distinguish image frames from different viewpoints and temporal sequences, this embodiment can generate an indicator corresponding to each image frame in the at least one image frame. This indicator identifies at least one of the viewpoint and temporal information of the image frame, thereby determining at least one indicator. In this way, even if the model is migrated to a different hardware platform, the model can accurately distinguish image frames from different viewpoints and temporal sequences on that hardware platform based on the indicator corresponding to each image frame. This allows for multi-viewpoint fusion at each moment, constructing continuous visual perception with a large viewpoint range, and improving the accuracy of subsequent trajectory prediction.

[0046] For example, the viewpoint of each image frame in at least one image frame can be angularly encoded, and / or the temporal information of each image frame, such as timestamps and frame numbers, can be temporally encoded. An indicator corresponding to each image frame is then generated based on the obtained angular and / or temporal encoding vectors. In this way, the spatial location and temporal order of image frames are identified through explicit encoding, generating an indicator for each image frame. This allows the indicator to clearly indicate the temporal relationship and / or viewpoint of at least one image frame input to the large language model, enabling the large language model to accurately understand the viewpoint and / or temporal relationship of image frames when applied to various hardware platforms, and thus perform model inference based on accurate understanding.

[0047] It should be noted that, Figure 1The execution order of S101 and S102 shown is only an optional example of the processing procedure. In actual applications, S101 and S102 can be executed in any order or in parallel. This application embodiment does not impose specific limitations.

[0048] S103. Using a large language model, reasoning is performed based on the semantic features of the instruction, at least one visual feature, and at least one indicator to determine the navigation trajectory corresponding to the natural language instruction.

[0049] In this embodiment of the application, the large language model can be trained on the hardware platform of the current electronic device or on other types of hardware platform electronic devices and applied to the hardware platform of the current electronic device.

[0050] In some embodiments, the visual features corresponding to each image frame can be combined with the indicator, and the instruction semantic features combined with at least one feature corresponding to at least one image frame can be provided together.

[0051] Using a large language model, based on the aforementioned instruction semantic features, at least one visual feature, and at least one indicator, multimodal fusion and inference of textual and visual features are performed. During the model processing of the large language model, the indicator corresponding to each image frame can be used to identify the viewpoint and / or temporal information of the visual features corresponding to that image frame, thereby effectively fusing multi-viewpoint and / or temporal dynamics. The large language model performs model inference for navigation tasks according to the instruction semantic features, predicting the navigation trajectory corresponding to the natural language instructions for navigation tasks.

[0052] For example, the navigation trajectory may include a set of trajectory points P = {P1, P2, ...} at multiple times, where the trajectory point P at each time t is... t ∈R 3 = (x,y,θ), where x and y represent the two-dimensional plane positions, and θ represents the orientation angle.

[0053] It is understood that the embodiments of this application utilize indicators to explicitly identify the viewpoint and / or temporal information of image frames. This helps the large language model distinguish different viewpoints and / or temporal sequences of image frames when applied on different hardware platforms, thereby achieving unified observation across different hardware platforms. This allows the model to uniformly process input multi-view images and / or the temporal relationships between images without depending on the specific hardware platform and camera configuration, effectively fusing multi-view and temporal dynamics to generate accurate navigation trajectories. Consequently, it improves the model's generalization ability across different hardware platforms and the accuracy of the generated navigation trajectories.

[0054] In some embodiments, the process of S103 described above may include:

[0055] Using a large language model, inference is performed based on instruction semantic features, at least one visual feature, and at least one indicator to determine the hidden state vector; using an action decoding model, the hidden state vector is decoded to determine the navigation trajectory corresponding to the natural language instructions for navigation tasks.

[0056] Among them, the large language model performs deep fusion and understanding based on the semantic features of the input instructions, at least one visual feature and at least one indicator, and outputs a set of semantically rich contextual representations, namely hidden state vectors.

[0057] In some embodiments, for navigation tasks, the large language model performs deep fusion and reasoning based on the current environment represented by at least one visual feature and at least one indicator, and the instruction intent represented by the instruction semantic features, to generate an action-oriented contextual representation as the latent state vector for the navigation task. The latent state vector for the navigation task may include high-level intent and preliminary path planning information required to perform the navigation task, providing the necessary semantic context for a subsequent dedicated action decoding model to generate navigation trajectory points. For navigation tasks, the latent state vector output by the large language model may include semantic information for generating actions. For example, for the natural language instruction "go to the kitchen to get an apple," the large language model understands the task intent based on the instruction semantic features (e.g., the task goal is to get an apple, and the target location is in the kitchen), visually observes the current environment based on at least one visual feature and at least one indicator (e.g., perceiving doors, hallways, and kitchen signs in an image), combines the task intent with the visual observation, formulates path planning for the task intent, and outputs an abstract, semantically rich latent state vector. The latent state vector contains the planning results of the large language model, such as: "First go through the living room door, then turn right into the hallway, the end is the kitchen, and the apple is on the table in the middle of the kitchen."

[0058] The action decoding model acts as a dedicated task layer. For navigation tasks, the large language model outputs the generated latent state vectors to the action decoding model, which decodes them and converts them into a set of navigation path points in physical space, thus obtaining the navigation trajectory. For example, the action decoding model can be implemented as a head network. This head network decodes the high-level intent and preliminary path planning information from the latent state vectors, converting them into specific, physically achievable navigation path points. For instance, "passing through the living room door" is transformed into "from the current point A, first walk to point B, then to point C," thus obtaining a set of navigation path points and generating a complete navigation trajectory based on these path points. Each trajectory point in the navigation trajectory represents a specific location and orientation, allowing the moving body to complete the navigation task.

[0059] Understandably, by inferring the hidden state vector from the large language model and then decoding it with the action decoding model to obtain the navigation trajectory, the decoupling of model planning and specific control execution is achieved. This allows for compatibility with action decoding models of different network types and performance in practical applications, further improving the model's generalization ability. Furthermore, since the inference speed of the large language model is not high, it is difficult to meet the real-time or high-frequency control requirements of applications such as robotics or autonomous driving if the navigation trajectory is generated by the large language model in practical applications. Therefore, generating the navigation trajectory by decoding the hidden state vector output by the large language model with the action decoding model improves the efficiency and real-time performance of navigation trajectory generation.

[0060] In some embodiments, the natural language instruction task type also includes visual question answering tasks, and the method further includes:

[0061] The hidden state vector output by the large language model is decoded using a language decoding model to determine the target text sequence; based on the target text sequence, the response text corresponding to the natural language instruction is generated.

[0062] In this embodiment, a large language model can be used to achieve multi-task reasoning. That is, in addition to navigation tasks, the large language model can also handle other task types, such as visual question-answering tasks. In some embodiments, visual question-answering tasks may include image question-answering tasks and video question-answering tasks. Video question-answering tasks may further include single-view video question-answering tasks and multi-view video question-answering tasks. Visual question-answering tasks are used to answer knowledge-based questions posed by users in natural language commands based on image or video content using the large language model.

[0063] In some embodiments, for natural language instructions in visual question-answering tasks, the same processing steps S101 and S102 are performed: feature processing is applied to the natural language instructions and at least one image frame to determine semantic features and at least one visual feature; and at least one indicator is determined based on at least one image frame. The large language model performs deep fusion and inference based on the semantic features of the instructions, at least one visual feature, and at least one indicator of the visual question-answering task to generate a content-response-oriented contextual representation as the latent state vector of the visual question-answering task.

[0064] For example, for a natural language instruction in a visual question-answering task such as "How many apples are in the picture?", the large language model performs fusion reasoning based on the instruction's semantic features, at least one visual feature, and at least one indicator to understand the task intent: to determine the number of apples, identify the apples in the image, and understand their location and quantity. The output latent state vector contains the answer information. A language decoding model decodes the latent state features output by the large language model based on the visual question-answering task, transforming the answer information in the latent state vector into a target text sequence in the form of words, and then organizing the text sequence into the response text.

[0065] For example, the language decoding model may include a visual question-answering head network, which can map the hidden state vectors onto a preset vocabulary probability distribution to generate the response text corresponding to the natural language instructions for visual question-answering tasks.

[0066] As can be seen, in this embodiment, by combining the action decoding model and the language decoding model, a dual-branch decoding network can be constructed at the output end of the large language model. The large language model realizes understanding, thinking, and planning, outputting a hidden state vector. The specific task network model decodes the hidden state vector to obtain the execution result for the specific task. It should be noted that, according to the needs of actual applications, decoding models corresponding to other task types can also be connected to the output end of the large language model. Through end-to-end training, a multi-branch decoding network can be implemented to decode the hidden state vector output by the large language model and achieve task execution. The specific selection depends on the actual situation, and this embodiment does not limit it.

[0067] In some embodiments, a large language model supporting both action decoding and language decoding models can be trained using visual question-answering sample data and navigation sample data. For example, a large number of navigation sample datasets can be collected across hardware platforms (such as humanoid robots, quadruped robots, drones, wheeled robots, and vehicles) and across tasks (such as visual and language navigation, tracking, and autonomous driving). This navigation sample dataset contains various types of instructions and scenarios involving different skills, such as following instructions, exploration, and obstacle avoidance, enabling the large language model to acquire richer, more general navigation knowledge and stronger zero-shot generalization ability. Furthermore, various open-world knowledge samples based on visual question answering, such as image-based question answering, video-based question answering, and multi-camera video question answering, are collected as visual question answering sample datasets. The initial large language model is pre-trained using the visual question answering sample dataset, and then trained on the navigation sample dataset in an end-to-end manner, thereby achieving large-scale training and comprehensive optimization to obtain the large language model. In this way, the large language model can simultaneously access navigation and question-answering data during its training. By answering knowledge-based questions ("What color is an apple?"), the model can learn and retain common-sense and attribute knowledge about the world. Thus, when performing navigation tasks (such as "go get an apple"), the knowledge about "apples" already learned within the large language model can guide or assist in the reasoning and decision-making of the navigation task, thereby generating a more reasonable navigation trajectory. Therefore, it avoids the model degenerating into a simple mapping machine that outputs monotonous navigation actions based on input, without understanding the meaning of the actions.

[0068] As can be seen, the dual-branch network structure not only supports more task types, enabling large language models to maintain language and knowledge reasoning capabilities while performing navigation tasks, but also avoids knowledge collapse caused by the black-box mapping from "instructions to trajectories" in a single architecture, thus improving the accuracy of navigation trajectory planning. This further enhances the model's cross-task generalization ability and the accuracy of the generated navigation trajectories.

[0069] Currently, video streams captured by image sensors deployed at at least one viewpoint contain a large number of image frames. These instant-captured image frames generate excessive visual features, thereby increasing model inference time and hindering practical application deployment. To accelerate model inference, related technologies typically address this issue in two ways:

[0070] Method 1: Feature Merging. This method uses a threshold to merge highly similar features. However, this strategy introduces additional computational overhead during the online merging process, leading to inconsistent inference speeds during evaluation, especially in new environments where it slows down significantly.

[0071] Method 2: Uniform sampling. This method often fails to capture the most recent observation data, thus increasing the difficulty of tasks such as tracking and autonomous driving.

[0072] To address the issue of excessive visual features generated in video streams, in some embodiments, embodiments of this application can determine the sampling strategy for image frames in the video stream using a budget constraint approach. For example... Figure 2 As shown, it includes:

[0073] S201. Obtain the current video stream.

[0074] In this embodiment, the current video stream includes image frames from at least one historical moment to the current moment, corresponding to at least one viewpoint. That is, each moment corresponds to an image frame acquired by each of the image sensors in the at least one viewpoint image sensor. For example, by using N currently deployed image sensors to acquire video stream data from N viewpoints, with each image sensor acquiring video stream data from its own viewpoint, N viewpoint image frames can be acquired at each moment.

[0075] S202. Determine the image frame sampling number threshold based on the preset total feature length threshold and the preset instruction semantic feature length threshold.

[0076] In S202, since the aforementioned at least one visual feature is obtained by feature processing of each image frame in at least one sampled image frame, it can be seen that the number of image frame samples affects the number of visual features input to the model. Since the feature length that the model can process has an upper limit, the image frame sampling number threshold can be determined based on the upper limit of the feature length that the model can process, and the instruction semantic features that also require the input feature length. Here, the image frame sampling number threshold represents the upper limit of the sampling number and is used to constrain the sampling of image frames in the current video stream.

[0077] The preset total feature length threshold represents the upper limit of the feature length that the large language model can process. In some embodiments, the aforementioned instruction semantic features, visual features, and indicators are input into the large language model in the form of tokens. Accordingly, the total feature length threshold may include the maximum number of tokens B that the large language model can process. token From B token Subtracting the maximum number of tokens that the preset instruction semantic features can occupy from the result, we can obtain the maximum number of tokens that at least one visual feature can occupy. This can then be converted into the maximum number of samples per image frame, i.e., the image frame sampling threshold B. frame In other words, the number of at least one image frame obtained by sampling the current video stream cannot exceed the image frame sampling number threshold B. frame.

[0078] S203. Determine the sampling probability corresponding to each moment based on the image frame sampling number threshold and the time difference between each moment and the current moment.

[0079] In this embodiment, since the most recent image frames can provide more up-to-date information for the model to understand and plan, the image frame sampling quantity threshold can be used as a constraint. Based on the time difference between each moment in the current video stream and the current moment, the sampling probability corresponding to each moment is determined, so as to determine whether to directly sample the image frame at that moment based on the sampling probability.

[0080] In this embodiment, the sampling probability is inversely proportional to the time difference; that is, image frames closer to the current time correspond to a higher sampling probability. Under the constraint of the image frame sampling number threshold, the sampling probability corresponding to each time moment can be determined by numerical analysis methods such as curve simulation, so as to ensure that, without exceeding the image frame sampling number threshold, recent frames that are close to the current time have a higher sampling probability.

[0081] Alternatively, in some embodiments, a predefined sampling probability function can be used to calculate the sampling probability corresponding to each moment. The sampling probability function is used to determine the sampling probability corresponding to each moment in the current video stream; the sampling probability corresponding to each moment decreases as the time difference between that moment and the current moment increases. For example, the sampling probability function may include a function expression in which the sampling probability increases exponentially as the time difference shortens, for example, as shown in formula (1), as follows:

[0082] P(t)=(1-ε)e k(t-T) / T +ε formula (1)

[0083] Where T represents the current time, and P(t) represents the sampling probability corresponding to time t in the current video stream. ε represents the lower limit of the sampling probability. For example, ε can be 0.1, or other values ​​can be taken according to actual needs. No specific limitation is made here. k in formula (1) is the attenuation parameter, which is used to characterize the rate at which the sampling probability decays with the time difference. k > 0. It can be seen that based on this predefined sampling probability function, the value of k needs to be solved, and then the sampling probability corresponding to each time moment can be calculated according to formula (1).

[0084] In some embodiments, since the sampling probability corresponding to each moment represents the likelihood of being sampled at that moment, integrating the sampling probability corresponding to each moment in the current video stream, for example, integrating the sampling probability P(t) corresponding to each moment in the current video stream, the resulting integral can represent the expected number of samples for the current video stream. The expected number of samples can be expressed by formula (2), as follows:

[0085]

[0086] In formula (2), E is the integral of the sampling probability P(t) at each moment in the current video stream. frame This represents the expected number of samples. It is understood that the expected number of samples should not exceed the threshold for the number of image frame samples, thus yielding formula (3) as follows:

[0087]

[0088] Thus, by using numerical methods, such as the Brent method, to solve formula (3), the value of the attenuation parameter k under the current T value can be obtained, and then the sampling probability P(t) corresponding to each time t can be calculated by combining formula (1).

[0089] S204. Based on the sampling probability corresponding to each moment, sample the current video stream to determine at least one image frame.

[0090] In step S204, the current video stream is sampled according to the sampling probability at each time moment. For example, if the sampling probability at time t is greater than a preset probability threshold, image frames corresponding to at least one viewpoint at time t are sampled; if the sampling probability at time t is not greater than the preset probability threshold, image frame sampling at time t is not performed, thereby completing the sampling of the current video stream and determining at least one image frame. Here, at least one image frame corresponds to at least one time moment, and each of the at least one time moment corresponds to an image frame at at least one viewpoint.

[0091] It is understood that the sampling strategy in this application embodiment can ensure that recent frames have a higher sampling probability while ensuring that the total sampling amount does not exceed the budget. This effectively balances long-term historical and current observation data with less computational overhead, which helps to improve the accuracy of subsequent large language model inference and navigation trajectory generation. Furthermore, as the computing power of the large language model increases, the image frame sampling number threshold will also be adjusted accordingly, thereby achieving dynamic sampling number constraints.

[0092] In some embodiments, to address the problem of an excessive number of visual feature tokens due to the large number of video frames, this application embodiment can also generate multi-scale visual features at multi-scale resolution for different task types and provide them to the large language model for inference. Exemplarily, the above-described S101 process can be implemented through the following steps:

[0093] Natural language instructions are subjected to feature extraction to determine semantic features; semantic features are extracted from at least one image frame to determine the image semantic features corresponding to each image frame in the at least one image frame; and according to the task type of the natural language instructions, feature extraction at the target scale is performed on at least one image frame to determine the image features corresponding to each image frame; the image semantic features and image features are combined to determine the initial visual features corresponding to each image frame; the initial visual features corresponding to each image frame are mapped to the latent space corresponding to the large language to determine the visual features corresponding to each image frame, thereby determining at least one visual feature.

[0094] Specifically, semantic features can be extracted from at least one image frame to determine the image semantic features corresponding to each image frame in the at least one image frame; and, according to the task type of the natural language instruction, features at different target scales can be extracted from at least one image frame to determine the image features corresponding to each image frame.

[0095] For example, the target scale for feature extraction of each image frame can be determined according to the task type of the natural language instruction. Then, according to the target scale, grid pooling is used to extract features from each image frame to obtain the image features corresponding to each image frame.

[0096] For example, a unified visual representation can be obtained by concatenating image semantic features and image features, serving as the initial visual feature for each image frame. The initial visual feature V for each image frame can be represented as R. P×C , where P represents the number of image patches into which the image frame is divided according to the target scale, and C represents the embedding dimension.

[0097] In some embodiments, a cross-modal mapping module can be used to map the initial visual features corresponding to each image frame to a latent space that matches a large language model, and the resulting visual embedding vector can be used as the visual feature corresponding to each image frame, thereby obtaining at least one visual feature. Exemplarily, the cross-modal mapping module can be implemented using a two-layer fully connected network MLP, and the embodiments of this application do not impose specific limitations on its network structure and parameters.

[0098] Understandably, by combining semantic features with image features to obtain the initial visual features corresponding to each image frame, the performance of vision-centric tasks can be significantly improved. Furthermore, determining the target scale for image feature extraction based on the task type allows for the provision of image features of the required granularity to large language models according to different task types, achieving a balance between model processing performance and efficiency, and thus improving model processing efficiency.

[0099] In some embodiments, when the task type includes a navigation task, at least one image frame includes: a historical image frame corresponding to at least one viewpoint at each historical moment in at least one historical moment, and a current image frame corresponding to at least one viewpoint at the current moment. In this case, the target scale corresponding to the current image frame can be determined as a first scale, and the target scale corresponding to the historical image frames can be determined as a second scale; wherein the second scale is larger than the first scale. Based on the first scale, feature extraction is performed on each current image frame to determine the image features corresponding to each current image frame; based on the second scale, feature extraction is performed on each historical image frame to determine the image features corresponding to each historical image frame.

[0100] In some embodiments, when the task type includes image question-answering tasks, the aforementioned at least one image frame is the target image frame corresponding to the natural language instruction. That is, the image question-answering task is to perform knowledge question answering on a single-view image frame pointed to by the natural language instruction. In this case, the target scale is determined as a first scale, and feature extraction is performed on the current image frame according to the first scale to determine the image features corresponding to the current image frame.

[0101] In some embodiments, video question-answering tasks are typically used to answer questions about a segment of video. When the task type includes video question-answering, at least one image frame may include at least one image frame from the video corresponding to the natural language instruction. Video question-answering tasks do not have high requirements for feature resolution; therefore, a second scale can be determined as the target scale. Based on the second scale, feature extraction is performed on each image frame in the at least one image frame in the video pointed to by the natural language instruction to determine the image features corresponding to each image frame.

[0102] In other words, for the current image frame in navigation tasks and the image frame in image question-answering tasks, a first-scale fine-grained feature extraction is performed; for historical image frames in navigation tasks and the image frame in video question-answering tasks, a second-scale coarse-grained feature extraction is performed. This fine-grained feature extraction allows the large language model to perform more refined understanding and analysis based on the latest navigation observation data or image content, ensuring the accuracy of navigation trajectory generation or image question answering. The coarse-grained feature extraction reduces the feature length or number of words in the large language model, lowering the model's processing burden and accelerating its processing speed.

[0103] In some embodiments, the process of determining at least one indicator based on at least one image frame in S102 above can be as follows: Figure 3 As shown, this is achieved through processes S301-S303, as follows:

[0104] S301. By encoding the viewpoint of the image frame, determine the angle embedding corresponding to each image frame in at least one image frame.

[0105] In this embodiment, the indicator may include an angle embedding for characterizing the viewpoint of an image frame. When generating the angle embedding for an image frame, the azimuth continuity must be maintained to ensure that the distance metric between embeddings reflects geometric proximity. In some embodiments, the viewpoint of each image frame can be converted into an embedding vector in a continuous annular space based on cosine or sine position coding to obtain the angle embedding for each image frame.

[0106] S302. Encode the image frame using its time information to determine the time embedding for each image frame.

[0107] In this embodiment, the indicator may include a temporal embedding for representing the temporal information of an image frame. When generating the temporal embedding for an image frame, it is necessary to ensure that the temporal embedding uniquely identifies the temporal order of image frames across all viewpoints, while maintaining robustness to irregular sampling intervals. In some embodiments, the temporal information of the image frame, such as the frame number, can be encoded based on a sinusoidal positional encoding method, converting it into a temporal vector to obtain the temporal embedding for each image frame. In this way, by encoding the temporal information of the image frame, different forms of temporal information from different hardware platforms can be normalized and processed into a unified representation that is platform-independent, easily understood and utilized by the model, thus obtaining the temporal embedding.

[0108] S303. Based on the baseline embedding obtained from pre-training, and combined with at least one of the angle embedding and the time embedding, generate an indicator corresponding to each image frame, thereby determining at least one indicator.

[0109] In this embodiment, the indicator also needs to meet the requirement of separability. That is, the indicator may encode perspective or temporal information (e.g., for video question-answering and navigation tasks), or it may completely exclude such information (e.g., for image question-answering tasks). To meet this requirement, this application designs a baseline embedding, and the indicator includes at least the baseline embedding. The baseline embedding is obtained through synchronous pre-training with a large language model, and is used to enable the large language model to recognize the input indicator. Exemplarily, an indicator containing at least the baseline embedding can be introduced into the initial large language model. An angular embedding and / or a temporal embedding are then superimposed on the baseline embedding for model training. This allows the large language model to recognize the indicative role of the baseline embedding during the training process and accurately identify the indicator based on the baseline embedding, so as to use the angular embedding and / or the temporal embedding therein for reasoning. This is equivalent to using the baseline embedding as the identifier of the indicator and passing the angular embedding and / or the temporal embedding to the large language model.

[0110] In some embodiments, based on the task type of the natural language instruction, at least one of angular embedding and temporal embedding can be combined with the baseline embedding to generate an indicator corresponding to each image frame, thereby determining at least one indicator.

[0111] For example, in the case of a task type including a single-view video question-and-answer task, an indicator corresponding to each image frame is generated based on the baseline embedding and the temporal embedding.

[0112] For example, in the case of a task type including multi-view video question answering tasks, an indicator corresponding to each image frame is generated based on the baseline embedding, angle embedding, and temporal embedding. For example, the indicator corresponding to each image frame can be generated by superimposing the angle embedding, baseline embedding, and temporal embedding after linear projection mapping.

[0113] For example, when the task type includes navigation tasks, an indicator corresponding to each image frame is generated based on the baseline embedding, angle embedding, and time embedding.

[0114] For example, in the case where the task type includes image question answering tasks, an indicator corresponding to each image frame is generated based on the baseline embedding.

[0115] In other words, indicators for navigation tasks can include angle embedding, temporal embedding, and reference embedding; indicators for single-view video question-answering tasks can include temporal embedding and reference embedding; indicators for multi-view video question-answering tasks can include angle embedding, temporal embedding, and reference embedding; and indicators for image question-answering tasks only include reference embedding. This achieves explicit encoding of the spatial location and temporal order of observation data, and allows for flexible combination of the encoded results to obtain corresponding indicators to guide model inference, improving the accuracy of model inference and its generalization ability across different hardware platforms, according to the needs of different task types.

[0116] Based on the natural language instruction processing method of this application, this application provides an intelligent model applicable to real-world scenarios, such as... Figure 4 As shown, the system includes a visual encoder, a pooling module, a cross-modal mapping module, a large language model, and a dual-branch execution network (including a navigation head network and a question-answering head network). The input to the intelligent model includes natural language instructions and a sequence of image frames acquired by multiple cameras over time (equivalent to at least one image frame as described above). Each image frame is in W×H×3 RGB format, where W represents the image width, H represents the image height, and 3 represents the number of color channels.

[0117] Each image frame in the image frame sequence is processed by a pre-trained visual encoder to extract fine-grained geometric and semantic features. The fine-grained geometric features of the image frames are compressed using a pooling module (equivalent to first- or second-scale feature extraction) to obtain the image features corresponding to each image frame. These image features are concatenated with the semantic features to form a unified visual representation (equivalent to the initial visual features). This unified visual representation is then mapped to a latent space matching the large language model via a cross-modal mapping module, resulting in a visual embedding (equivalent to the aforementioned visual features). Natural language instructions are embedded into language feature vectors (equivalent to instruction semantic features) through the large language model.

[0118] Furthermore, the visual encoder encodes the viewpoint and temporal information for each image frame, and generates an indicator for each image frame based on the baseline embedding, according to different task types. For example, the indicator can be in token form, allowing the token-based language feature vector to be combined with the visual embedding and input into a large language model. Different tasks correspond to different token organization methods: the indicator for image question answering includes the baseline embedding, and the visual embedding for image question answering is a fine-grained visual embedding; the indicator for video question answering includes the temporal embedding (…). Figure 4 In the context of video question answering tasks (represented by 0 to T), the visual embedding is coarse-grained; the indicators for navigation tasks include baseline embedding, temporal embedding, and angular embedding. Figure 4 In this model, 0, 1 to T represent the temporal embeddings of each image frame, and 0 to θ represent the angular embeddings of each image frame. The visual embeddings for navigation tasks include coarse-grained historical image frame visual embeddings and fine-grained current image frame visual embeddings. The large language model infers based on the input language feature vector token and different visual embedding tokens and indicator tokens corresponding to different tasks, predicting the hidden state vectors corresponding to the corresponding tasks. For example, it predicts the navigation hidden state vectors corresponding to navigation tasks, or the question-answering hidden state vectors corresponding to visual question-answering tasks (including image question-answering tasks and video question-answering tasks). Depending on the task type, the large language model inputs the hidden state vectors into the corresponding head network in the dual-branch execution network. For example, the navigation hidden state vectors are input into the navigation head network, which decodes them to generate the navigation trajectory corresponding to the natural language command. The question-answering hidden state vectors are input into the question-answering head network, which decodes them to generate the response text corresponding to the natural language command.

[0119] It is understood that the embodiments of this application generate indicators by dynamically encoding angle and time-series information from multiple camera perspectives, effectively integrating multi-viewpoints and temporal dynamics to improve trajectory prediction accuracy. Furthermore, by unifying task definitions—abstracting navigation tasks as: inputting video and commands, generating hidden state vectors, and generating trajectories—it is compatible with visual language navigation, tracking, and driving tasks, facilitating expansion and application. Moreover, through efficient token organization and a dual-branch structure, it achieves high inference speed, reaching nearly 10Hz in actual tests, meeting the real-time navigation needs of robots and autonomous driving.

[0120] The aforementioned embodied intelligence model architecture can be widely applied to navigation scenarios across various hardware platforms and task modes, as exemplified below:

[0121] I. Indoor Navigation: The intelligent model of this application embodiment is deployed on a service robot equipped with multiple cameras. Inputting multi-view video of the indoor environment and voice commands (such as "go to the sofa in the living room"), the intelligent model outputs a navigation trajectory. In this scenario, there is no need to retrain for different room types and camera configurations; the robot can complete navigation in unfamiliar home or office environments, significantly reducing deployment costs and improving task completion rates.

[0122] II. Outdoor Navigation and Following: The intelligent model of this application embodiment is deployed on a humanoid robot or quadruped robot, and autonomous following is achieved by combining natural language commands (such as "follow me to the parking lot") and utilizing multi-view video perception and trajectory planning. In this scenario, the intelligent model can process dynamic environments (crowds, obstacles) in real time, maintain stable following and obstacle avoidance capabilities in complex outdoor scenes, and exhibit strong generalization ability.

[0123] III. Autonomous Driving: By integrating multiple camera video inputs and voice navigation commands (such as "Take me to the nearest supermarket") into the intelligent vehicle, the intelligent model in this embodiment directly outputs an executable driving trajectory. In this scenario, compared to traditional end-to-end driving models, the intelligent model in this embodiment possesses stronger multi-view fusion and command understanding capabilities, reducing reliance on high-precision maps and improving adaptability in unknown urban roads.

[0124] IV. Unmanned Aerial Vehicle (UAV) Inspection and Search and Rescue: A model is deployed on a UAV, utilizing multi-directional camera input and voice / text commands (such as "search for the red box") to output an aerial flight path through the intelligent model in this application embodiment. In this scenario, it can quickly adapt to complex disaster sites or industrial inspection tasks, reducing human intervention and achieving autonomous target search and path planning.

[0125] V. Warehousing and Logistics Robots: Combining multi-view cameras and voice commands (such as "deliver the goods to shelf number 3"), the robot utilizes the intelligent model of this application embodiment to automatically generate obstacle avoidance paths and complete the handling task. In this scenario, flexibility in the warehousing environment can be improved, the need for manual programming and task fine-tuning can be reduced, and the level of automation can be increased.

[0126] VI. Smart Homes and Elderly / Disabled Assistance: In home service robots, through natural language interaction (such as "Take me to the kitchen"), the intelligent model of this application's embodiments generates safe paths based on multi-camera input. In this scenario, intelligent assistance can be provided to the elderly and people with mobility impairments, improving the convenience and safety of family life.

[0127] This application provides a natural language instruction processing device 1, such as... Figure 5 As shown, it includes:

[0128] Feature processing module 11 is used to perform feature processing on natural language instructions and at least one image frame to determine semantic features of the instructions and at least one visual feature; the at least one image frame includes image frames acquired from at least one viewpoint at at least one moment; the task type of the natural language instructions includes navigation tasks;

[0129] The indicator generation module 12 is used to determine at least one indicator based on the at least one image frame; the indicator is used to identify at least one of the viewpoint and time information of the image frame;

[0130] The reasoning module 13 is used to determine the navigation trajectory corresponding to the natural language instruction by reasoning based on the semantic features of the instruction, the at least one visual feature and the at least one indicator through a large language model.

[0131] In some embodiments, the reasoning module 13 is further configured to determine the hidden state vector by reasoning based on the instruction semantic features, the at least one visual feature and the at least one indicator through the large language model; and to decode the hidden state vector through the action decoding model to determine the navigation trajectory corresponding to the natural language instruction of the navigation task.

[0132] In some embodiments, the task type of the natural language instruction further includes a visual question-answering task, and the natural language instruction processing device 1 further includes a decoding module. The decoding module is used to decode the hidden state vector output by the large language model through a language decoding model to determine the target text sequence; and generate the response text corresponding to the natural language instruction based on the target text sequence.

[0133] In some embodiments, the natural language instruction processing device 1 further includes an acquisition module, which is configured to acquire a current video stream; the current video stream includes image frames from at least one historical moment to the current moment corresponding to at least one viewpoint; determine an image frame sampling number threshold based on a preset total feature length threshold and a preset instruction semantic feature length threshold; determine a sampling probability corresponding to each moment based on the image frame sampling number threshold and the time difference between each moment and the current moment; the sampling probability is inversely proportional to the time difference; and sample the current video stream based on the sampling probability corresponding to each moment to determine the at least one image frame.

[0134] In some embodiments, the acquisition module is further configured to determine an attenuation parameter in the sampling probability function based on the image frame sampling quantity threshold; the sampling probability function is used to determine the sampling probability corresponding to each moment in the current video stream; the sampling probability corresponding to each moment attenuates as the time difference between that moment and the current moment increases; the attenuation parameter characterizes the rate at which the sampling probability attenuates with the time difference; and the sampling probability corresponding to each moment is determined according to the sampling probability function, the attenuation parameter, and the time difference between each moment and the current moment.

[0135] In some embodiments, the feature processing module 11 is further configured to: extract features from the natural language instruction to determine the instruction semantic features; extract semantic features from the at least one image frame to determine the image semantic features corresponding to each image frame in the at least one image frame; extract features at the target scale from the at least one image frame according to the task type of the natural language instruction to determine the image features corresponding to each image frame; combine the image semantic features and the image features to determine the initial visual features corresponding to each image frame; map the initial visual features corresponding to each image frame to the latent space corresponding to the large language model to determine the visual features corresponding to each image frame, thereby determining the at least one visual feature.

[0136] In some embodiments, the target scale includes: a first scale and a second scale; the at least one image frame includes: a historical image frame corresponding to at least one historical moment from at least one perspective and a current image frame corresponding to at least one current perspective from the current moment; the feature processing module 11 is further configured to, when the task type includes a navigation task, extract features from each of the at least one current image frames according to the first scale to determine the image features corresponding to each current image frame; extract features from the at least one historical image frame according to the second scale to determine the image features corresponding to each historical image frame; the second scale is greater than the first scale.

[0137] In some embodiments, the target scale includes a first scale; the at least one image frame includes a target image frame corresponding to the natural language instruction; the feature processing module 11 is further configured to, when the task type includes an image question-and-answer task, extract features from the target image frame according to the first scale to determine the image features corresponding to the target image frame.

[0138] In some embodiments, the target scale includes a second scale; the feature processing module 11 is further configured to, when the task type includes a video question-and-answer task, extract features from each of the at least one image frame according to the second scale to determine the image features corresponding to each image frame.

[0139] In some embodiments, the indicator generation module 12 is further configured to: determine the angle embedding corresponding to each image frame in the at least one image frame by encoding the viewpoint of the image frame; determine the time embedding corresponding to each image frame by encoding the time information of the image frame; and generate an indicator corresponding to each image frame by combining at least one of the angle embedding and the time embedding based on the pre-trained benchmark embedding, thereby determining the at least one indicator; the benchmark embedding is used to enable the large language model to recognize the input indicator through pre-training.

[0140] In some embodiments, the indicator generation module 12 is further configured to: generate an indicator corresponding to each image frame based on the baseline embedding and the temporal embedding when the task type includes a single-view video question-and-answer task; generate an indicator corresponding to each image frame based on the baseline embedding, the angle embedding, and the temporal embedding when the task type includes a multi-view video question-and-answer task; and generate an indicator corresponding to each image frame based on the baseline embedding, the angle embedding, and the temporal embedding when the task type includes a navigation task.

[0141] In some embodiments, the indicator generation module 12 is further configured to generate an indicator corresponding to each image frame based on the benchmark embedding when the task type includes an image question-and-answer task.

[0142] It should be noted that the description of the above device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects. For technical details not disclosed in the device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0143] This application also provides an electronic device. Figure 6This is a schematic diagram of an optional structure of an electronic device provided in an embodiment of this application. For example... Figure 6 As shown, the electronic device 3 includes a memory 32 and a processor 33. The memory 32 and the processor 33 are connected via a communication bus 34. The memory 32 is used to store executable instructions. The processor 33 is used to implement the natural language instruction processing method provided in this application embodiment when executing the executable instructions stored in the memory 32.

[0144] This application provides a computer-readable storage medium storing executable instructions. When the executable instructions are executed by the processor, the processor will execute the natural language instruction processing method provided in this application.

[0145] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0146] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0147] As an example, executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts within a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files storing one or more modules, subroutines, or code sections). As an example, executable instructions may be deployed to execute on a single computing device, or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.

[0148] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0149] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0150] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0151] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0152] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A natural language instruction processing method, characterized in that, include: Perform feature processing on natural language instructions and at least one image frame to determine semantic features of the instructions and at least one visual feature; The at least one image frame includes an image frame acquired from at least one viewpoint at at least one moment; the task type of the natural language instruction includes navigation tasks; Based on the at least one image frame, at least one indicator is determined; Indicators are used to identify at least one of the viewpoint and time information of an image frame; By using a large language model, reasoning is performed based on the semantic features of the instruction, the at least one visual feature, and the at least one indicator to determine the navigation trajectory corresponding to the natural language instruction.

2. The method according to claim 1, characterized in that, The step of determining the navigation trajectory corresponding to the natural language instruction by reasoning based on the semantic features of the instruction, the at least one visual feature, and the at least one indicator using a large language model includes: Using the large language model, reasoning is performed based on the instruction semantic features, the at least one visual feature, and the at least one indicator to determine the hidden state vector; By decoding the hidden state vector using an action decoding model, the navigation trajectory corresponding to the natural language command of the navigation task can be determined. The natural language instruction task type also includes visual question answering tasks, and the method further includes: The target text sequence is determined by decoding the hidden state vector output by the large language model using a language decoding model. Based on the target text sequence, generate the response text corresponding to the natural language instruction.

3. The method according to claim 1 or 2, characterized in that, The method further includes; Acquire the current video stream; the current video stream includes image frames from at least one viewpoint corresponding to each moment from at least one historical moment to the current moment; The threshold for the number of image frame samples is determined based on the preset total feature length threshold and the preset instruction semantic feature length threshold. The sampling probability corresponding to each moment is determined based on the image frame sampling number threshold and the time difference between each moment and the current moment; the sampling probability is inversely proportional to the time difference. Based on the sampling probability corresponding to each moment, the current video stream is sampled to determine the at least one image frame.

4. The method according to any one of claims 1 to 3, characterized in that, The step of performing feature processing on natural language instructions and at least one image frame to determine semantic features of the instructions and at least one visual feature includes: Feature extraction is performed on the natural language instruction to determine its semantic features; Semantic feature extraction is performed on the at least one image frame to determine the image semantic features corresponding to each image frame in the at least one image frame, and feature extraction at the target scale is performed on the at least one image frame according to the task type of the natural language instruction to determine the image features corresponding to each image frame; By combining the image semantic features with the image features, the initial visual features corresponding to each image frame are determined; The initial visual features corresponding to each image frame are mapped to the latent space corresponding to the large language model to determine the visual features corresponding to each image frame, thereby determining the at least one visual feature.

5. The method according to claim 4, characterized in that, The target scale includes at least one of a first scale and a second scale; based on the task type of the natural language instruction, feature extraction is performed on the at least one image frame at the target scale to determine the image features corresponding to each image frame in the at least one image frame, including: When the task type includes navigation tasks, feature extraction is performed on each of the at least one current image frames according to the first scale to determine the image features corresponding to each current image frame; the at least one current image frame includes at least one current image frame with a current viewpoint corresponding to the current time in the at least one image frame; Based on the second scale, feature extraction is performed on the at least one historical image frame to determine the image features corresponding to each historical image frame; the second scale is greater than the first scale; the at least one historical image frame includes at least one historical image frame from at least one viewpoint corresponding to each historical moment in the at least one image frame. When the task type includes image question answering tasks, feature extraction is performed on the target image frame corresponding to the natural language instruction according to the first scale to determine the image features corresponding to the target image frame; When the task type includes video question-and-answer tasks, feature extraction is performed on each of the at least one image frame according to the second scale to determine the image features corresponding to each image frame.

6. The method according to any one of claims 1 to 5, characterized in that, Determining at least one indicator based on the at least one image frame includes: By encoding the viewpoint of the image frame, the angle embedding corresponding to each image frame in the at least one image frame is determined; Encode the time information of the image frames to determine the time embedding corresponding to each image frame; Based on the baseline embedding obtained from pre-training, and combined with at least one of the angle embedding and the temporal embedding, an indicator corresponding to each image frame is generated, thereby determining the at least one indicator; the baseline embedding is used to enable the large language model to recognize the input indicator through pre-training.

7. The method according to claim 6, characterized in that, The step of generating an indicator for each image frame by combining at least one of the angle embedding and the temporal embedding based on the pre-trained baseline embedding includes: When the task type includes a single-view video question-and-answer task, an indicator corresponding to each image frame is generated based on the baseline embedding and the temporal embedding. When the task type includes multi-view video question-and-answer tasks, an indicator corresponding to each image frame is generated based on the baseline embedding, the angle embedding, and the time embedding. When the task type includes navigation tasks, an indicator corresponding to each image frame is generated based on the baseline embedding, the angle embedding, and the time embedding. The method further includes: When the task type includes image question-answering tasks, an indicator corresponding to each image frame is generated based on the baseline embedding.

8. A natural language instruction processing device, characterized in that, The device includes: A feature processing module is used to perform feature processing on natural language instructions and at least one image frame to determine semantic features of the instructions and at least one visual feature; the at least one image frame includes image frames acquired from at least one viewpoint at at least one moment; the task type of the natural language instructions includes navigation tasks. An indicator generation module is used to determine at least one indicator based on the at least one image frame; the indicator is used to identify at least one of the viewpoint and time information of the image frame; The reasoning module is used to determine the navigation trajectory corresponding to the natural language instruction by reasoning based on the semantic features of the instruction, the at least one visual feature, and the at least one indicator through a large language model.

9. An electronic device, characterized in that, The electronic device includes: Memory is used to store executable instructions or computer programs. A processor, when executing computer-executable instructions or computer programs stored in the memory, implements the method of any one of claims 1 to 7.

10. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, the method of any one of claims 1 to 7 is implemented.

Images