Modular urban embodied intelligent agent reasoning system and method, device and medium

Through the modular urban embodied intelligent body inference system, the combination of multiple functional modules is used to solve the problem of adaptability and expansion of embodied intelligent body in complex urban environments, and the advanced functions and multiple functions of embodied intelligent body are realized. Modal reasoning ability meets the needs of complex scenarios.

CN118798365BActive Publication Date: 2024-11-26TSINGHUA UNIVERSITY +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411245261.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-06
Publication Date
2024-11-26
Estimated Expiration
2044-09-06

AI Technical Summary

Technical Problem

The existing embodied intelligent technology is difficult to adapt to complex urban environments, cannot meet the needs of diversity and high complexity scenarios, and lacks scalability and versatility.

Method used

A modular urban embodied intelligent inference system is proposed. Through the combination of the intelligent image module, perception module, memory storage module, reasoning and analysis module and behavior module, it uses the database, urban environment base platform and model warehouse to realize the materialization of the database. Personalized settings and complex problem solving of body intelligence.

Benefits of technology

It provides higher degrees of freedom and operability, can handle various complex problems in urban environments, supports user customization, and realizes advanced functions and multimodal reasoning capabilities of embodied agents.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118798365B_ABST
    Figure CN118798365B_ABST
Patent Text Reader

Abstract

The present disclosure relates to the field of artificial intelligence technology. In particular, it relates to a modular urban embodied intelligent body reasoning system and method, device and medium, the system includes an intelligent body portrait module that obtains information from a database and then forms an intelligent body portrait; a perception module obtains perception data from an urban environment base platform and processes it to form a perception data set, and the perception data includes behavior information and environmental data; a memory storage module stores the perception data set and information of the intelligent body portrait; the reasoning and analysis module calls the target model in the model warehouse according to the target information obtained from the memory storage module to perform reasoning to obtain the reasoning decision result including the behavior information; the behavior module performs structured processing on the reasoning decision result and sends it to the urban environment base platform. Based on the modular solution, the embodied intelligent body is abstracted into multiple functional modules, which can be customized, with higher degrees of freedom and operability, ensuring that it has the ability to handle complex problems in various scenarios in the urban environment.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to a modular urban embodied intelligent agent reasoning system and method, device and medium. Background Art

[0002] Embodied agents refer to intelligent systems with perception, reasoning and action capabilities, which can interact with the surrounding environment in physical or virtual environments. For embodied agents in urban environments, they need to be able to simulate the behavior of urban residents, understand and respond to the surrounding environment, perform tasks, make decisions, and communicate with human users or other agents. However, the complexity of the urban environment and the diversity of the construction of embodied agents require that their supporting frameworks be able to adapt to various scenarios in urban scenarios and have high degrees of freedom and scalability, but the solutions in related technologies cannot meet these requirements. Summary of the invention

[0003] In view of this, the present disclosure proposes a modular urban embodied intelligent agent reasoning system and method, device and medium.

[0004] According to one aspect of the present disclosure, a modular urban embodied agent reasoning system is provided, the system comprising:

[0005] An agent portrait module, configured to obtain basic information and prior knowledge information corresponding to the embodied agent from a database in response to a configuration operation, and to form an agent portrait based on the basic information and prior knowledge information;

[0006] A perception module, used to obtain the perception data of the embodied intelligent body in the current space from the urban environment base platform, and process the perception data to form a perception data set that meets the format requirements, wherein the perception data includes the behavior information of the embodied intelligent body in the urban environment and the environmental data of the current space;

[0007] A memory storage module, used to store the perception data set and the information of the agent portrait;

[0008] A reasoning and analysis module, used to obtain target information required for reasoning and analysis from the memory storage module, and call the matching target model in the model warehouse to perform reasoning and analysis according to the target information, so as to obtain a reasoning and decision result corresponding to the target information, wherein the reasoning and decision result includes behavior information describing the behavior of the embodied intelligent body in the space in which it is located;

[0009] The behavior module is used to perform structured processing on the inference and decision results and send them to the urban environment base platform, so that the urban environment base platform forms perception data storage based on the processed inference and decision results and then continues to be acquired by the perception module to realize circular reasoning analysis on the behavior of the embodied intelligent body.

[0010] In a possible implementation, the agent portrait module includes:

[0011] A data loading and processing submodule, for acquiring basic information and prior knowledge information corresponding to the embodied intelligent body from a database in response to a configuration operation, and performing corresponding processing on the basic information and the prior knowledge information to obtain processed basic information and processed prior knowledge information;

[0012] A basic information submodule, used to configure and load basic information in the agent portrait based on the processed basic information;

[0013] The knowledge and experience submodule is used to load the configuration of knowledge and experience in the intelligent agent portrait based on the processed prior knowledge information.

[0014] In a possible implementation, the perception module includes:

[0015] A data perception submodule, used to obtain the perception data of the embodied intelligent body in the current space through the perception interface provided by the urban environment base platform;

[0016] A perception data processing submodule, used to process the perception data to form a perception data set that meets format requirements;

[0017] The memory docking submodule is used to send the perception data set to the memory storage module.

[0018] In a possible implementation, the memory storage module includes:

[0019] A memory manager, for managing a working memory storage unit and at least one long-term memory storage unit,

[0020] The working memory storage unit is used to store working memory, and the working memory includes at least one of the following: each data in the perception data set, part or all of the received information of the agent portrait, and the index of the long-term memory stored in the long-term memory storage unit;

[0021] The long-term memory storage unit is used to store long-term memory, and the long-term memory includes at least one of the following: part or all of the information of the received agent portrait, and the long-term memory to be stored sent by the working memory storage unit;

[0022] The memory control submodule is used to configure part or all of the storage information in the at least one long-term memory storage unit in response to the user's setting operation, and to store the memory-related information input by the user through the memory loading interface into the corresponding working memory storage unit and / or long-term memory storage unit, and to send the working memory and all the long-term memories to the corresponding memory bank through the memory storage interface for storage when the storage conditions are met.

[0023] In a possible implementation, the reasoning and analysis module includes:

[0024] A reasoning analysis control submodule, used for acquiring target information required for reasoning analysis from the memory storage module;

[0025] A reasoning and analysis manager, configured to analyze the target information using a plurality of reasoning and analysis units to determine at least one target model to be called, and to generate a reasoning request for each target model based on the target information, wherein each reasoning and analysis unit has a different analysis function;

[0026] The reasoning analysis control submodule is further used to send each of the reasoning requests to the model warehouse through the model calling interface, so that the model warehouse sends each of the reasoning requests to the corresponding target model and returns the reasoning results obtained after each of the target models executes the corresponding reasoning request;

[0027] The reasoning analysis control submodule is further used to receive the reasoning results returned by the model warehouse through the model result interface, and send the reasoning decision results corresponding to the target information formed according to the reasoning results to the behavior module.

[0028] In a possible implementation, the analysis functions of each of the reasoning and analysis units include any one of the following: itinerary planning, dialogue generation, environment understanding, shopping selection, and target generation; the target model includes at least one of the following: a large language base model, an image generation model, and a speech generation model;

[0029] The reasoning and analysis control submodule is further used to send the target information to each of the reasoning and analysis units in descending order of priority when each of the reasoning and analysis units is set with a corresponding priority;

[0030] Each of the reasoning and analysis units is used to analyze the target information when it receives the target information, and when it is determined after the analysis that the target model needs to be called, generate a reasoning request for the target model based on the target information and send it to the reasoning analysis control submodule.

[0031] In a possible implementation, the behavior module includes:

[0032] A reasoning and analysis docking submodule, used for receiving the reasoning decision result from the reasoning and analysis module;

[0033] A data structuring submodule, used for performing structural processing on the inference and decision result according to the data structure requirements of the urban environment base platform to obtain a processed inference and decision result;

[0034] The behavior information synchronization submodule is used to send the processed reasoning decision results to the urban environment base platform, so that the urban environment base platform forms and stores perception data based on the processed reasoning decision results.

[0035] In a possible implementation, the system further includes at least one of the database, the urban environment base platform, and the model warehouse;

[0036] Wherein, the database is used to store various pre-set basic information and prior knowledge information;

[0037] The urban environment base platform is used to determine the behavior information of the corresponding embodied intelligent body according to the received reasoning decision result, and determine the environmental data of the space where the corresponding embodied intelligent body is located based on the behavior information;

[0038] The model warehouse is used to store multiple available models, and when receiving an inference request, sends the inference request to a target model among the multiple available models that can execute the inference request, so that the target model that receives the inference request executes the inference request and returns an inference result;

[0039] The database also includes the urban environment base platform.

[0040] According to another aspect of the present disclosure, a modular urban embodied agent reasoning method is provided, which is applied to a modular urban embodied agent reasoning system, and the method includes:

[0041] Acquire the perception data of the embodied intelligent body in the current space from the urban environment base platform, process the perception data to form a perception data set that meets the format requirements, and store it in the memory storage module, wherein the perception data includes the behavior information of the embodied intelligent body in the urban environment and the environmental data of the current space;

[0042] Acquire target information required for reasoning analysis from the memory storage module, and call the matching target model in the model warehouse to perform reasoning analysis according to the target information, so as to obtain a reasoning decision result corresponding to the target information, wherein the reasoning decision result includes behavior information describing the behavior of the embodied intelligent body in the space where it is located;

[0043] The reasoning decision result is sent to the urban environment base platform after structured processing, so that the urban environment base platform forms a perception data storage based on the processed reasoning decision result and then continues to be acquired to realize a circular reasoning analysis on the behavior of the embodied intelligent body;

[0044] Among them, the memory storage module stores the perception data set and stores information of the intelligent agent portrait, and the intelligent agent portrait is formed based on basic information and prior knowledge information corresponding to the embodied intelligent agent obtained from the database in response to the configuration operation.

[0045] In a possible implementation, the perception data of the embodied intelligent body in the current space is obtained from the urban environment base platform, and the perception data is processed to form a perception data set that meets the format requirements and stored in the memory storage module, including:

[0046] Acquiring the perception data of the embodied intelligent body in the current space through the perception interface provided by the urban environment base platform;

[0047] Processing the perception data to form a perception data set that meets format requirements;

[0048] The sensing data set is sent to the memory storage module.

[0049] In a possible implementation, the memory storage module includes a memory manager, which is used to manage a working memory storage unit and at least one long-term memory storage unit.

[0050] The working memory storage unit is used to store working memory, and the working memory includes at least one of the following: each data in the perception data set, part or all of the received information of the agent portrait, and the index of the long-term memory stored in the long-term memory storage unit;

[0051] The long-term memory storage unit is used to store long-term memory, and the long-term memory includes at least one of the following: part or all of the information of the received agent portrait, and the long-term memory to be stored sent by the working memory storage unit;

[0052] The memory control submodule is used to configure part or all of the storage information in the at least one long-term memory storage unit in response to the user's setting operation, and to store the memory-related information input by the user through the memory loading interface into the corresponding working memory storage unit and / or long-term memory storage unit, and to send the working memory and all the long-term memories to the corresponding memory bank through the memory storage interface for storage when the storage conditions are met.

[0053] In a possible implementation, target information required for reasoning analysis is obtained from the memory storage module, and a matching target model in the model warehouse is called to perform reasoning analysis according to the target information to obtain a reasoning decision result corresponding to the target information, including:

[0054] Acquire target information required for reasoning analysis from the memory storage module;

[0055] Analyzing the target information using multiple reasoning and analysis units to determine at least one target model that needs to be called, and generating reasoning requests for each target model based on the target information, wherein each reasoning and analysis unit has a different analysis function;

[0056] Sending each of the inference requests to the model warehouse through a model calling interface, so that the model warehouse sends each of the inference requests to a corresponding target model and returns an inference result obtained after each of the target models executes the corresponding inference request;

[0057] The inference results returned by the model warehouse are received through the model result interface, and the inference decision results corresponding to the target information are formed according to the inference results.

[0058] In a possible implementation, the analysis functions of each of the reasoning and analysis units include any one of the following: itinerary planning, dialogue generation, environment understanding, shopping selection, and target generation; the target model includes at least one of the following: a large language base model, an image generation model, and a speech generation model; the method further includes:

[0059] In the case where each of the reasoning and analysis units is set with a corresponding priority, the target information is sent to each of the reasoning and analysis units in order from high to low priority;

[0060] When each of the reasoning and analysis units receives the target information, it analyzes the target information, and when it is determined after the analysis that a target model needs to be called, it generates a reasoning request for the target model based on the target information.

[0061] In a possible implementation, the reasoning decision result is structured and then sent to the urban environment base platform, including:

[0062] Receiving the inference decision result;

[0063] Performing structural processing on the inference and decision result according to the data structure requirements of the urban environment base platform to obtain a processed inference and decision result;

[0064] The processed inference decision result is sent to the urban environment base platform.

[0065] According to another aspect of the present disclosure, a modular urban embodied intelligent agent reasoning device is provided, comprising: a processor; a memory for storing processor executable instructions; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0066] According to another aspect of the present disclosure, a non-volatile computer-readable storage medium is provided, on which computer program instructions are stored, wherein the computer program instructions implement the above method when executed by a processor.

[0067] According to another aspect of the present disclosure, a computer program product is provided, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0068] Through the modular urban embodied intelligent body reasoning system and method, device and medium provided by the embodiment of the present disclosure, the modular urban embodied intelligent body, based on the modular solution, abstracts the embodied intelligent body in the urban environment into an aggregate of multiple functional modules, while supporting user customization, providing higher degrees of freedom and operability, and ensuring that the embodied intelligent body has the ability to handle complex problems in various scenarios in the urban environment. The system consists of multiple closely connected functional modules, which work together to realize various advanced functions of the urban embodied intelligent body. In order to meet the functional requirements of the embodied intelligent body, the system is associated with multiple peripheral components including a database, an urban environment base platform and a model warehouse. Among them, the database, as a storage carrier for persistent data, provides basic data support for the intelligent portrait module and the memory storage module, and manages and stores the data assets of the embodied intelligent body; the urban environment base platform defines the interactive environment of the embodied intelligent body, and becomes the source of information and the influence object of behavior when the embodied intelligent body is running by exposing data acquisition and interaction interfaces to the outside; the model warehouse contains one or more basic models, which provide reasoning and analysis capabilities for the embodied intelligent body, and becomes the thinking dependence of the embodied intelligent body when it is running by exposing the reasoning request interface. The system consists of five main modules: agent portrait module, perception module, memory storage module, reasoning and analysis module, and behavior module. The perception module and behavior module are directly associated with the smart city base platform, and are used to obtain perception information and synchronize behavior information respectively. The agent portrait module provides support for personalized urban embodied agents. The memory storage module stores the long-term memory and working memory of the embodied agent, and is the runtime data source of the embodied agent. The long-term memory and working memory are interconnected through memory storage and memory indexing. The reasoning and analysis module is the thinking carrier of the embodied agent. Based on the data in the memory storage module and the reasoning and analysis capabilities provided by the basic models in the model warehouse, it provides support for the embodied agent to solve various complex problems. Moreover, each module in the system provides a rich extension interface to support the construction of personalized embodied agents.

[0069] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the disclosure and, together with the description, serve to explain the principles of the disclosure.

[0071] Figure 1 A block diagram of a modular urban embodied agent reasoning system according to an embodiment of the present disclosure is shown.

[0072] Figure 2A block diagram of an intelligent agent portrait module according to an embodiment of the present disclosure is shown.

[0073] Figure 3 A block diagram of a perception module and a behavior module according to an embodiment of the present disclosure is shown.

[0074] Figure 4 A block diagram of a memory storage module according to an embodiment of the present disclosure is shown.

[0075] Figure 5 A block diagram of a reasoning and analysis module according to an embodiment of the present disclosure is shown.

[0076] Figure 6 A schematic diagram showing a system construction according to an embodiment of the present disclosure.

[0077] Figure 7 It is a block diagram of a device 1900 for modular urban embodied agent reasoning according to an exemplary embodiment. DETAILED DESCRIPTION

[0078] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0079] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0080] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0081] In related technologies, with the rapid development of large language model technology, embodied intelligent agent construction solutions based on large language model technology have attracted widespread attention due to its human-like decision-making and performance capabilities in a given corpus environment and its ability to provide support for personalized purposes. For example, based on the large language model, a lifelong learning intelligent agent with autonomous exploration, acquisition of diverse skills and new discoveries was built in the Minecraft game. Some solutions discuss how to use large language models to build embodied intelligent agents that can effectively cooperate in decentralized control, primitive sensory observation, expensive communication and multi-objective tasks. There are also solutions that build an end-to-end multimodal basic model Embodied GPT (embodied GPT is a concept that refers to a method or system that combines a generative pre-trained transformer (GPT) model with an embodied agent) with "thinking chain" capabilities, enabling embodied agents to interact with the physical world in a more natural and intuitive way. However, related technologies only consider embodied intelligent agents in simple environments, such as specific room environments or virtual game environments; moreover, they are all designed and constructed specifically for specific simple tasks, and their limited scalability makes them unable to be migrated to complex urban environments and cannot meet the needs of various complex tasks.

[0082] In the related art, there are the following exemplary solutions for embodied intelligent agents:

[0083] Solution 1: A memory system for embodied agents (which can be virtual objects, digital entities, or robots) that is capable of populating memories in real time and supporting writing is described. The embodied agent influences or guides its behavior through experience memory stores that can simulate human memory, allowing mental imagery to be re-experienced or external reality to be simulated during recall. The memory database is generated in a simple, writeable way, allowing experiences to be learned or written during the real-time operation of the embodied agent. An eligibility-based learning mechanism determines which aspects of multimodal information flows are stored in the experience memory store.

[0084] Solution 2: In response to the challenges faced by embodied agents in handling object disambiguation tasks, a method and system are provided to convert object descriptions in natural language into semantic graph representations, thereby providing a clear formal definition of the referred object and its instances. By converting the object described in natural language into a structured semantic graph, it not only helps the embodied agent understand the attributes and relationships of the object, but also supports real-time multi-view aggregation algorithms.

[0085] Solution 3: An innovative method is proposed that can automatically generate corresponding animations and gestures for the input text content. This method aims to make the movements of the virtual agent more natural and humane. To achieve this goal, an algorithm called Text-to-Gesture (TTG) is proposed. Before the virtual agent speaks, the algorithm will conduct an in-depth analysis of the input text, taking into account the meaning of the text, the relevant context, the emotional color, and the gesture style or personality characteristics of the virtual agent. Based on these analysis results, the TTG algorithm will match the text content with appropriate and expressive gesture markers.

[0086] However, the related technologies have the following limitations: embodied agents designed for specific tasks (such as voice interaction tasks, object grasping tasks, etc.) or for specific scenarios and environments (game environments, single isolated environments, etc.) have their own characteristics, are not universal and scalable, and lack solutions for urban environment scenarios. In the absence of guidance for building an embodied agent framework for urban environments, existing solutions cannot adapt to the diversity and complexity of urban scenarios.

[0087] In order to solve the above technical problems, the embodiment of the present disclosure provides a modular urban embodied intelligent body reasoning system and method. The modular urban embodied intelligent body, based on a modular solution, abstracts the embodied intelligent body in the urban environment into an aggregate of multiple functional modules, while supporting user customization, providing higher degrees of freedom and operability, and ensuring that the embodied intelligent body has the ability to handle complex problems in various scenarios in the urban environment. The system consists of multiple closely connected functional modules, which work together to realize various advanced functions of the urban embodied intelligent body. In order to meet the functional requirements of the embodied intelligent body, the system is associated with multiple peripheral components including a database, an urban environment base platform, and a model warehouse. Among them, the database, as a storage carrier for persistent data, provides basic data support for the intelligent portrait module and the memory storage module, and manages and stores the data assets of the embodied intelligent body; the urban environment base platform defines the interactive environment of the embodied intelligent body, and becomes the source of information and the influence object of behavior of the embodied intelligent body at runtime by exposing data acquisition and interaction interfaces to the outside; the model warehouse contains one or more basic models, which provide reasoning and analysis capabilities for the embodied intelligent body, and becomes the thinking dependence of the embodied intelligent body at runtime by exposing the reasoning request interface. The system consists of five main modules: agent portrait module, perception module, memory storage module, reasoning and analysis module, and behavior module. The perception module and behavior module are directly associated with the smart city base platform, and are used to obtain perception information and synchronize behavior information respectively. The agent portrait module provides support for personalized urban embodied agents. The memory storage module stores the long-term memory and working memory of the embodied agent, and is the runtime data source of the embodied agent. The long-term memory and working memory are interconnected through memory storage and memory indexing. The reasoning and analysis module is the thinking carrier of the embodied agent. Based on the data in the memory storage module and the reasoning and analysis capabilities provided by the basic models in the model warehouse, it provides support for the embodied agent to solve various complex problems. Moreover, each module in the system provides a rich extension interface to support the construction of personalized embodied agents.

[0088] like Figure 1 As shown, a modular urban embodied agent reasoning system provided by the embodiment of the present disclosure includes: an agent portrait module 100, a perception module 200, a memory storage module 300, a reasoning and analysis module 400, and a behavior module 500. In some embodiments, Figure 1 As shown, the system may further include at least one of a database 600 , an urban environment base platform 700 , and a model repository 800 . The database 600 may include the urban environment base platform 700 .

[0089] The agent portrait module 100 is used to obtain basic information and prior knowledge information corresponding to the embodied agent from the database 600 in response to the configuration operation, and form an agent portrait based on the basic information and prior knowledge information. The formed agent portrait can be described by the basic information and the knowledge and experience network.

[0090] In this embodiment, the user can set the basic information and prior knowledge information of the embodied intelligent agent to be simulated according to their own needs, so that different embodied intelligent agents can set their different characteristics. The embodied intelligent agent can be used to simulate the ability of different humans to behave in an urban environment, understand and respond to the surrounding environment, perform tasks, make decisions, and communicate with human users or other intelligent agents. For example, the user can set the basic information and prior knowledge information of the embodied intelligent agent through configuration operations to simulate humans with different characteristics such as a 30-year-old factory worker and a 40-year-old teacher.

[0091] In this embodiment, the database 600 is used to store various pre-set basic information and prior knowledge information. The basic information and prior knowledge information in the database 600 may be pre-set templates that can be called and used according to the configuration operation of the user.

[0092] In some embodiments, Figure 2 As shown, the intelligent agent portrait module 100 may include: a data loading and processing submodule, a basic information submodule and a knowledge and experience submodule.

[0093] The data loading and processing submodule is used to obtain basic information and prior knowledge information corresponding to the embodied intelligent body from the database 600 in response to the configuration operation, and to perform corresponding processing on the basic information and the prior knowledge information to obtain processed basic information and processed prior knowledge information.

[0094] In this embodiment, the basic information and prior knowledge information can be transmitted from the database 600 to the data loading and processing submodule in the form of a file or a data set. The configuration operation can be an operation of selecting and setting the basic information and prior knowledge information. The basic information can include information describing the basic situation of the embodied intelligent body, such as age, name, gender, education level, economic level, living habits, etc., and the present disclosure does not limit this. Prior knowledge information can be experience-type and knowledge-type information that the embodied intelligent body can possess, such as social science knowledge, biomedical knowledge, life experience, social experience, etc.

[0095] In this embodiment, the processing of the basic information and the prior knowledge information by the data loading and processing submodule may include format adjustment, information extraction, etc., which is not limited in the present disclosure.

[0096] The basic information submodule is used to configure and load the basic information in the intelligent agent portrait based on the processed basic information.

[0097] In this embodiment, if Figure 2 As shown, the basic information submodule can also be used for user customized settings, including customized data content (such as defining what content is included in the basic information), customized storage mode (such as defining the storage mode of information as JSON storage, YAML storage, etc.), interface definition (such as building interface functions to achieve association with other modules and submodules, data transmission, etc.).

[0098] The knowledge and experience submodule is used to load the configuration of knowledge and experience in the intelligent agent portrait based on the processed prior knowledge information.

[0099] In this embodiment, if Figure 2 As shown, the knowledge and experience submodule can configure different categories of knowledge and experience based on prior knowledge information, and can set the categories of knowledge and experience in response to user settings. For example, the knowledge of an embodied intelligent body can be set to include two categories: social science and biomedicine, and the experience can include two categories: life experience and social experience. In addition, the knowledge and experience submodule can also customize the knowledge network model (i.e., the association model of knowledge data, such as knowledge graphs, knowledge lists, etc.), customize knowledge encoding and indexing (i.e., the expression model of knowledge data, such as plain text, ASCII encoding, word vector encoding, etc.), and update and optimize knowledge (i.e., the method and frequency of knowledge update and optimization, etc.) based on the configured knowledge and experience.

[0100] In some embodiments, at least one of the basic information and the knowledge and experience information can be configured in a spatial topology map, a structure, etc. The configuration method is selected to match the type of information, and the present disclosure does not limit this. For example, the agent portrait module reads a basic information configuration file containing multiple key-value entries, and the output configuration of the basic information can be an agent basic information structure. The agent portrait module reads a file containing multiple spatial association data entries, and the output configuration of the knowledge or experience information can be a spatial topology map.

[0101] In this embodiment, after the agent portrait module completes the configuration of the basic information and the knowledge and experience information, the agent portrait of the embodied agent is obtained, and the agent portrait module can also send the configured basic information and the knowledge and experience information to the memory storage module 300. In some embodiments, the agent portrait module can also send the information related to reasoning and analysis in the configured basic information and the knowledge and experience information to the reasoning and analysis module 400.

[0102] In this way, based on the intelligent agent portrait module, users can realize personalized embodied intelligent agent settings and provide support for the embodied intelligent agent background data (basic information, knowledge and experience) in various application scenarios.

[0103] The perception module 200 is used to obtain the perception data of the embodied intelligent body in the current space from the urban environment base platform 700, and process the perception data to form a perception data set that meets the format requirements. The perception data may include the behavior information of the embodied intelligent body in the urban environment and the environmental data of the current space.

[0104] like Figure 3 As shown, the perception module 200 may include a memory docking submodule, a perception data processing submodule and a data perception submodule.

[0105] The data perception submodule is used to obtain the perception data of the embodied intelligent body in the current space through the perception interface provided by the urban environment base platform 700. The perception data may include the behavior information of the embodied intelligent body in the urban environment and the environmental data of the current space.

[0106] In this embodiment, the urban environment base platform 700 is used to determine the behavior information of the corresponding embodied intelligent body according to the received reasoning decision result, and to determine the environmental data of the space where the corresponding embodied intelligent body is located based on the behavior information. Among them, the behavior information includes the description information of the detailed behavior actions made by the embodied intelligent body, the location information of the location, and other information related to the behavior. After the urban environment base platform 700 obtains the latest reasoning decision result, it can determine the spatial position of the embodied intelligent body based on the location information in the behavior information, and then simulate and determine the environmental data. The environmental data can be information describing the environment of the space, such as temperature, humidity, environmental space type (kitchen, intersection, classroom, sidewalk, etc.) and related photos, weather, and information such as areas and places that the embodied intelligent body will be interested in in the current space. The present disclosure does not limit this. In addition, in the initial state, the urban environment base platform 700 can set the initial position and related environmental data for the embodied intelligent body.

[0107] In some embodiments, the urban environment base platform 700 can be configured with multiple interfaces, such as a perception interface, including an urban area of ​​interest (AOI) data acquisition interface, an urban point of interest (POI) data acquisition interface, a surrounding intelligent body data acquisition interface, an interactive element acquisition interface, etc. It can also include an urban embodied behavior push interface, such as a mobile interface, a dialogue interface, a shopping interface, etc., which is not limited by the present disclosure. The urban environment base platform 700 can transmit different types of data with other modules of the system through different interfaces.

[0108] In this embodiment, the urban environment base platform 700 may include a map service platform, an environment detection platform, and other existing or created platforms that can achieve the required functions.

[0109] The perception data processing submodule is used to process the perception data to form a perception data set that meets the format requirements.

[0110] In this embodiment, the usage requirements of each module in different data systems in the perception data are different, so the perception data needs to be processed so that the processed data meets the usage requirements and forms a perception data set. The processing of the perception data may include data structuring, data flattening, etc., which can be set according to actual needs, and the present disclosure does not limit this. For example, if the perception data includes a native street view picture of the current location, it can be processed to form a usable picture processed according to the orientation of the embodied intelligent body, FOV (Field of View, field of view) and other configuration parameters.

[0111] The memory docking submodule is used to send the sensed data set to the memory storage module 300 so that the memory storage module 300 can store the data in the sensed data set in a timely manner.

[0112] The memory storage module 300 is used to store the perception data set and the information of the agent portrait.

[0113] like Figure 4 As shown, the memory storage module 300 may include: a memory manager and a memory control submodule. The memory manager is used to manage a working memory storage unit and at least one long-term memory storage unit.

[0114] The working memory storage unit is used to store working memory, and the working memory includes each data in the perception data set and the index of the long-term memory stored in the long-term memory storage unit. In some embodiments, the working memory also includes part or all of the information of the received agent portrait.

[0115] In this embodiment, the perception module docking subunit in the working memory storage unit is used to realize data transmission (such as transmission of perception data set) with the perception module 200 through the data stream processing interface. The reasoning and analysis module docking subunit in the working memory storage unit is used to realize data transmission (such as transmission of target information) with the reasoning and analysis module 400 through the data stream processing interface.

[0116] In this embodiment, after receiving the perception data set, the perception module docking subunit in the working memory storage unit will store each data in the perception data set according to the working memory type. Figure 4 As shown, the working memory type may include environmental information for describing the spatial environment, social information for describing the social situation, travel information for describing the travel situation, other information, etc., and the present disclosure does not limit this.

[0117] In this embodiment, the working memory storage unit is also used to screen the internally stored data after receiving the perception data set or regularly, and send the information that needs to be stored for a long time to the long-term memory storage unit as the long-term memory to be stored. After receiving the long-term memory to be stored, the long-term memory storage unit stores it according to its corresponding long-term memory type and returns the index of the long-term memory to be stored to the working memory storage unit. After receiving the index of the long-term memory to be stored, the working memory storage unit stores the index so that when the subsequent target information includes the long-term memory, the corresponding long-term memory is obtained from the long-term memory storage unit based on the index and sent to the reasoning and analysis module 400.

[0118] The long-term memory storage unit is used to store long-term memory, and the long-term memory includes at least one of the following: part or all of the information of the received agent portrait, and the information to be stored sent by the working memory storage unit. In some embodiments, after receiving the data, the long-term memory storage unit will classify the data according to the long-term memory type and store it separately. Figure 4 As shown, the long-term memory type may include urban space information for describing the space, environment, etc. of different cities; social network information for describing the social network situation of embodied intelligent bodies; other information, etc., which is not limited by the present disclosure.

[0119] The memory control submodule is used to configure part or all of the storage information in the at least one long-term memory storage unit in response to the user's setting operation, such as setting the memory type (including the working memory type of working memory and the long-term memory type of long-term memory). The number of long-term memory storage units (wherein the long-term memory storage unit includes at least one) can also be increased or decreased, such as Figure 4 In the “Custom Implementation”, a new long-term memory storage unit is set, etc., and the present disclosure does not impose any restrictions on this.

[0120] The memory control submodule is also used to store the memory-related information input by the user through the memory loading interface into the corresponding working memory storage unit and / or long-term memory storage unit. The memory-related information can be stored in the memory library for the user to call and load into the memory storage module 300 at any time. The memory loading interface is directly associated with the external persistent memory library. By implementing this interface, the external loading of memory information can be realized, providing support for the functions of periodic memory snapshots and continuous execution of the embodied intelligent body.

[0121] The memory control submodule is also used to send the working memory and all the long-term memories to the corresponding memory bank for storage through the memory storage interface when the storage conditions are met. The storage conditions can be receiving an instruction to interrupt reasoning, receiving an instruction to end reasoning, reaching the time point for storing memory information (working memory and long-term memory), etc. Storing memory information facilitates memory loading for continuing reasoning after interruption and facilitates users to view the memory storage status during reasoning. The memory storage interface can be directly associated with an external persistent memory bank. By implementing this interface, external storage of memory information can be realized, providing support for functions such as regular memory storage and terminal preservation of embodied intelligent bodies.

[0122] The memory control submodule is also used to communicate with the agent portrait module 100 through the data stream processing interface, classify the agent portrait information sent by the agent portrait module 100 or obtained by the memory control submodule from the agent portrait module 100, and send the part belonging to the working memory to the working memory storage unit for storage, and send the part belonging to the long-term memory to the long-term memory storage unit for storage. The data stream processing interface is used to define the processing flow of the memory storage module 300 when receiving the data stream, and provide support for functions such as data structuring and data caching.

[0123] In this embodiment, the difference between working memory and long-term memory is that working memory can be information needed for the current real-time reasoning analysis of the embodied intelligent body, and long-term memory can be information that will not be used or is uncertain whether it will be used in the current real-time reasoning analysis of the embodied intelligent body. In this way, by storing information in long-term memory and working memory, the speed at which the subsequent reasoning and analysis module 400 obtains the target information from the memory storage module 300 can be accelerated, while also ensuring that all information related to the embodied intelligent body is stored in the memory storage module 300.

[0124] The reasoning and analysis module 400 is used to obtain the target information required for the required reasoning and analysis from the memory storage module, and call the matching target model in the model warehouse 800 to perform reasoning and analysis based on the target information to obtain the reasoning decision result corresponding to the target information, and the reasoning decision result includes behavioral information describing the behavior of the embodied intelligent body in the space where it is located.

[0125] In this embodiment, the target information may include: part or all of the information in the agent portrait required for reasoning analysis, part or all of the information in the working memory, and part of the information in the long-term memory, which is not limited by the present disclosure. The reasoning and analysis module 400 or the memory storage module 300 can determine which reasoning analysis needs to be performed based on the latest behavior information of the embodied agent (that is, the behavior information obtained by the perception module 200 from the urban environment base platform 700), and then determine the information required to implement these reasoning analyses to form target information. In some embodiments, the target information may also include a reasoning target, which is used to indicate which results need to be obtained after the reasoning analysis.

[0126] like Figure 5 As shown, the reasoning and analysis module 400 may include: a reasoning and analysis control submodule and a reasoning and analysis manager.

[0127] The reasoning analysis control submodule is used to obtain the target information required for reasoning analysis from the working memory storage unit in the memory storage module 300. In some embodiments, the reasoning analysis control submodule can send a target information request to the reasoning and analysis module docking subunit. The working memory storage unit returns the corresponding working memory based on the information requirements in the target information request, and the corresponding long-term memory stored in the long-term memory storage unit based on the index as the target information to the reasoning analysis control submodule.

[0128] The reasoning and analysis manager is used to use multiple reasoning and analysis units to analyze the target information to determine at least one target model that needs to be called, and generate reasoning requests for each target model based on the target information. Each reasoning and analysis unit has different analysis functions.

[0129] In some embodiments, the analysis functions of each of the reasoning and analysis units include any one of the following: itinerary planning, dialogue generation, environment understanding, shopping selection, and target generation; the target model includes at least one of the following: a large language base model, an image generation model, and a speech generation model. The number of reasoning and analysis units, the analysis functions, and the functions of each model in the model warehouse can be set according to actual needs, and this disclosure does not limit this.

[0130] In this embodiment, the user can set the priority of multiple reasoning and analysis units according to actual needs through the reasoning and analysis manager or the reasoning and analysis manager itself.

[0131] The reasoning and analysis control submodule is also used to send the target information to each of the reasoning and analysis units in order from high to low priority when each of the reasoning and analysis units is set with a corresponding priority.

[0132] Each of the reasoning and analysis units is used to analyze the target information when it receives the target information, and when it is determined after the analysis that the target model needs to be called, generate a reasoning request for the target model based on the target information and send it to the reasoning analysis control submodule.

[0133] In this embodiment, after receiving the target information, each reasoning and analysis unit extracts features of the target information according to its own analysis function. If feature information corresponding to the analysis function is extracted, the target model to be called is further determined based on the feature information, and a corresponding reasoning request is generated according to the feature information. If the feature information corresponding to the analysis function cannot be extracted, it is determined that the analysis required by the reasoning and analysis unit does not need to be performed for the target information, and no reasoning request needs to be generated, and a report that no reasoning is required can be returned to the reasoning analysis control submodule.

[0134] In some embodiments, the reasoning and analysis control submodule may send target information to the reasoning and analysis unit with the highest priority among the remaining reasoning and analysis units after determining that the previous reasoning and analysis unit has completed the analysis (received the reasoning request of the previous reasoning and analysis unit, and no reasoning report is required). Alternatively, the reasoning and analysis control submodule may also send target information to each reasoning and analysis unit in a sequence and at intervals corresponding to the time for each reasoning and analysis unit to perform the analysis. Alternatively, the reasoning and analysis control submodule may also send target information to the reasoning and analysis unit with the highest priority among the remaining reasoning and analysis units after determining that the reasoning result corresponding to the reasoning request of the previous reasoning and analysis unit has been returned.

[0135] The reasoning analysis control submodule is also used to send each reasoning request to the model warehouse 800 through the model call interface, so that the model warehouse 800 sends each reasoning request to the corresponding target model and returns the reasoning result obtained after each target model executes the corresponding reasoning request.

[0136] The reasoning analysis control submodule is further used to receive the reasoning results returned by the model warehouse 800 through the model result interface, and send the reasoning decision results corresponding to the target information formed according to the reasoning results to the behavior module 500.

[0137] In this way, by calling the model in the model warehouse 800 to perform more specific reasoning, more accurate reasoning results can be obtained.

[0138] For example, the inference request carries a street view image from the working memory (i.e., the target information), and the inference decision result can be "move forward". The inference request carries the perceptual information from the working memory, and the inference target is to find an area of ​​interest. The inference decision result is a Boolean value used to indicate whether there is an area of ​​interest of the specified type around. The inference request instructs the mobile plan to be planned based on the current situation of the embodied intelligent agent, and the inference decision result can be a mobile plan. The inference request instructs the generation of a dialogue based on the current social information of the embodied intelligent agent, and the inference decision result can be the content of the dialogue.

[0139] The behavior module 500 is used to perform structured processing on the reasoning and decision results and send them to the urban environment base platform 700, so that the urban environment base platform 700 forms perception data storage based on the processed reasoning and decision results and then continues to be acquired by the perception module 200 to realize a circular reasoning analysis of the embodied intelligent body behavior.

[0140] like Figure 3 As shown, the behavior module 500 may include a reasoning and analysis docking submodule, a data structuring submodule, and a behavior information synchronization submodule.

[0141] The reasoning and analysis docking submodule is used to receive the reasoning decision result from the reasoning and analysis module.

[0142] The data structuring submodule is used to perform structural processing on the inference decision result according to the data structure requirements of the urban environment base platform 700 to obtain the processed inference decision result. In some embodiments, the structural processing includes serialization, textualization, etc., which is not limited by the present disclosure.

[0143] The behavior information synchronization submodule is used to send the processed reasoning decision results to the urban environment base platform 700, so that the urban environment base platform 700 forms and stores perception data based on the processed reasoning decision results.

[0144] In some embodiments, data transmission between the behavior information synchronization submodule and the urban environment base platform 700 can be carried out through corresponding interfaces, and the corresponding information in the processed reasoning decision results can be transmitted through different types of interfaces, such as the transmission of corresponding information through the above-mentioned mobile interface, dialogue interface, and shopping interface.

[0145] Through the above method, the intelligent agent portrait module 100 reads the basic information and prior knowledge information of the embodied intelligent agent from the database 600, processes it and obtains the "intelligent agent portrait", which then serves as context information to influence the reasoning and analysis performed by the reasoning and analysis module 400.

[0146] When the embodied intelligent body is running: the perception module 200 obtains the required perception data through the data interface provided by the urban environment base platform 700, and pushes it to the memory storage module 300 after processing. The memory storage module 300 receives the data from the perception module 200 and stores the perception data in working memory and / or long-term memory according to the storage scheme and requirements defined by the user. The reasoning and analysis module 400 obtains the required target information from the memory storage module 300, and then generates a reasoning request based on the information of the intelligent body portrait in the target information, the reasoning target and other target information and sends it to the model warehouse 800, and receives the reasoning results returned by the model warehouse 800 to form the reasoning decision result. The reasoning and analysis module 400 pushes the reasoning decision result to the behavior module 500. The behavior module 500 processes the reasoning decision result from the reasoning and analysis module 400 and sends it to the urban environment base platform 700, realizing a reasoning operation. After the perception module 200 obtains the required perception data from the urban environment base platform 700 again, the above process is repeated for the next reasoning operation.

[0147] The following is an example of an application according to an embodiment of the present disclosure, using "a user implements embodied intelligent body simulation reasoning for street view navigation through the above system" as an exemplary application scenario, to facilitate understanding of the process of the modular urban embodied intelligent body reasoning system. Those skilled in the art should understand that the following application examples are only for the purpose of facilitating understanding of the embodiments of the present disclosure and should not be regarded as limiting the embodiments of the present disclosure.

[0148] The street view navigation problem is defined as follows: Given a discrete road network G =<V,E> (V represents a node in the road network, and E represents an edge between different nodes in the road network). The embodied intelligent agent moves through the nodes on the road network through the edges. The end point d is described by the relative angle relationship between a and several landmarks 1. The embodied intelligent agent compares the street view of the current position and the angle relationship between the current position and landmark 1 with the description of the end point d, and uses the reasoning ability of LLM (Large Language Model) to determine the direction of the end point relative to the current position and find a suitable path to reach the end point.

[0149] like Figure 6As shown, the user first performs configuration and preparation, which mainly includes three steps: 1. Urban environment base platform docking: The user achieves docking with the corresponding urban environment base platform 700 through other methods. Since street view navigation is required, the urban environment base platform 700 may include map service platforms such as Google map service platform and Baidu map service platform. Taking Baidu map service platform as an example, the user applies for Baidu map service access key, obtains Baidu map service link and other necessary information to achieve docking. 2. Model warehouse 800 docking: The user achieves docking with the model warehouse 800 through corresponding methods. The model warehouse can be either a self-built model warehouse or a public warehouse. Among them, taking the model warehouse 800 as Alibaba Qianwen model warehouse as an example, the user can apply for the access key of Qianwen model, Qianwen model access address and other necessary information to achieve docking; 3. Data preparation: The user completes data preparation through corresponding methods, including the necessary peripheral auxiliary data of the embodied intelligent body of street view navigation, such as basic configuration information data, related knowledge and experience data, etc., and stores the data in the database 600, such as MongoDB database.

[0150] After completing the configuration and preparation, the user constructs the perception module and the behavior module.

[0151] For the perception module, the target data of the embodied intelligent body of street view navigation is the street view picture of the embodied intelligent body's current location. The user needs to define and set the perception module as follows: the perception module first implements the approach picture acquisition function based on the API (Application Programming Interface) provided by the Baidu Map Service Platform; secondly, the perception module can realize the processing function of target information according to task characteristics and requirements, such as the need to cut the panoramic street view picture into a human-like perspective; finally, the perception module needs to realize the working memory docking function, that is, push the processed street view picture to the working memory storage unit.

[0152] For the behavior module, the behavior of the embodied intelligent body of the street view navigation is movement. The user needs to define and set the behavior module as follows: first, define the docking interface with the reasoning and analysis module, including the type of data received and the amount of data; second, implement the data structuring function, and convert the reasoning and decision results transmitted by the reasoning and analysis module into a data format that can be understood by the Baidu map service platform (such as the longitude and latitude coordinates under the specified coordinate system); finally, implement the behavior information synchronization function, that is, send the corresponding processed reasoning and decision results to the Baidu map service platform based on the push interface provided by the Baidu map service platform.

[0153] After completing the construction of the perception module and the behavior module, the user proceeds to construct the core functional modules, including the intelligent agent portrait module, the memory storage module, and the reasoning and analysis module.

[0154] For the agent portrait module, the user needs to clearly define the data requirements of the embodied agent of the street view navigation; construct a data loading and processing submodule to realize information loading based on MongoDB (i.e., database 600); load the basic information and prior knowledge information corresponding to the data requirements, and then configure them through the basic information submodule and the knowledge and experience submodule.

[0155] For the memory storage module, the user needs to instantiate a working memory and a long-term memory. The working memory is used to store street view images and current location coordinates from the perception module, and the long-term memory contains urban space information to determine whether the destination has been reached.

[0156] For the reasoning and analysis module, users need to build a destination arrival reasoning and analysis unit and a street view navigation reasoning and analysis unit:

[0157] For the destination arrival reasoning class, its working memory docking unit obtains the current location information from the working memory, and requires the working memory to index the long-term memory to obtain the associated spatial information. The reasoning decision result is output as a Boolean type (if it is True, it means that the destination has been reached), and its behavior module docking unit is empty; a reasoning request is sent to the Qianwen model through the model call interface, requiring the Qianwen model to determine whether the specified location has been reached based on the given environmental information; the reasoning result returned from the Qianwen model is received through the model result interface and saved as a Boolean type.

[0158] For the street view navigation reasoning class, its working memory docking unit obtains the street view image from the working memory and outputs the direction of the next step; its behavior module docking unit pushes the inferred direction data to the behavior module; a reasoning request is issued to the Qianwen multimodal model through the model call interface, requiring the Qianwen multimodal model to return the direction of the next step (that is, the reasoning result); the reasoning result returned from the Qianwen multimodal model is received through the model result interface and saved in the specified data format.

[0159] Among them, the user can set the priority of the two reasoning and analysis units in the reasoning and analysis manager to two levels, among which the priority of the destination arrival reasoning and analysis unit is higher than the street view navigation reasoning and analysis unit. Therefore, during the execution process, it will first determine whether the destination has been reached, and then perform navigation reasoning.

[0160] After completing the construction of the core functional modules, the user runs the street view navigation embodied agent and verifies the correctness of the results.

[0161] The present disclosure also provides a modular urban embodied agent reasoning method, which is applied to the above modular urban embodied agent reasoning system. The method includes:

[0162] Acquire the perception data of the embodied intelligent body in the current space from the urban environment base platform, process the perception data to form a perception data set that meets the format requirements, and store it in the memory storage module, wherein the perception data includes the behavior information of the embodied intelligent body in the urban environment and the environmental data of the current space;

[0163] Acquire target information required for the required reasoning analysis from the memory storage module, and call the matching target model in the model warehouse to perform reasoning analysis according to the target information to obtain a reasoning decision result corresponding to the target information, wherein the reasoning decision result includes behavior information describing the behavior of the embodied intelligent body in the space where it is located;

[0164] The reasoning decision result is sent to the urban environment base platform after structured processing, so that the urban environment base platform forms a perception data storage based on the processed reasoning decision result and then continues to be acquired to realize a circular reasoning analysis on the behavior of the embodied intelligent body;

[0165] Among them, the memory storage module stores the perception data set, and stores part or all of the information of the intelligent agent portrait, and the intelligent agent portrait is formed based on the basic information and prior knowledge information corresponding to the embodied intelligent agent obtained from the database in response to the configuration operation.

[0166] It should be noted that although the above embodiments are used as examples to introduce the modular urban embodied intelligent agent reasoning system and method, those skilled in the art will understand that the present disclosure should not be limited thereto. In fact, users can flexibly set each module, unit and step according to personal preferences and / or actual application scenarios, as long as they comply with the technical solution of the present disclosure.

[0167] In some embodiments, the functions or modules included in the system provided by the embodiments of the present disclosure can be used to execute the methods described in the above method embodiments. The specific implementations of the systems and methods can refer to each other and will not be repeated here for the sake of brevity.

[0168] The embodiment of the present disclosure also provides a computer-readable storage medium on which computer program instructions are stored, and the computer program instructions implement the above method when executed by a processor. The computer-readable storage medium can be a volatile or non-volatile computer-readable storage medium.

[0169] The disclosed embodiment also proposes a modular urban embodied intelligent agent reasoning device, comprising: a processor; a memory for storing processor executable instructions; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0170] The embodiments of the present disclosure also provide a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0171] Figure 7 1 is a block diagram of a device 1900 for modular urban embodied agent reasoning according to an exemplary embodiment. For example, the device 1900 can be provided as a server or a terminal device. Figure 7 , the apparatus 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions, such as an application, that can be executed by the processing component 1922. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.

[0172] The device 1900 may also include a power supply component 1926 configured to perform power management of the device 1900, a wired or wireless network interface 1950 configured to connect the device 1900 to a network, and an input / output interface 1958 (I / O interface). The device 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server 2000. TM , MacOS X TM , Unix TM , Linux TM , FreeBSD TM or similar.

[0173] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the device 1900 to perform the above method.

[0174] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0175] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. More specific examples (a non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the above. The computer-readable storage medium used herein is not to be interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0176] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0177] The computer program instructions for performing the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect through the Internet). In some embodiments, by using the state information of the computer-readable program instructions to personalize an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), the electronic circuit may execute the computer-readable program instructions, thereby implementing various aspects of the present disclosure.

[0178] Various aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of boxes in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0179] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0180] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0181] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0182] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A modular urban embodied agent reasoning system, characterized in that: The system comprises: An agent portrait module, configured to obtain basic information and prior knowledge information corresponding to the embodied agent from a database in response to a configuration operation, and to form an agent portrait based on the basic information and prior knowledge information; A perception module, used to obtain the perception data of the embodied intelligent body in the current space from the urban environment base platform, and process the perception data to form a perception data set that meets the format requirements, wherein the perception data includes the behavior information of the embodied intelligent body in the urban environment and the environmental data of the current space; A memory storage module, used to store the perception data set and the information of the agent portrait; A reasoning and analysis module, used to obtain target information required for reasoning and analysis from the memory storage module, and call the matching target model in the model warehouse to perform reasoning and analysis according to the target information, so as to obtain a reasoning and decision result corresponding to the target information, wherein the reasoning and decision result includes behavior information describing the behavior of the embodied intelligent body in the space in which it is located; A behavior module is used to perform structured processing on the inference decision result and then send it to the urban environment base platform, so that the urban environment base platform forms a perception data storage based on the processed inference decision result and then continues to be acquired by the perception module to realize a circular reasoning analysis on the behavior of the embodied intelligent body; Wherein, the reasoning and analysis module includes: A reasoning analysis control submodule, used for acquiring target information required for reasoning analysis from the memory storage module; A reasoning and analysis manager, configured to analyze the target information using a plurality of reasoning and analysis units to determine at least one target model to be called, and to generate a reasoning request for each target model based on the target information, wherein each reasoning and analysis unit has a different analysis function; The reasoning analysis control submodule is further used to send each of the reasoning requests to the model warehouse through the model calling interface, so that the model warehouse sends each of the reasoning requests to the corresponding target model and returns the reasoning results obtained after each of the target models executes the corresponding reasoning request; The reasoning analysis control submodule is further used to receive the reasoning results returned by the model warehouse through the model result interface, and send the reasoning decision results corresponding to the target information formed according to the reasoning results to the behavior module.

2. The system according to claim 1, characterized in that The intelligent agent portrait module includes: A data loading and processing submodule, for acquiring basic information and prior knowledge information corresponding to the embodied intelligent body from a database in response to a configuration operation, and performing corresponding processing on the basic information and the prior knowledge information to obtain processed basic information and processed prior knowledge information; A basic information submodule, used to configure and load basic information in the agent portrait based on the processed basic information; The knowledge and experience submodule is used to load the configuration of knowledge and experience in the intelligent agent portrait based on the processed prior knowledge information.

3. The system according to claim 1, characterized in that The perception module comprises: A data perception submodule, used to obtain the perception data of the embodied intelligent body in the current space through the perception interface provided by the urban environment base platform; A perception data processing submodule, used to process the perception data to form a perception data set that meets format requirements; The memory docking submodule is used to send the perception data set to the memory storage module.

4. The system according to claim 1 or 3, characterized in that: The memory storage module comprises: A memory manager, for managing a working memory storage unit and at least one long-term memory storage unit, The working memory storage unit is used to store working memory, and the working memory includes at least one of the following: each data in the perception data set, part or all of the received information of the agent portrait, and the index of the long-term memory stored in the long-term memory storage unit; The long-term memory storage unit is used to store long-term memory, and the long-term memory includes at least one of the following: part or all of the information of the received agent portrait, and the long-term memory to be stored sent by the working memory storage unit; The memory control submodule is used to configure part or all of the storage information in the at least one long-term memory storage unit in response to the user's setting operation, and to store the memory-related information input by the user through the memory loading interface into the corresponding working memory storage unit and / or long-term memory storage unit, and to send the working memory and all the long-term memories to the corresponding memory bank through the memory storage interface for storage when the storage conditions are met.

5. The system according to claim 1, characterized in that The analysis functions of each of the reasoning and analysis units include any one of the following: itinerary planning, dialogue generation, environment understanding, shopping selection, and target generation; the target model includes at least one of the following: a large language base model, an image generation model, and a speech generation model; The reasoning and analysis control submodule is further used to send the target information to each of the reasoning and analysis units in descending order of priority when each of the reasoning and analysis units is set with a corresponding priority; Each of the reasoning and analysis units is used to analyze the target information when it receives the target information, and when it is determined after the analysis that the target model needs to be called, generate a reasoning request for the target model based on the target information and send it to the reasoning analysis control submodule.

6. The system according to claim 1, characterized in that The behavior module includes: A reasoning and analysis docking submodule, used for receiving the reasoning decision result from the reasoning and analysis module; A data structuring submodule, used for performing structural processing on the inference and decision result according to the data structure requirements of the urban environment base platform to obtain the processed inference and decision result; The behavior information synchronization submodule is used to send the processed reasoning decision results to the urban environment base platform, so that the urban environment base platform forms and stores perception data based on the processed reasoning decision results.

7. The system according to claim 5, characterized in that The system further includes at least one of the database, the urban environment base platform, and the model warehouse; Wherein, the database is used to store various pre-set basic information and prior knowledge information; The urban environment base platform is used to determine the behavior information of the corresponding embodied intelligent body according to the received reasoning decision result, and determine the environmental data of the space where the corresponding embodied intelligent body is located based on the behavior information; The model warehouse is used to store multiple available models, and when receiving an inference request, sends the inference request to a target model among the multiple available models that can execute the inference request, so that the target model that receives the inference request executes the inference request and returns an inference result; The database also includes the urban environment base platform.

8. A modular urban embodied agent reasoning method, characterized in that: Applied to the modular urban embodied agent reasoning system as claimed in any one of claims 1 to 7, the method comprising: Acquire the perception data of the embodied intelligent body in the current space from the urban environment base platform, process the perception data to form a perception data set that meets the format requirements, and store it in the memory storage module, wherein the perception data includes the behavior information of the embodied intelligent body in the urban environment and the environmental data of the current space; Acquire target information required for reasoning analysis from the memory storage module, and call the matching target model in the model warehouse to perform reasoning analysis according to the target information, so as to obtain a reasoning decision result corresponding to the target information, wherein the reasoning decision result includes behavior information describing the behavior of the embodied intelligent body in the space where it is located; The reasoning decision result is sent to the urban environment base platform after structured processing, so that the urban environment base platform forms a perception data storage based on the processed reasoning decision result and then continues to be acquired to realize a circular reasoning analysis on the behavior of the embodied intelligent body; Among them, the memory storage module stores the perception data set and stores information of the intelligent agent portrait, and the intelligent agent portrait is formed based on basic information and prior knowledge information corresponding to the embodied intelligent agent obtained from the database in response to the configuration operation.

9. A modular urban embodied agent reasoning device, characterized in that: include: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to implement the method of claim 8 when executing the instructions stored in the memory.

10. A non-volatile computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method of claim 8 is implemented.