Intelligent agent interpretation graph processing method and device, equipment, medium and product

Abstract

The embodiment of the invention provides an agent interpretation graph processing method, which can be applied to the technical field of artificial intelligence. The interpretation graph processing method of the intelligent agent comprises the steps of obtaining a first interpretation graph generated by resource initialization operation of the intelligent agent; according to a preset interpretation graph representation rule and the detected environment detection information, the first interpretation graph is updated to generate a second interpretation graph, and the preset interpretation graph representation rule is a representation rule of object information of the second interpretation graph and the relation between objects; and performing processing on the second interpreted graph. The embodiment of the invention further provides an agent interpretation graph processing device and equipment, a storage medium and a program product.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and more specifically to a method, device, equipment, medium and product for processing an interpretation graph of an intelligent agent. Background Art

[0002] Artificial Intelligence (AI) is an important driving force for the new round of scientific and technological revolution and industrial transformation. It is a new key technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. As an important part of intelligent science, artificial intelligence attempts to understand the essence of intelligence and produce a new intelligent machine (i.e., intelligent agent) that can respond in a similar way to human intelligence.

[0003] As an important carrier for intelligent agents to recognize and understand environmental information, the interpretation map can be dynamically updated as the intelligent agent perceives changes in the environment, so that the intelligent agent can accurately understand the environment and complete intelligent behavior. In this process, the intelligent agent's processing of the interpretation map is particularly important. The processing cost of the intelligent agent in the prior art for processing the interpretation map is high, and it is easy to lack a more accurate description for complex new objects. Overall, the processing efficiency of the interpretation map is low, and the range of objects covered is narrow. Summary of the invention

[0004] In view of at least one of the above problems, embodiments of the present invention are directed to an interpretation graph processing method, apparatus, device, medium and product that can significantly reduce the complexity of interpretation graph processing to greatly improve the efficiency of interpretation graph processing.

[0005] An aspect of an embodiment of the present invention provides a method for processing an interpretation graph of an intelligent agent, which includes: obtaining a first interpretation graph generated by a resource initialization operation of the intelligent agent; updating the first interpretation graph according to preset interpretation graph representation rules and detected environmental detection information to generate a second interpretation graph, wherein the preset interpretation graph representation rules are representation rules for object information and relationships between objects in the second interpretation graph; and performing processing on the second interpretation graph.

[0006] According to one embodiment of the present invention, in obtaining the first interpretation graph generated by the resource initialization operation of the intelligent agent, it includes: applying for the corresponding virtual memory through the preset operating system; marking all virtual memory pages in the virtual memory as private; initializing the memory allocator according to the base address and capacity of the virtual memory after the marking is completed, so as to obtain the first interpretation graph.

[0007] According to one embodiment of the present invention, in updating a first interpretation diagram according to a preset interpretation diagram representation rule and detected environmental detection information to generate a second interpretation diagram, it includes: updating the first interpretation diagram according to an object-centric memory storage layout method and object information of environmental detection information in the preset interpretation diagram representation rule to generate a second interpretation diagram; and / or updating the first interpretation diagram according to a component-centric memory storage layout method and object information of environmental detection information in the preset interpretation diagram representation rule to generate a second interpretation diagram.

[0008] According to one embodiment of the present invention, in updating the first interpretation map according to preset interpretation map representation rules and detected environmental detection information to generate a second interpretation map, it also includes: updating the first interpretation map according to the representation method of the relationship between objects in the preset interpretation map representation rules and the object relationship information of the environmental detection information to generate a second interpretation map.

[0009] According to an embodiment of the present invention, performing processing on the second interpretation graph includes: performing a modification operation on the second interpretation graph according to a preset stream incremental processing rule.

[0010] According to an embodiment of the present invention, the processing of the second interpretation graph includes: when it is found that the corresponding virtual memory capacity is insufficient during the execution of the modification operation of the second interpretation graph, performing a capacity expansion operation.

[0011] According to one embodiment of the present invention, in the process of executing processing on the second interpretation graph, it also includes executing a copy operation on the second interpretation graph to generate a third interpretation graph, which includes: applying for a virtual memory of the third interpretation graph with the same capacity as the second interpretation graph through a preset operating system, and mapping the data content of the second interpretation graph to the virtual memory page of the virtual memory of the third interpretation graph; marking all virtual memory pages in the virtual memory of the second interpretation graph and all virtual memory pages in the virtual memory of the third interpretation graph as shared; copying the memory allocator of the second interpretation graph to the third interpretation graph, and then correcting the base address of the virtual memory of the memory allocator of the third interpretation graph to the base address of the virtual memory of the third interpretation graph.

[0012] According to an embodiment of the present invention, the process of executing the second interpretation graph further includes: in response to a destroy instruction of the second interpretation graph, releasing the virtual memory of the second interpretation graph through a preset operating system.

[0013] Another aspect of an embodiment of the present invention provides an interpretation graph processing device of an intelligent agent, characterized in that it includes a resource initialization module, an interpretation graph representation module and an interpretation graph processing module. The resource initialization module is used to obtain a first interpretation graph generated by the resource initialization operation of the intelligent agent; the interpretation graph representation module is used to update the first interpretation graph according to a preset interpretation graph representation rule and detected environmental detection information to generate a second interpretation graph, wherein the preset interpretation graph representation rule is a representation rule for object information and the relationship between objects of the second interpretation graph; and the interpretation graph processing module is used to perform processing on the second interpretation graph.

[0014] Another aspect of an embodiment of the present invention provides an electronic device, comprising one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the above-mentioned intelligent agent's interpretation graph processing method.

[0015] Another aspect of an embodiment of the present invention provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to execute the above-mentioned intelligent agent interpretation graph processing method.

[0016] Another aspect of an embodiment of the present invention provides a computer program product, including a computer program, which implements the above-mentioned intelligent agent interpretation graph processing method when executed by a processor.

[0017] The intelligent agent interpretation graph processing method provided by the embodiment of the present invention can at least partially solve the problems of high complexity, low processing efficiency and poor processing accuracy in the interpretation graph processing process in the related art, and thus can achieve at least one of the following technical effects: (1) The copy and destroy operations of the interpretation graph can be transformed into the management and release operations of the underlying memory pages. At the same time, with the help of the write-time copy of the virtual memory pages, the large amount of memory data copying caused by the interpretation graph copy operation is eliminated. Finally, the time complexity of the interpretation graph copy and destroy operations is reduced from O(n) to O(1), where n represents the number of objects in the interpretation graph.

[0018] (2) The concept of components is introduced through the representation rules of objects and their relationships. Each component can represent a function or attribute unit, and each object can be composed of several components. The combination of components can represent some multifunctional, complex, and new objects that cannot be anticipated in advance during design (n components can be combined into 2^n different types of objects). The types of objects that can be represented grow exponentially with the number of component types. For users of the interpretation graph (such as planners), when their internal logic depends only on components, they can be easily generalized to new objects that appear later. The component-centric memory layout ensures that component data of the same type is continuous in memory, with a high cache hit rate during traversal, and facilitates parallel processing using multi-core CPUs or GPUs.

[0019] (3) The relationship table is further introduced through the representation rules of the relationship between objects. The relationship table can record the {source-relationship descriptor-target} three-tuple relationship pairs, and the relationship descriptors are divided into four types: 1 to 1, 1 to N, N to 1, and N to N. Therefore, the relationship table can support the following operations with O(1) complexity: relationship conflict detection, querying the relationships (forward / reverse) owned by an object, and querying the source set of a relationship (forward / reverse).

[0020] (4) Through streaming incremental parsing, the calculation scope can be narrowed from all objects and component sets to the objects and component sets that have changed, further reducing the complexity of processing or updating operations on the interpretation graph.

[0021] Therefore, the interpretation graph processing method of the embodiment of the present invention can make the interpretation graph of the intelligent body more accurately and comprehensively reflect the objective information of the real environment, achieve the effect of flexibly and efficiently representing and processing the interpretation graph, greatly reduce the complexity of the interpretation graph processing operation, and avoid the high computational cost caused by the cascade creation, copying, and destruction of a large number of small objects caused by interpretation graph processing operations (such as copying and destruction, etc.); and can accurately describe newly emerging multifunctional or more complex new objects, avoid inconsistencies such as missing or extra attributes, and achieve more efficient batch processing, greatly reducing the computational cost.

[0022] It should be understood that the above general description and the following detailed description are merely exemplary and illustrative and are not intended to limit the scope of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The above contents and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which: Figure 1 A diagram schematically shows an application scenario of an intelligent agent interpretation graph processing method, device, equipment, medium and program product according to an embodiment of the present invention; Figure 2A A flowchart of a method for processing an interpretation graph of an intelligent agent according to an embodiment of the present invention is schematically shown; Figure 2B A specific application scenario diagram of the interpretation graph processing method of an intelligent agent according to an embodiment of the present invention is schematically shown; Figure 3A Schematically shows a copy-on-write scene graph of the interpretation graph processing method of an intelligent agent according to an embodiment of the present invention; Figure 3B A scene graph of an interpretation graph initialization of an interpretation graph processing method of an intelligent agent according to an embodiment of the present invention is schematically shown; Figure 3C A scene graph representing an interpretation graph with an object as the center is schematically shown in the interpretation graph processing method of an intelligent agent according to an embodiment of the present invention; Figure 3D A scene graph of a component-centric interpretation graph is schematically shown in the interpretation graph processing method of an intelligent agent according to an embodiment of the present invention; Figure 3E Schematically shows a structural diagram of an interpretation graph relationship table of an interpretation graph processing method of an intelligent agent according to an embodiment of the present invention; Figure 4A Schematically shows a structural diagram of an interpretation graph pointer of an interpretation graph processing method of an intelligent agent according to an embodiment of the present invention; Figure 4B A scene diagram of virtual memory expansion is schematically shown in the method for processing the interpretation graph of an intelligent agent according to an embodiment of the present invention; Figure 4C Schematically shows a scene graph of an interpretation graph replication of an interpretation graph processing method of an intelligent agent according to an embodiment of the present invention; Figure 5 A schematic diagram showing a structural block diagram of an interpretation graph processing device of an intelligent agent according to an embodiment of the present invention; and Figure 6 A block diagram of an electronic device suitable for implementing a method for processing an interpretation graph of an intelligent agent according to an embodiment of the present invention is schematically shown.

[0024] The above-mentioned drawings are part of the specification of the embodiments of the present invention, which illustrate exemplary embodiments of the present invention. The attached drawings and the description of the specification are used together to illustrate the principles of the embodiments of the present invention. It should be understood that the above general description of the drawings and the following specific implementations are only exemplary and illustrative, and they cannot limit the scope of the present invention. DETAILED DESCRIPTION

[0025] In order to make the objectives, technical solutions and advantages of the embodiments of the present invention more clearly understood, the spirit of the contents disclosed by the present invention will be clearly explained with the accompanying drawings and detailed descriptions below. After understanding the embodiments of the contents of the present invention, any technician in the relevant technical field can change and modify the techniques taught by the contents of the present invention without departing from the spirit and scope of the contents of the present invention.

[0026] The exemplary embodiments and descriptions of the present invention are used to explain the present invention, but are not intended to limit the present invention. In addition, elements / components with the same or similar reference numerals used in the drawings and embodiments are used to represent the same or similar parts.

[0027] The terms “first”, “second”, etc. used in the present invention do not particularly refer to an order or sequence, nor are they used to limit the present invention. They are only used to distinguish elements or operations described with the same technical terms.

[0028] The directional terms used in the present invention, such as up, down, left, right, front or back, etc., are only used to refer to the directions of the drawings. Therefore, the directional terms used are used to illustrate and not to limit the present invention.

[0029] The words “include,” “including,” “have,” “contain,” etc. used in the present invention are open-ended terms, meaning including but not limited to.

[0030] The term "and / or" used in the present invention includes any or all combinations of items.

[0031] Regarding the present invention, "plurality" includes "two" and "more than two"; regarding the present invention, "plurality of groups" includes "two groups" and "more than two groups".

[0032] The terms "substantially" and "approximately" used in the present invention are used to modify any quantity or error that may vary slightly, but these slight changes or errors do not change their essence. Generally speaking, the range of slight changes or errors modified by such terms may be 20% in some embodiments, 10% in some embodiments, 5% in some embodiments, or other values. Those skilled in the art should understand that the aforementioned values ​​can be adjusted according to actual needs and are not limited thereto.

[0033] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0034] When using expressions such as "at least one of A, B, and C, etc.", it should generally be interpreted according to the meaning of the expression generally understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.). Those skilled in the art should also understand that any transitional conjunctions and / or phrases that substantially represent two or more optional items, whether in the specification, claims, or drawings, should be understood to give the possibility of including one of these items, either of these items, or both of these items. For example, the phrase "A or B" should be understood to include the possibility of "A" or "B", or "A and B".

[0035] In the field of artificial intelligence technology, a parse graph is a way to describe the environment in which an intelligent agent is located. The parse graph contains both objective information in the environment (such as the objects contained in the environment, the categories, functions, quantities, coordinates, colors, shapes and other attributes of the objects), as well as the relationships between objects (such as spatial relationships such as support and attachment) and the motion state of objects (speed, direction, posture and action of movement, etc.); at the same time, the parse graph can also contain the subjective information of the intelligent agent on these objects (such as the friendliness and belonging of the objects, etc.). Therefore, the parse graph is the key data for the intelligent agent to recognize and understand the world.

[0036] The interpretation map can be dynamically updated as the environment changes, so that the intelligent agent can use the interpretation map to understand the environment, and at the same time combine the intelligent agent's own goals and tasks to perform learning, cognition, planning and other behaviors. In this process, the intelligent agent will frequently read, copy, modify, and destroy the interpretation map (for example, updating the objective information in the interpretation map in the perception unit, updating the subjective information in the interpretation map in the cognition unit, and copying and modifying the interpretation map in the planning unit to achieve simulation), so it is very important to flexibly and efficiently represent and process the interpretation map.

[0037] The interpretation graph can usually contain a large number of objects, and each object in the computer can usually be composed of one or more small objects (or structures, the terminology varies in different programming languages). Due to cognitive and simulation needs, the intelligent agent will copy and destroy the interpretation graph. However, these operations often cause the cascade creation, copying, and destruction of a large number of small objects, resulting in high computational costs.

[0038] Moreover, conventional schemes often use an object type plus a number of loosely distributed attributes to represent objects in the interpretation graph. This leads to the following three situations: (1) Since object types usually come from a predetermined set, such as "refrigerator, microwave oven, oven, chair, bed, toy, book, etc.", some multifunctional or complex new objects that appear later, such as "microwave oven and fryer" and "multifunctional recliner", cannot be accurately described by these object types. (2) Object types and attributes are usually stored separately, so the generator and user of the interpretation graph need to agree in advance on which attributes each object type corresponds to. This leads to coupling between the two parties, which can easily lead to inconsistencies such as missing or extra attributes. At the same time, the reliance on object types makes it impossible for the user of the interpretation graph (such as the planner) to generalize to new objects. (3) Object information of the same type is often discontinuous in the memory address space, which also leads to low batch processing efficiency.

[0039] In addition, conventional solutions usually treat object relationships as object attributes. For example, if a teacup is on a table, that is, the teacup has a support relationship with the table, then a support attribute is added to the teacup, and the value is set to the ID of the table. This leads to the following two problems: (1) The maintenance and query efficiency of reverse relationships is low. Taking the support relationship as an example, when there is a need to quickly query which items are on the table, the simple way is to add a reverse support relationship to the table, but this will cause the forward and reverse relationships to be scattered on different objects, which can easily lead to inconsistency and omission of relationships; if the reverse support relationship is not added and the support is kept as a unidirectional relationship, then the above reverse query needs to traverse the entire interpretation graph to be realized, which results in a high computational cost. (2) When querying which items have a support relationship, it is also necessary to traverse the entire interpretation graph to obtain it, which is computationally expensive.

[0040] Furthermore, considering that the interpretation graph will be continuously updated as the environment changes, each time the interpretation graph is updated, when full processing or update operations are performed on it, the computational cost is high when the object size is large.

[0041] In view of at least one of the technical problems existing in the above-mentioned prior art, the embodiments of the present invention aim to provide an intelligent body's interpretation graph processing method, device, equipment, medium and product that can significantly reduce the complexity of interpretation graph processing to greatly improve the efficiency of interpretation graph processing.

[0042] An aspect of an embodiment of the present invention provides a method for processing an interpretation graph of an intelligent agent, which includes: obtaining a first interpretation graph generated by a resource initialization operation of the intelligent agent; updating the first interpretation graph according to preset interpretation graph representation rules and detected environmental detection information to generate a second interpretation graph, wherein the preset interpretation graph representation rules are representation rules for object information and relationships between objects in the second interpretation graph; and performing processing on the second interpretation graph.

[0043] Figure 1 The application scenario diagram of the interpretation graph processing method, device, equipment, medium and program product of the intelligent agent according to an embodiment of the present invention is schematically shown.

[0044] like Figure 1 As shown, the application scenario 1000 according to this embodiment may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for a communication link between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0045] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only examples).

[0046] The terminal devices 101 , 102 , and 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0047] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the terminal devices 101, 102, and 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0048] It should be noted that the interpretation graph processing method of the intelligent agent provided in the embodiment of the present invention can generally be executed by the server 105. Accordingly, the interpretation graph processing device of the intelligent agent provided in the embodiment of the present invention can generally be set in the server 105. The interpretation graph processing method of the intelligent agent provided in the embodiment of the present invention can also be executed by a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105. Correspondingly, the interpretation graph processing device of the intelligent agent provided in the embodiment of the present invention can also be set in a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105.

[0049] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0050] The following will be based on Figure 1 The scene described by Figure 2A to Figure 4C The interpretation graph processing method of the intelligent agent in the disclosed embodiment is described in detail.

[0051] Figure 2A The flowchart of the interpretation graph processing method of the intelligent agent according to an embodiment of the present invention is schematically shown.

[0052] like Figure 2A As shown, one aspect of an embodiment of the present invention provides a method for processing an interpretation graph of an intelligent agent, which includes operations S201 to S203.

[0053] In operation S201, a first interpretation graph generated by a resource initialization operation of an agent is obtained; In operation S202, the first interpretation map is updated according to a preset interpretation map representation rule and detected environment detection information to generate a second interpretation map, wherein the preset interpretation map representation rule is a representation rule for object information and a relationship between objects in the second interpretation map; and In operation S203, processing is performed on the second interpretation graph.

[0054] The agent may be the executor of the interpretation graph processing method of the agent in the embodiment of the present invention, or may be an executor controlled by the interpretation graph processing method of the agent, and may specifically be a humanoid intelligent robot or other AI device, which usually has an actuator capable of completing specific action tasks. For example, a humanoid robot may use a mechanical manipulator to complete the action task of picking up an object.

[0055] Resource initialization is the initialization of necessary resources and components (such as memory allocators, component sets, entity sets, and relationship tables) for the construction of the interpretation graph, which is used to generate the basic architecture of the interpretation graph. Specifically, the initialization may involve operations such as applying for virtual memory and initializing memory allocators. This ensures that objects, components, relationships between objects, etc. can be added to the first interpretation graph in the future to generate a second interpretation graph.

[0056] The first interpretation graph may only have the initial component structure of the interpretation graph. According to the generated content of the initial structure of the interpretation graph, the first interpretation graph may be further represented with environmental information nodes according to the representation rules and requirements of the interpretation graph to generate a clearer and more complete interpretation graph. The so-called environmental information node may be the objective information in the environment of the agent displayed by the current frame image obtained by the agent at the current moment. For example, the environmental information node may include the object itself, such as objective objects such as humans, dogs, tables, chairs, etc., and may also include behavioral actions, such as pouring water, opening cabinet doors, and other objective behaviors. The first interpretation graph may include functional units that the system has for configuring the interpretation graph structure, such as Figure 2B The interpretation graph 300 shown may include a memory allocator, an entity set, a component set, a relationship table, etc. Therefore, subsequent modification, copying, and destruction operations on the interpretation graph may be made more efficient, and the computational complexity of detection, reasoning, and other modules that rely on the interpretation graph may be reduced.

[0057] Environmental detection information is the state information generated by the agent based on its perception of the environment at the current moment, which may include the objects contained in the environment at the current moment and the attribute information of the objects (such as category, function, quantity, coordinates, color, shape, etc.), the relationship information between objects (such as support, attachment and other spatial relationships), the object motion state information (such as speed, direction, posture and action, etc.) and the subjective information of the agent (such as friendliness, belonging, etc.). Among them, environmental detection information can be obtained through the visual and tactile sensors of the agent, such as direct detection by the agent through "two eyes" (such as camera), "two ears" (such as microphone) and "two hands" (such as dexterous hands).

[0058] The content of the first interpretation graph can be represented and processed according to the information content of the acquired environmental detection information to form a second interpretation graph. The second interpretation graph can be an interpretation graph composition structure that improves environmental node information such as objects, object attributes, relationships between objects, object motion states, subjective understanding information of the intelligent agent, and relationships between nodes on the basis of the basic interpretation graph structure of the first interpretation graph, and can be used to comprehensively represent the objective information of the environment in which the intelligent agent is located in the current frame image and the subjective understanding information of the intelligent agent. The second interpretation graph can be continuously updated according to the changes in the environmental state detected by the intelligent agent to add new nodes and new node relationships.

[0059] The object information of the second interpretation diagram may be information of each object and the corresponding object component in the second interpretation diagram, wherein the component may be a functional unit or information unit of the object, such as the function of a refrigerator and a refrigerator is refrigeration. Accordingly, the relationship between objects in the second interpretation diagram may be the interactive relationship between objects in the second interpretation diagram, such as the cup is on the table, and the table has a supporting relationship with the cup. In addition, it may also include the subjective understanding information of the intelligent agent on the relationship between objects, such as after seeing the cup on the table, the intelligent agent can infer that there may be water in the cup.

[0060] The processing of the second interpretation graph may include operations such as reading, modifying, copying, updating and destroying the interpretation graph, and specifically, the processing of the interpretation graph may be performed by control units such as sensors and planners of the intelligent agent according to actual scene requirements (such as cognition, simulation, etc.). Specifically, it may involve copying, deleting, modifying each node in the second interpretation graph and updating the relationship between each node.

[0061] The first interpretation diagram is represented by the preset interpretation diagram representation rules based on object information and the relationship between objects. Compared with the traditional scheme that usually represents object relationships as object attributes, the complexity of the interpretation diagram processing operation can be reduced; at the same time, various newly emerging multifunctional complex objects can also be accurately described; and the effect of flexible and efficient representation and processing of interpretation diagrams is achieved.

[0062] Figure 2B A specific application scenario diagram of the interpretation graph processing method of an intelligent agent according to an embodiment of the present invention is schematically shown.

[0063] like Figure 2A-Figure 4C As shown, according to one embodiment of the present invention, in operation S201, the first interpretation graph generated by the resource initialization operation of the agent is obtained, including: In response to the start-up command or the wake-up command, the environment detection information is detected.

[0064] For the intelligent agent, the detection of environmental detection information can be executed according to the received start-up instruction or wake-up instruction. The start-up instruction or wake-up instruction can be a control instruction for controlling the perception sensor of the intelligent agent (such as a camera, etc.) to perform detection on the state information of the environment in which the intelligent agent is currently located. When the sensor 100 of the intelligent agent receives the start-up instruction or the wake-up instruction, the perception sensor of the intelligent agent can be controlled to detect and identify the environment, such as obtaining data such as audio, RGB images, depth maps, etc. Figure 2B The detection and recognition of the sensor 100 in operation S220 is shown.

[0065] Among them, when the intelligent agent is turned on for the first time, it can generate the startup instruction according to the power-on instruction and send it to the sensor 100 of the intelligent agent for controlling the perception of the environment; in other scenarios, the intelligent agent can also generate the wake-up instruction according to the user's voice control instruction and send it to the sensor 100, so that the intelligent agent can perform detection and recognition of the environment according to the target execution task content contained in the wake-up instruction. For example, the user suddenly sends a voice control instruction to the intelligent agent, and the intelligent agent turns its head to the user according to the voice control instruction, generating a wake-up instruction. The sensor can control the intelligent agent to detect the environmental state and its changes from the head turning process to the time when the user is seen according to the wake-up instruction; the intelligent agent can stop the environmental detection after the target and task are completed, so as to run in low power mode to save power before receiving new targets and tasks. It should be noted that some intelligent agents may continue to perform environmental detection before shutting down for safety reasons.

[0066] By perceiving and detecting the environment, the original media information collected by the sensors controlled by the sensor can be obtained, such as audio, RGB images, depth maps and other media information. Among them, after parsing and extracting the features of the original media information, the basic information of the environment in which the agent is located can be determined, such as the existence of people, objects, pets, etc., and the corresponding actions of these people, objects, pets, etc. and some of the most directly related attribute information (such as a moving car has wheels). These basic information can be added to the first interpretation map through the above-mentioned preset interpretation map representation rules, or updated to the second interpretation map, to form a second interpretation map corresponding to the environment at the current moment.

[0067] Among them, Figure 2B As shown, the interpretation graph 300 may include a memory allocator, an entity set, a component set, and a relationship table, etc. In the first interpretation graph, these components are in a ready state after initialization, and do not yet contain data such as object information in the environment and relationships between objects. Among them, the entity set can be understood as a set of identifiers or identification IDs of objects.

[0068] In response to the agent's start / wake-up instruction, after the initialization operation of the interpretation graph, the first interpretation graph is formed. This involves the initialization of the necessary resources and components of the interpretation graph (memory allocator, component set, relationship table, etc.), such as applying for virtual memory, initializing the memory allocator, etc. This ensures that objects, components, and relationships can be added to the interpretation graph to form a second interpretation graph.

[0069] In the embodiment of the present invention, Figure 2BAs shown, the processing flow of the interpretation graph actually needs to rely on the processing flow of the sensor 100 and the planner 200, such as cognitive reading, simulation and execution of the interpretation graph, which can reflect various operation modes such as initialization, modification and copying on the interpretation graph. In addition, the destruction processing operation of the interpretation graph can also be realized based on resource management, such as Figure 2B The operations S230-S270 are shown. The data content of the interpretation graph can be stored in a virtual memory page in the memory, and the modification and update operation of the interpretation graph can be implemented based on a virtual memory page with a copy-on-write function.

[0070] Figure 3A A copy-on-write scene graph of the interpretation graph processing method of an intelligent agent according to an embodiment of the present invention is schematically shown.

[0071] like Figure 3A As shown, the virtual memory can contain multiple virtual memory pages, and each virtual memory page can have the function of copy-on-write, where copy-on-write needs to rely on the following execution principles: First, it is necessary to consider that each virtual memory page corresponds to a shared tag, and the shared tag has two state values: private and shared. Among them, the virtual memory page in the private state can directly perform the write operation; for the virtual memory page in the shared state, when performing the write operation, the physical memory page corresponding to the virtual memory page must be copied first, and the virtual memory page must be mapped to the newly copied physical memory page, and the shared tag is set to private, and then the write operation is performed.

[0072] Therefore, by using the execution of the write-time copy for the virtual memory page, the copy and destroy operations of the interpretation graph can be converted into remapping and release operations of the underlying memory pages, eliminating the copying of a large amount of memory data caused by the interpretation graph copy operation. Ultimately, the time complexity of the interpretation graph copy and destroy operations is reduced from O(n) to O(1), where n represents the number of objects in the interpretation graph.

[0073] Therefore, in response to the agent's startup / wake-up instructions, the basic skeleton of the first interpretation graph is determined, making subsequent modification, copying, and destruction operations on the interpretation graph more efficient, reducing the computational complexity of detection, reasoning, and other modules that rely on the interpretation graph.

[0074] Figure 3B A scene graph for initializing an interpretation graph of an interpretation graph processing method of an intelligent agent according to an embodiment of the present invention is schematically shown.

[0075] like Figure 2A-Figure 4C As shown, according to an embodiment of the present invention, in operation S201, obtaining a first interpretation graph generated by the resource initialization operation of the agent includes: Apply for corresponding virtual memory through the preset operating system; Mark all virtual memory pages in virtual memory as private; The memory allocator is initialized according to the base address and capacity of the virtual memory after marking is completed to obtain the first interpretation graph.

[0076] like Figure 2B In operation S210, for the construction of the first interpretation graph, before creating the second interpretation graph, it is necessary to perform a resource initialization operation to realize the initial architecture generation of the interpretation graph, thereby obtaining the first interpretation graph.

[0077] First, if Figure 3B As shown, apply for the virtual memory required by the first interpretation graph. Among them, a section of virtual memory can be applied for through the API interface of the preset operating system, and the capacity of the virtual memory can be determined according to the size of the second interpretation graph estimated in advance. The operating system API (such as mmap on the Linux operating system) must meet the following requirements: the API only reserves virtual memory of corresponding capacity, but does not actually allocate physical memory of corresponding capacity, and only when a write operation occurs for the first time on the virtual memory page of the virtual memory, the corresponding physical memory page is allocated. Therefore, the time complexity of the operation has nothing to do with the memory capacity applied for. Among them, the preset operating system can be a platform for normal operation control of the intelligent agent, such as Linux and other operating systems.

[0078] After that, all virtual memory pages of the requested virtual memory are marked as private, such as Figure 3B Memory page #1 to memory page #3 in the virtual memory of the illustrated interpretation diagram are all marked as private.

[0079] After the private marking operation is completed on the virtual memory page, the memory allocator is initialized with the base address and virtual memory capacity of the requested virtual memory, such as Figure 2B The memory allocator of the interpretation graph 300 shown. The memory allocator is used to manage virtual memory and to implement the allocation of virtual memory by the memory allocator. In an embodiment of the present invention, the memory allocator itself can be located within the virtual memory. Of course, the memory allocator can also be designed to be outside the virtual memory in other application scenarios. The memory allocator can select any memory allocation algorithm as needed. For example, if the interpretation graph does not need to be read and written concurrently, a non-concurrency safe memory allocation algorithm can be selected to improve the allocation efficiency.

[0080] At the end of the initialization operation, the memory allocator that has completed the initialization operation can be used to initialize the relevant containers that store object information and relationships between objects in the interpretation graph, and obtain a relationship table, component set, object set (entity set) of size 0, where the relevant containers may include arrays, sets, dictionaries, smart pointers, etc. These containers can be understood as functional modules or units that perform modification operations such as adjusting objects and object relationships on the interpretation graph.

[0081] Therefore, through the resource initialization operation in the generation process of the above-mentioned first interpretation graph, many functions of the first interpretation graph can be realized by managing the virtual memory pages (remapping, copy-on-write, release), and at the same time, the time complexity of subsequent operations on the interpretation graph can be made independent of the applied virtual memory capacity, ensuring that the time complexity of the processing process of the second interpretation graph can be reduced from O(n) to O(1), thereby greatly improving the processing efficiency of the interpretation graph.

[0082] It should be noted that, considering that the intelligent agent can continue to run after the first startup, it continuously performs the cycle of perception, cognition, and planning. In this process, after the initialization operation and the generation of the first interpretation map, the interpretation map is continuously updated according to the acquired environmental state change data. Figure 2B As shown, during the resource initialization process of operation S210, a basic architecture of an interpretation graph can be generated as a first interpretation graph according to the above initialization operation. Thereafter, in operation S220, the original media data (such as audio, RGB image, depth image, etc.) and basic information of the environment detected therefrom (such as object information and the relationship between objects) can be obtained through the detection and recognition of the environment by the sensor 100. Therefore, the first interpretation graph after the initialization operation can be modified and processed (such as adding objects, establishing relationships between objects, etc.) through these data to construct a second interpretation graph.

[0083] Further, after the second interpretation diagram is constructed based on the first interpretation diagram, the second interpretation diagram can also be constructed according to the above Figure 2B In operation S220, the sensor 100 continuously detects and identifies the environment, and continuously updates the second interpretation map itself according to the acquired detection and identification data, so that the second interpretation map is constantly adjusted and changed, and describes the latest environmental status in a timely and accurate manner.

[0084] Figure 3C A scene graph representing an interpretation graph with an object as the center is schematically shown according to the interpretation graph processing method of an intelligent agent according to an embodiment of the present invention. Figure 3D A scene graph that represents an interpretation graph with components as the center is schematically shown according to the interpretation graph processing method of an intelligent agent according to an embodiment of the present invention.

[0085] like Figure 2A-Figure 4C As shown, according to an embodiment of the present invention, in operation S202, the first interpretation map is updated according to the preset interpretation map representation rule and the detected environment detection information to generate the second interpretation map, including: The first interpretation graph is updated according to the object-centric memory storage layout mode and the object information of the environment detection information in the preset interpretation graph representation rule to generate a second interpretation graph; and / or The first interpretation graph is updated according to the component-centric memory storage layout mode and the object information of the environmental detection information in the preset interpretation graph representation rule to generate a second interpretation graph.

[0086] In an embodiment of the present invention, the preset interpretation diagram representation rule may include an object-centric memory storage layout method, which can be understood as a rule that different types of component data belonging to the same object are closer in memory layout, thereby generating a second interpretation diagram based on the first interpretation diagram; in addition, it may also include a component-centric memory storage layout method, which can be understood as a rule that the same type of component data from different objects are closer in memory layout, thereby generating a second interpretation diagram based on the first interpretation diagram. Both interpretation diagram representation methods can realize interpretation diagram representation of environmental objects.

[0087] Among them, the component in the embodiment of the present invention can be a functional unit or information unit of an object, and each component can contain attribute information strongly related to the function or information unit, and the object can be represented by combining the components. Therefore, an object can be composed of multiple components, and the object will only mount the corresponding component if and only if the object has the corresponding function and information unit. Among them, the object information of the environmental detection information can be the corresponding information of each object and object component in the environmental detection data.

[0088] Therefore, through the combination of the above components, some multifunctional, complex and unforeseen objects can be represented (n components can be combined into 2^n different types of objects), and the types of objects that can be represented grow exponentially with the number of component types. For users of the interpretation graph (such as planners), when their internal logic depends only on components, they can easily generalize to new types of objects that have never been seen before.

[0089] First, for the object-centric memory storage layout, the object stores the mapping from the mounted component type to the component data. Figure 3C As shown, object #1 can be represented by component data #1A, #1B and component data #1D. For each component type, a corresponding component index can be optionally established, which records the object ID mounted with the component type and the corresponding component data pointer, such as Figure 3CAs shown, according to the component A-index, it can be determined that the object ID corresponding to the component A includes object #1 to object #3, and the component pointer corresponding to each object is represented as pointer #1A to pointer 3A. Therefore, through the component index, the object list mounted with the component type can be quickly queried, or the component type can be quickly batch processed.

[0090] In addition, for the component-centric memory storage layout, the same type of component data from different objects can be stored in containers such as arrays or hash tables to ensure that the same type of component data is continuous in memory space. The container type does not have to be fixed, and the container type used by the underlying lock can be automatically switched according to the sparsity of the component. For example, if a component is dense (almost all objects have mounted the component), an array can be used to store it; if a component is sparse (only a very small number of objects have mounted the component), a hash table can be used to store it. For each object, an index can be optionally created to record the type of component mounted (owned) by the object, so as to quickly query the list of components owned by the object. For example Figure 3D As shown, for component A, it can be represented based on object ID and component data. Object #1 with object ID #1 can mount component data #1A of component A, component data #1B of component B, and component data #1D of component D. Therefore, by implementing the representation of objects in this way, the component list owned by each object can be quickly queried.

[0091] In an embodiment of the present invention, there may further be a situation in which the above-mentioned object-centric memory storage layout method and component-centric memory storage layout method are mixed, that is, some components use the component-centric memory storage layout method to represent objects, and the remaining components can use the object-centric memory storage layout method to perform object representation.

[0092] The advantage of the object-centric memory storage layout is that it is relatively simple to implement. The advantage of the component-centric memory storage layout is that it ensures that the component data of the same type is continuous in memory, has a high cache hit rate during traversal, and is convenient for parallel processing using multi-core CPUs or GPUs.

[0093] In summary, the above-mentioned interpretation graph processing method of the embodiment of the present invention introduces the concept of components, each component represents a function or attribute unit, and each object is composed of several components. Through the combination of components, some multifunctional, complex, and new objects that cannot be anticipated in advance during design can be represented (n components can be combined into 2^n different types of objects), and the types of objects that can be represented grow exponentially with the number of component types. For users of the interpretation graph (such as planners), when their internal logic depends only on components, they can be easily generalized to new objects that appear later. Through the component-centric memory layout method, it is ensured that the component data of the same type is continuous in memory, the cache hit rate is high during traversal, and it is convenient to use multi-core CPUs or GPUs for parallel processing.

[0094] Figure 3E A structural diagram of a translation graph relationship table of a translation graph processing method of an intelligent agent according to an embodiment of the present invention is schematically shown.

[0095] like Figure 2A-Figure 4C As shown, according to an embodiment of the present invention, in operation S202, the first interpretation map is updated according to the preset interpretation map representation rule and the detected environment detection information to generate the second interpretation map, and further includes: The first interpretation graph is updated according to the representation method of the relationship between objects in the preset interpretation graph representation rule and the relationship information between objects in the environment detection information to generate a second interpretation graph.

[0096] For the representation of the relationship between objects in the second interpretation graph, such as Figure 3E As shown in , the relationship is separated from the object and the attribute, and a new relationship table is added to manage the relationship between objects. The relationship table records all the relationships between objects. Each relationship is recorded with two positive and negative 3-tuples ("source, relationship descriptor and target" and "target, reverse relationship descriptor and source"). The relationship 3-tuple is stored in an associative container (such as a hash table) with "source, relationship descriptor" as the key and "target" as the value. Figure 3E In the relationship 3-tuple list shown, source #1, relationship descriptor R1 and target #2 and corresponding source #2, relationship descriptor reverse R1 and target #1. Therefore, the time complexity of querying the forward relationship set and reverse relationship set of an object is O(1). Among them, the inter-object relationship information of the environmental detection information can be the interactive relationship information between objects.

[0097] Relationship descriptors can be divided into the following four categories: 1 to 1, 1 to N, N to 1, and N to N. Each time a relationship is inserted into the relationship table, it will be checked whether the type constraint of the relationship descriptor is violated (for example, for a 1 to N relationship descriptor, multiple sources cannot be associated with the same target). At the same time, the relationship table has a built-in relationship index. When a relationship is inserted, the relationship index will record the relationship descriptor and the corresponding source object at the same time, such as Figure 3E The relationship descriptor R1 shown corresponds to source sets #1 and #2.

[0098] Since the conflict detection of the insertion operation is implemented by querying the above-mentioned relation 3-tuple associative container, the time complexity of the insertion operation in the relation table is also O(1). Because of the existence of relation indexes, the complexity of querying the set of objects with a specific relation is also O(1).

[0099] By checking the constraints of relationship descriptors during insertion, errors can be exposed in a timely manner, making it easier to locate relationship errors and inconsistencies. Moreover, the time complexity of querying, inserting, and deleting relationships is O(1), making traversal operations on the relationship graph very efficient (such as detecting whether a relationship is a cycle). In addition, all relationships are stored in the relationship table. Compared with the scattered distribution on object attributes or components, the relationships can be processed in a centralized and unified manner (such as relationship graph drawing or serialization), and when adding or deleting relationship descriptors, there is no need to modify the above centralized and unified processing logic implementation. Finally, by introducing the relationship table, the relationship table can record the {source-relationship descriptor-target} 3-tuple relationship pair, and the relationship descriptors are divided into 4 types: 1 to 1, 1 to N, N to 1, and N to N. Therefore, the relationship table can support the following three operations with O(1) complexity: relationship conflict detection, querying the relationships (forward / reverse) owned by the object, and querying the source set of the relationship (forward / reverse).

[0100] Through the above representation of objects and relationships between objects, the subsequent processing of the generated second interpretation graph can greatly reduce the processing complexity, significantly reduce the coupling between the intelligent agent planner and the perceptron, and improve the versatility of the two.

[0101] like Figure 2A-Figure 4C As shown, according to an embodiment of the present invention, in operation S203, processing the second interpretation graph includes: A modification operation is performed on the second interpretation graph according to a preset streaming incremental processing rule.

[0102] like Figure 2B As shown, in operation S230, the second interpretation graph may be subjected to streaming incremental processing through the sensor 100, which may mainly involve modification and update operations, such as updating entities, components or relationship tables, and involving modifications to these entities, components or relationship tables.

[0103] Considering that the second interpretation graph will be continuously updated as the intelligent agent continuously detects changes in the environment state, the computational cost is high when full processing or update operations are performed on it after each interpretation graph is updated when the object size is large.

[0104] Assume that there is an operation on the interpretation graph: This operation needs to calculate whether any two objects have a support relationship based on the position, shape and other information of the objects (for example, a teacup placed on a table is considered to have a support relationship with the table). When this operation needs to be performed on all objects on the interpretation graph, its complexity is O(n^2), where n is the number of objects in the interpretation graph.

[0105] When the streaming incremental processing solution is adopted, the set of objects and components that have changed relative to the previous frame is recorded when each frame interpretation map is updated. Subsequent processing operations only need to process the objects in the change set, without processing all the objects in the interpretation map. Among them, the preset streaming incremental processing rules can be modification rules for modifying the second interpretation map of the current frame based on the change difference between the previous frame interpretation map and the current frame interpretation map. Specifically, the change difference between the previous frame interpretation map and the current frame interpretation map can be obtained based on the streaming incremental analysis of the interpretation map.

[0106] For example, for the calculation of the support relationship above, if the number of objects whose position components change in each frame is m, the update complexity of the support relationship of the interpretation graph will be reduced to O(n*m). In actual scenes, m is very small and generally has nothing to do with n, so the operation complexity can be regarded as O(n), which is a significant improvement over the original operation complexity O(n^2).

[0107] It can be seen that through streaming incremental parsing, the calculation scope can be narrowed from all objects and component sets to the objects and component sets that have changed, greatly reducing the complexity of processing or updating operations on the second interpretation graph.

[0108] It should be noted that the streaming incremental processing modification module is only one of many modules that may modify the interpretation graph. Detection and recognition, incremental processing, cognition, simulation and other modules of the intelligent agent may involve performing modification operations on the interpretation graph.

[0109] like Figure 2A-Figure 4C As shown, according to an embodiment of the present invention, in the processing of the second interpretation graph, the process includes: When it is found that the corresponding virtual memory capacity is insufficient during the execution of the modification operation of the second interpretation graph, an expansion operation is performed.

[0110] Among them, the second interpretation graph contains containers such as arrays, sets, dictionaries, and smart pointers. Modification operations on the interpretation graph (such as adding and deleting objects, adding and deleting components, adding object relationships, etc.) can be implemented through these containers, and these containers are also used to store object information in the interpretation graph.

[0111] Figure 4A The structure diagram of an interpretation graph pointer of the interpretation graph processing method of an intelligent agent according to an embodiment of the present invention is schematically shown. Figure 4B A scene graph of virtual memory expansion according to the interpretation graph processing method of an intelligent agent according to an embodiment of the present invention is schematically shown.

[0112] like Figure 4B As shown, the execution of the modification operation on the second interpretation graph depends on the implementation of the containers of the interpretation graph, and these containers need to meet the following requirements: First, whenever these containers need to apply for or release memory, they can use the memory allocator in the interpretation graph to apply for or release memory. When the memory allocator finds that the virtual memory capacity is insufficient, it expands the capacity through the following steps: first, determine a larger new virtual memory capacity based on the virtual memory growth strategy (this strategy can be selected as needed, such as expanding by 1 times); then apply for a virtual memory of the above capacity, or expand the original virtual memory to the above capacity. The new virtual memory must meet the requirements of "the new virtual memory and the original virtual memory, the physical memory pages and shared tags corresponding to the virtual memory pages at the same offset are the same, and the shared tags of the newly expanded pages are set to private", such as Figure 4B The “newly expanded memory page” marked as “private” in the virtual memory of the expanded interpretation graph is shown.

[0113] If the new virtual memory is not expanded from the original virtual memory, the original virtual memory is released. After that, the memory allocator is notified of the new virtual memory base address and the expanded virtual memory capacity.

[0114] Whenever these containers or elements stored in the containers need to use a pointer, it is necessary to distinguish whether the address pointed to by the pointer is within the virtual memory range used by the interpretation graph.

[0115] If it is within the virtual memory range used by the interpretation graph, a relative pointer type should be used. When writing, the relative pointer type records the offset value of the absolute virtual memory address pointed to by the pointer and the memory address where the pointer is located; when reading, the absolute virtual memory address actually pointed to is obtained by adding the memory address where the pointer is located and the offset value recorded when writing. Figure 4A shown.

[0116] If it is not within the virtual memory range used by the interpretation graph, an absolute pointer type should be used. The pointer directly records the virtual memory address it points to, as shown in the following example. Figure 4A shown.

[0117] Due to the requirement in the above steps that "the physical memory pages and shared tags corresponding to the virtual memory pages at the same offset in the new virtual memory and the original virtual memory are the same", and the requirement for the use of "relative pointers and absolute pointers", the data contained in the interpretation graph after expansion and the interpretation graph before expansion (such as relationship tables, component sets, entity (object) sets, etc.) are essentially the same. Therefore, this solution meets the semantic requirement that "after expansion, the virtual memory capacity increases, but the interpretation graph data remains unchanged".

[0118] Through the above steps, the expansion operation of the interpretation graph can be converted into a mapping operation of the underlying virtual memory page, thereby effectively avoiding the movement or copying of all memory data in the interpretation graph virtual memory in the traditional interpretation graph expansion implementation scheme. Therefore, the above steps reduce the time complexity of the interpretation graph expansion operation from O(n) to O(1), where n represents the number of objects in the interpretation graph. This greatly improves the efficiency of a large number of modification operations on the interpretation graph (such as adding and modifying object / component information and adding and deleting relationships) that are widely present in the detection and recognition, streaming incremental processing, cognition, simulation and other modules of the intelligent agent.

[0119] Figure 4C A scene graph of an interpretation graph replication of the interpretation graph processing method of an intelligent agent according to an embodiment of the present invention is schematically shown.

[0120] like Figure 2A-Figure 4C As shown, according to an embodiment of the present invention, in operation S203, the second interpretation graph is processed, and the process further includes performing a copy operation on the second interpretation graph to generate a third interpretation graph, which includes: Applying for a virtual memory of a third interpretation map having the same capacity as the second interpretation map through a preset operating system, and mapping the data content of the second interpretation map to a virtual memory page of the virtual memory of the third interpretation map; Marking all virtual memory pages in the virtual memory of the second interpretation graph and all virtual memory pages in the virtual memory of the third interpretation graph as shared; The memory allocator of the second interpretation graph is copied to the third interpretation graph, and then the base address of the virtual memory of the memory allocator of the third interpretation graph is corrected to the base address of the virtual memory of the third interpretation graph.

[0121] like Figure 2BAs shown, in operation S250, the planner 200 can perform simulation operations. For example, the agent can find that the water cup is placed on the table through the interpretation graph, and then calculate the possibility of the water cup falling from the table based on the proximity between the water cup and the edge of the table to determine whether the water cup is safe. If it is not safe, a safe position of the water cup is screened out through simulation, and finally an action is generated to move the water cup to a safe position. In this simulation process, the interpretation graph may be copied, updated, and destroyed.

[0122] When a new interpretation graph (a copy of the source interpretation graph, also referred to as the third interpretation graph below) needs to be copied from an existing second interpretation graph (i.e., the source interpretation graph), such as Figure 4C As shown, first, a new virtual memory can be applied for as the virtual memory of the third interpretation graph according to the second interpretation graph, and the capacity of the virtual memory of the third interpretation graph can be the same as the virtual memory capacity of the source interpretation graph. The virtual memory of the third interpretation graph must satisfy "the virtual memory and the virtual memory of the source interpretation graph, the virtual memory page at the same offset position corresponds to the same physical memory page, and the sharing mark of all memory pages in the new virtual memory page is set to shared", such as Figure 4C The copy of the source interpretation graph in the new virtual memory shown involves memory page #1, memory page #2, memory page #3, etc., which are all marked as shared.

[0123] Due to the requirement in the above steps that "the virtual memory page at the same offset position in the virtual memory of the source interpretation graph corresponds to the same physical memory page", and the above requirements for the use of "relative pointers and absolute pointers", the data contained in the new interpretation graph and the source interpretation graph (such as relationship tables, component sets, entity (object) sets) are essentially the same. At the same time, with the help of the copy-on-write function of the virtual memory page, after the copy operation, whether the source interpretation graph or the new interpretation graph (a copy of the source interpretation graph) is modified, it is only visible to the interpretation graph itself. Therefore, this solution meets the semantic requirement of "after copying, the source interpretation graph and the new interpretation graph have the same data, but the modification operations on the two do not affect each other."

[0124] Through the above steps, the copy operation of the interpretation graph can be converted into a mapping operation of the underlying virtual memory page, thereby effectively avoiding the copying of all memory data in the virtual memory of the interpretation graph in the traditional interpretation graph copy implementation scheme. And the subsequent modification operation of the source interpretation graph or the new interpretation graph can automatically complete the copying and remapping of the virtual memory page where the modified data is located through the write-time copy function of the virtual memory page, and no additional processing is required for the virtual memory page where the data has not changed. Therefore, the above steps realize the reduction of the time complexity of the interpretation graph copy operation from O(n) to O(1), where n represents the number of objects in the interpretation graph. This greatly improves the efficiency of a large number of interpretation graph copy operations that are widely present in modules such as the simulation of intelligent agents.

[0125] like Figure 2A-Figure 4C As shown, according to an embodiment of the present invention, in operation S203, the second interpretation graph is processed, further comprising: In response to the destroy instruction of the second interpretation graph, the virtual memory of the second interpretation graph is released by a preset operating system.

[0126] like Figure 2B As shown, in operation S270, when the second interpretation graph is no longer used, the second interpretation graph can be destroyed by executing resource cleanup. Specifically, the operating system can be notified by a destruction instruction to release the virtual memory applied for by the second interpretation graph, without the need to cascade release various containers and objects in the interpretation graph as in the conventional technology. The destruction instruction can be a system control instruction for controlling the operating system to execute the release of a specified virtual memory.

[0127] Therefore, by converting the destruction operation of the interpretation graph into the release operation of the underlying memory pages, the time complexity of the destruction operation of the interpretation graph is finally reduced from O(n) to O(1), where n represents the number of objects in the interpretation graph. This greatly reduces the workload of interpreting graph processing and improves processing efficiency.

[0128] It should be noted that, for the above-mentioned modification and copying operations on the second interpretation graph, the operations involving applying for virtual memory can be completely implemented with the help of the operating system API, and can also be implemented through the user mode page fault interrupt processing mechanism provided by the operating system.

[0129] In addition, due to Figure 4A The addresses of the virtual memory range used by the required interpretation graph are shown as relative pointers. Therefore, even if the interpretation graph obtained in the expansion and copying operations of the second interpretation graph is not the same as the virtual memory base address of the source interpretation graph, these relative pointers will also obtain the correct memory address after reading, so these pointers do not need to be updated.

[0130] Moreover, since the above-mentioned copy and destroy operations only involve the copy and destruction of page tables, and do not involve the copy and modification of the content stored in virtual memory, the time complexity of the copy and destroy operations depends on the page table size, which is inversely proportional to the memory page size. The current mainstream memory page size is 4k, and some operating systems support the specified page table size when applying for memory (such as MAP_HUGE_2MB and MAP_HUGE_1GB on Linux). Therefore, compared with the conventional solution of cascading copy / destruction interpretation graphs, the performance of this solution can be improved by at least 4k times. Under current hardware conditions and practices, this complexity can be regarded as a constant, that is, O(1).

[0131] The intelligent agent interpretation graph processing method provided by the embodiment of the present invention can at least partially solve the problems of high complexity, low processing efficiency and poor processing accuracy in the interpretation graph processing process in the related art, and thus can achieve at least one of the following technical effects: (1) The copy and destroy operations of the interpretation graph can be transformed into the management and release operations of the underlying memory pages. At the same time, with the help of the write-time copy of the virtual memory pages, the large amount of memory data copying caused by the interpretation graph copy operation is eliminated. Finally, the time complexity of the interpretation graph copy and destroy operations is reduced from O(n) to O(1), where n represents the number of objects in the interpretation graph.

[0132] (2) The concept of components is introduced through the representation rules of objects and their relationships. Each component can represent a function or attribute unit, and each object can be composed of several components. The combination of components can represent some multifunctional, complex, and new objects that cannot be anticipated in advance during design (n components can be combined into 2^n different types of objects). The types of objects that can be represented grow exponentially with the number of component types. For users of the interpretation graph (such as planners), when their internal logic depends only on components, they can be easily generalized to new objects that appear later. The component-centric memory layout ensures that component data of the same type is continuous in memory, with a high cache hit rate during traversal, and facilitates parallel processing using multi-core CPUs or GPUs.

[0133] (3) The relationship table is further introduced through the representation rules of the relationship between objects. The relationship table can record the {source-relationship descriptor-target} three-tuple relationship pairs, and the relationship descriptors are divided into four types: 1 to 1, 1 to N, N to 1, and N to N. Therefore, the relationship table can support the following operations with O(1) complexity: relationship conflict detection, querying the relationships (forward / reverse) owned by an object, and querying the source set of a relationship (forward / reverse).

[0134] (4) Through streaming incremental parsing, the calculation scope can be narrowed from all objects and component sets to the objects and component sets that have changed, further reducing the complexity of processing or updating operations on the interpretation graph.

[0135] Therefore, the interpretation graph processing method of the embodiment of the present invention can make the interpretation graph of the intelligent body more accurately and comprehensively reflect the objective information of the real environment, achieve the effect of flexibly and efficiently representing and processing the interpretation graph, greatly reduce the complexity of the interpretation graph processing operation, and avoid the high computational cost caused by the cascade creation, copying, and destruction of a large number of small objects caused by interpretation graph processing operations (such as copying and destruction, etc.); and can accurately describe newly emerging multifunctional or more complex new objects, avoid inconsistencies such as missing or extra attributes, and achieve more efficient batch processing, greatly reducing the computational cost.

[0136] Based on the above-mentioned intelligent agent interpretation graph processing method, the present invention also provides an intelligent agent interpretation graph processing device. Figure 5 The device is described in detail.

[0137] Figure 5 The structure block diagram of the interpretation graph processing device of an intelligent agent according to an embodiment of the present invention is schematically shown.

[0138] like Figure 5 As shown, the intelligent agent's interpretation graph processing device 500 of this embodiment includes a resource initialization module 510, an interpretation graph representation module 520 and an interpretation graph processing module 530.

[0139] The resource initialization module 510 is used to obtain the first interpretation graph generated by the resource initialization operation of the agent. In one embodiment, the resource initialization module 510 can be used to perform the operation S201 described above, which will not be repeated here.

[0140] The interpretation diagram representation module 520 is used to update the first interpretation diagram according to the preset interpretation diagram representation rule and the detected environment detection information to generate a second interpretation diagram, wherein the preset interpretation diagram representation rule is a representation rule for the object information and the relationship between the objects in the second interpretation diagram. In one embodiment, the interpretation diagram representation module 520 can be used to perform the operation S202 described above, which will not be repeated here.

[0141] The interpretation graph processing module 530 is used to process the second interpretation graph. In one embodiment, the interpretation graph processing module 530 can be used to perform the operation S203 described above, which will not be described in detail here.

[0142] According to an embodiment of the present invention, any multiple modules among the resource initialization module 510, the interpretation diagram representation module 520 and the interpretation diagram processing module 530 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present invention, at least one of the resource initialization module 510, the interpretation diagram representation module 520 and the interpretation diagram processing module 530 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware and firmware or in any appropriate combination of any of them. Alternatively, at least one of the resource initialization module 510, the interpretation diagram representation module 520 and the interpretation diagram processing module 530 can be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding function can be executed.

[0143] Figure 6 A block diagram of an electronic device suitable for implementing a method for processing an interpretation graph of an intelligent agent according to an embodiment of the present invention is schematically shown.

[0144] The electronic device provided by an embodiment of the present invention includes one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the above-mentioned intelligent agent's interpretation graph processing method.

[0145] like Figure 6 As shown, the electronic device 600 according to an embodiment of the present invention includes a processor 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 to a random access memory (RAM) 603. The processor 601 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 601 may also include an onboard memory for caching purposes. The processor 601 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0146] In RAM 603, various programs and data required for the operation of electronic device 600 are stored. Processor 601, ROM 602 and RAM 603 are connected to each other via bus 604. Processor 601 performs various operations of the method flow according to the embodiment of the present invention by executing the program in ROM 602 and / or RAM 603. It should be noted that the program can also be stored in one or more memories other than ROM 602 and RAM 603. Processor 601 can also perform various operations of the method flow according to the embodiment of the present invention by executing the program stored in the one or more memories.

[0147] According to an embodiment of the present invention, the electronic device 600 may further include an input / output (I / O) interface 605, which is also connected to the bus 604. The electronic device 600 may further include one or more of the following components connected to the I / O interface 605: an input portion 606 including a keyboard, a mouse, etc.; an output portion 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 608 including a hard disk, etc.; and a communication portion 609 including a network interface card such as a LAN card, a modem, etc. The communication portion 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage portion 608 as needed.

[0148] The present invention also provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to execute the above-mentioned intelligent agent interpretation graph processing method.

[0149] The computer-readable storage medium may be included in the device / apparatus / system described in the above embodiment; or it may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present invention is implemented.

[0150] According to an embodiment of the present invention, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present invention, the computer-readable storage medium may include the ROM 602 and / or RAM 603 described above and / or one or more memories other than ROM 602 and RAM 603.

[0151] An embodiment of the present invention also includes a computer program product, which includes a computer program, and when the computer program is executed by a processor, the above-mentioned intelligent agent interpretation graph processing method is implemented.

[0152] The computer program includes program codes for executing the method shown in the flow chart. When the computer program product is run in a computer system, the program codes are used to enable the computer system to implement the method provided by the embodiment of the present invention.

[0153] The computer program executes the above functions defined in the system / device of the embodiment of the present invention when it is executed by the processor 601. According to the embodiment of the present invention, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0154] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 609, and / or installed from a removable medium 611. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0155] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the processor 601, the above functions defined in the system of the embodiment of the present invention are performed. According to the embodiment of the present invention, the system, device, means, module, unit, etc. described above can be implemented by a computer program module.

[0156] According to an embodiment of the present invention, the program code for executing the computer program provided by the embodiment of the present invention can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0157] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0158] It will be appreciated by those skilled in the art that the features described in the various embodiments and / or claims of the present invention may be combined and / or combined in various ways, even if such combinations and / or combinations are not explicitly described in the present invention. In particular, the features described in the various embodiments and / or claims of the present invention may be combined and / or combined in various ways without departing from the spirit and teachings of the present invention. All of these combinations and / or combinations fall within the scope of the present invention.

[0159] The embodiments of the present invention are described above. However, these embodiments are only for the purpose of illustration, and are not intended to limit the scope of the present invention. Although each embodiment is described above, it does not mean that the measures in each embodiment cannot be used in combination. The scope of the present invention is defined by the attached claims and their equivalents. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. A method for processing an interpretation graph of an intelligent agent, characterized in that: include: Obtaining a first interpretation graph generated by a resource initialization operation of the agent; updating the first interpretation map according to a preset interpretation map representation rule and detected environment detection information to generate a second interpretation map, wherein the preset interpretation map representation rule is a representation rule for object information and relationships between objects in the second interpretation map; and Processing is performed on the second interpretation graph.

2. The method according to claim 1, characterized in that: The first interpretation graph generated by the resource initialization operation of obtaining the agent includes: Apply for corresponding virtual memory through the preset operating system; Marking all virtual memory pages in the virtual memory as private; The memory allocator is initialized according to the base address and capacity of the virtual memory after marking is completed to obtain the first interpretation graph.

3. The method according to claim 1, characterized in that In the step of updating the first interpretation map according to the preset interpretation map representation rule and the detected environment detection information to generate the second interpretation map, the method includes: updating the first interpretation map according to the object-centric memory storage layout in the preset interpretation map representation rule and the object information of the environmental detection information to generate the second interpretation map; and / or The first interpretation graph is updated according to the component-centric memory storage layout method in the preset interpretation graph representation rule and the object information of the environmental detection information to generate the second interpretation graph.

4. The method according to claim 1 or 3, characterized in that: The first interpretation map is updated according to the preset interpretation map representation rule and the detected environment detection information to generate the second interpretation map, further comprising: The first interpretation graph is updated according to the representation method of the relationship between objects in the preset interpretation graph representation rule and the relationship information between objects in the environment detection information to generate the second interpretation graph.

5. The method according to claim 1, characterized in that The step of performing processing on the second interpretation graph includes: A modification operation is performed on the second interpretation graph according to a preset streaming incremental processing rule.

6. The method according to claim 1, characterized in that The step of performing processing on the second interpretation graph includes: When it is found that the corresponding virtual memory capacity is insufficient during the execution of the modification operation of the second interpretation graph, an expansion operation is performed.

7. The method according to claim 1, characterized in that The processing of the second interpretation graph further includes performing a copy operation on the second interpretation graph to generate a third interpretation graph, which includes: Applying for a virtual memory of a third interpretation graph having the same capacity as the second interpretation graph through a preset operating system, and mapping the data content of the second interpretation graph to a virtual memory page of the virtual memory of the third interpretation graph; marking all virtual memory pages in the virtual memory of the second interpretation graph and all virtual memory pages in the virtual memory of the third interpretation graph as shared; The memory allocator of the second interpretation graph is copied to the third interpretation graph, and then the base address of the virtual memory of the memory allocator of the third interpretation graph is corrected to the base address of the virtual memory of the third interpretation graph.

8. The method according to claim 1, characterized in that: In the performing of processing on the second interpretation graph, the method further includes: In response to the destroy instruction of the second interpretation graph, the virtual memory of the second interpretation graph is released through a preset operating system.

9. An intelligent agent interpretation graph processing device, characterized in that: include: A resource initialization module, used to obtain a first interpretation graph generated by a resource initialization operation of the agent; an interpretation diagram representation module, configured to update the first interpretation diagram according to a preset interpretation diagram representation rule and detected environment detection information to generate a second interpretation diagram, wherein the preset interpretation diagram representation rule is a representation rule for object information and relationships between objects in the second interpretation diagram; and The interpretation graph processing module is used to process the second interpretation graph.

10. An electronic device comprising: one or more processors; a memory for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors execute the method according to any one of claims 1 to 8.

11. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to execute the method according to any one of claims 1 to 8.

12. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images