Multimodal data storage control device, method, intelligent agent and equipment for intelligent agent

Abstract

The embodiment of the present invention provides a multimodal data storage control device of an intelligent agent, which can be applied to the field of artificial intelligence technology. The multimodal data storage control device of the intelligent agent includes a data receiving layer, a data service layer and a storage control layer. The data receiving layer is used to receive upstream input data, identify the upstream input data, and define the business service logic that matches the data type of the upstream input data; the data service layer is used to perform processing services on the upstream input data received by the data receiving layer according to the business service logic; the storage control layer is used to perform data storage control services according to the processing services of the data service layer. The embodiment of the present invention also provides an intelligent agent, a multimodal data storage control method, device, storage medium and program product of the intelligent agent.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, specifically to the field of intelligent body data processing technology, and more specifically to a multimodal data storage control device, method, intelligent body, equipment, medium and product of an intelligent body. Background Art

[0002] Artificial Intelligence (AI) is an important driving force for the new round of scientific and technological revolution and industrial transformation. It is a new key technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. As an important part of intelligent science, artificial intelligence attempts to understand the essence of intelligence and produce a new intelligent machine (i.e., intelligent agent) that can respond in a similar way to human intelligence.

[0003] In the field of artificial intelligence technology, different application scenarios will rely on different multimodal data obtained by intelligent agents to realize corresponding human-computer or even machine-machine interaction functions, which refers to data with multiple forms of expression collected through different methods or perspectives. Among them, multimodality refers to the way of expressing, communicating and understanding information in a variety of different forms or perception channels, usually including vision, hearing, text, touch and other sensory input and output methods. Correspondingly, multimodal data usually covers multiple types such as text, images, audio, video, etc., with different structures, semantics and representation methods. Multimodal data is widely present in the real world. For example, in autonomous driving systems, camera images, lidar data and the vehicle's own sensor data together constitute a multimodal data set. ‌

[0004] For intelligent agents, multimodal data must exist in the intelligent agent for a long time, just like the memory of real people, that is, to form the memory of the intelligent agent. However, in reality, considering that intelligent agents need to interact with the outside world continuously, most intelligent agents usually have major defects in the processing and storage of multimodal data. For example, they can only process single-modal data, and the pure natural language form and unstructured memory data in the multimodal environment are usually incomplete, poor in expression, and difficult to parse. This will further cause the question-answering system based on memory data expressed in natural language to have hallucinations or wrong answers, which seriously affects the intelligence level of embodied intelligence. Summary of the invention

[0005] In view of at least one of the above problems, embodiments of the present invention aim to provide a multimodal data storage control device, method, intelligent agent, equipment, medium and product for an intelligent agent, so as to enable a data processing architecture for modeling, collection and storage of the intelligent agent's memory data, a comprehensive document database, a vector database and other functions, to realize the storage of the intelligent agent's multimodal perception and cognitive data, and to support various real-time question-and-answer scenarios with technologies such as retrieval enhancement, so that the embodied intelligence can process multimodal data more efficiently and smoothly, and enable the intelligent agent to accurately store and query multimodal data according to different interaction scenarios, thereby achieving a higher level of intelligence.

[0006] One aspect of an embodiment of the present invention provides a multimodal data storage control device of an intelligent agent, which includes a data receiving layer, a data service layer and a storage control layer. The data receiving layer is used to receive upstream input data, identify the upstream input data, and define business service logic that matches the data type of the upstream input data; the data service layer is used to perform processing services on the upstream input data received from the data receiving layer according to the business service logic; and the storage control layer is used to perform data storage control services according to the processing services of the data service layer.

[0007] According to an embodiment of the present invention, the data service layer includes a data writing module. The data writing module is used to perform writing services on received upstream input data according to different data types according to the business service logic.

[0008] According to an embodiment of the present invention, the data writing module includes a difference generation service unit and a key-value pair data service unit. The difference generation service unit is used to generate triple difference data based on the cognitive understanding data of the upstream input data to capture and describe the amount of change before and after the data; the key-value pair data service unit is used to perform key-value pair data conversion services on the triple difference data and the cognitive understanding data of the upstream input data, the self-action data and the visual input data.

[0009] According to an embodiment of the present invention, the data writing module further includes a text vector service unit and a vector data service unit. The text vector service unit is used to perform data vectorization services on triple difference data, cognitive understanding data, and self-action data; and the vector data service unit is used to perform vector data writing on vectorized data generated by the data vectorization service.

[0010] According to an embodiment of the present invention, the data service layer further includes a data retrieval module. The data retrieval module is used to call the storage control layer to perform retrieval services according to different data types in response to a data retrieval request.

[0011] According to an embodiment of the present invention, the data retrieval module includes a difference data retrieval service unit, a cognitive data retrieval service unit, an action data retrieval service unit and a visual data retrieval service unit. The difference data retrieval service unit is used to query the triple difference data stored by the data storage control service; the cognitive data retrieval service unit is used to query the cognitive understanding data stored by the data storage control service; the action data retrieval service unit is used to query the self-action data stored by the data storage control service; and the visual data retrieval service unit is used to query the visual data stored by the data storage control service.

[0012] According to an embodiment of the present invention, the data retrieval module further includes a vector retrieval service unit. The vector retrieval service unit is used to perform vectorized retrieval on the vectorized data stored after the vector data service unit performs vector data writing.

[0013] According to an embodiment of the present invention, the storage control layer includes a plurality of storage control service units, and each of the plurality of storage control service units is used to perform storage control services on a matching database.

[0014] Another aspect of an embodiment of the present invention provides an intelligent agent, which includes the multimodal data storage control device of the intelligent agent mentioned above.

[0015] Another aspect of an embodiment of the present invention provides a multimodal data storage control method for an intelligent body, which is implemented based on the multimodal data storage control device of the above-mentioned intelligent body, and includes: identifying received upstream input data and defining business service logic that matches the data type of the upstream input data; performing processing services on the received upstream input data according to the business service logic; and performing data storage control services according to the processing services.

[0016] Another aspect of an embodiment of the present invention provides an electronic device, comprising one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the above-mentioned multimodal data storage control method of the intelligent agent.

[0017] Another aspect of an embodiment of the present invention provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to execute the above-mentioned multimodal data storage control method of the intelligent agent.

[0018] Another aspect of an embodiment of the present invention provides a computer program product, including a computer program, which implements the above-mentioned multimodal data storage control method of the intelligent agent when executed by a processor.

[0019] The multimodal data storage control device for an intelligent body provided by an embodiment of the present invention can at least partially solve the problem of low intelligence level in the process of embodied intelligence executing processing on intelligent body memory data in the related art, and thus can achieve at least one of the following technical effects:

[0020] Based on the multimodal data storage control device of the embodiment of the present invention, a data processing architecture with functions such as modeling, collection and storage of intelligent body memory data, integrated document database, vector database, etc. is constructed to realize the storage of multimodal data such as multimodal perception and cognitive data of the intelligent body, and support various real-time question and answer scenarios with technologies such as retrieval enhancement, so that the embodied intelligence can process multimodal data more efficiently and smoothly, and enable the intelligent body to accurately realize the storage and query of multimodal data according to different interaction scenarios, thereby achieving a higher level of intelligence.

[0021] It should be understood that the above general description and the following detailed description are merely exemplary and illustrative and are not intended to limit the scope of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The above contents and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0023] Figure 1 A schematic diagram shows a structural block diagram of a multimodal data storage control device for an intelligent agent according to an embodiment of the present invention;

[0024] Figure 2A Schematically shows a structural block diagram of a data writing application scenario of a multimodal data storage control device for an intelligent agent according to an embodiment of the present invention;

[0025] Figure 2B Schematically shows a flow chart of a data writing application scenario of a multimodal data storage control device of an intelligent agent according to an embodiment of the present invention;

[0026] Figure 2C Schematically shows a flow chart of another data writing application scenario of the multimodal data storage control device of an intelligent agent according to an embodiment of the present invention;

[0027] Figure 2D Schematically shows a flow chart of another data writing application scenario of the multimodal data storage control device of an intelligent agent according to an embodiment of the present invention;

[0028] Figure 3 Schematically shows a structural block diagram of a data retrieval application scenario of a multimodal data storage control device for an intelligent agent according to an embodiment of the present invention;

[0029] Figure 4A schematic diagram showing an application scenario of a multimodal data storage control method, apparatus, device, medium and program product of an intelligent agent according to an embodiment of the present invention;

[0030] Figure 5 Schematically shows a flow chart of a multimodal data storage control method for an intelligent agent according to an embodiment of the present invention;

[0031] Figure 6 A block diagram of an electronic device suitable for implementing a multimodal data storage control method for an intelligent agent according to an embodiment of the present invention is schematically shown.

[0032] The above-mentioned drawings are part of the specification of the embodiments of the present invention, which illustrate exemplary embodiments of the present invention. The attached drawings and the description of the specification are used together to illustrate the principles of the embodiments of the present invention. It should be understood that the above general description of the drawings and the following specific implementations are only exemplary and illustrative, and they cannot limit the scope of the present invention. DETAILED DESCRIPTION

[0033] In order to make the objectives, technical solutions and advantages of the embodiments of the present invention more clearly understood, the spirit of the contents disclosed by the present invention will be clearly explained with the accompanying drawings and detailed descriptions below. After understanding the embodiments of the contents of the present invention, any technician in the relevant technical field can change and modify the techniques taught by the contents of the present invention without departing from the spirit and scope of the contents of the present invention.

[0034] The exemplary embodiments and descriptions of the present invention are used to explain the present invention, but are not intended to limit the present invention. In addition, elements / components with the same or similar reference numerals used in the drawings and embodiments are used to represent the same or similar parts.

[0035] The terms “first”, “second”, etc. used in the present invention do not particularly refer to an order or sequence, nor are they used to limit the present invention. They are only used to distinguish elements or operations described with the same technical terms.

[0036] The directional terms used in the present invention, such as up, down, left, right, front or back, etc., are only used to refer to the directions of the drawings. Therefore, the directional terms used are used to illustrate and not to limit the present invention.

[0037] The words “include,” “including,” “have,” “contain,” etc. used in the present invention are open-ended terms, meaning including but not limited to.

[0038] The term "and / or" used in the present invention includes any or all combinations of the items mentioned.

[0039] Regarding the present invention, "plurality" includes "two" and "more than two"; regarding the present invention, "plurality of groups" includes "two groups" and "more than two groups".

[0040] The terms "substantially" and "approximately" used in the present invention are used to modify any quantity or error that may vary slightly, but these slight changes or errors do not change their essence. Generally speaking, the range of slight changes or errors modified by such terms may be 20% in some embodiments, 10% in some embodiments, 5% in some embodiments, or other values. Those skilled in the art should understand that the aforementioned values ​​can be adjusted according to actual needs and are not limited thereto.

[0041] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0042] In the case of using expressions such as "at least one of A, B, and C, etc.", it should generally be interpreted in accordance with the meaning of the expression generally understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but not be limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or a system having A, B, C, etc.). In the case of using expressions such as "at least one of A, B, or C, etc.", it should generally be interpreted in accordance with the meaning of the expression generally understood by those skilled in the art (for example, "a system having at least one of A, B, or C" should include but not be limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or a system having A, B, C, etc.). Those skilled in the art should also understand that any transitional conjunctions and / or phrases that substantially represent two or more optional items, whether in the specification, claims, or drawings, should be understood to give the possibility of including one of these items, either of these items, or both of these items. For example, the phrase "A or B" should be understood to include the possibilities of "A" or "B", or "A and B".

[0043] In the field of artificial intelligence technology, the memory of an intelligent agent is a relatively abstract concept. Generally speaking, it can be considered that the memory of an intelligent agent is coherent, just like people in the real world. The time series of all the events that an intelligent agent experiences (everything it sees, hears, and does) during its life cycle can form the memory of the intelligent agent.

[0044] Memory is a data abstraction and data organization mode from the perspective of the intelligent agent. It is different from the session data or module at the system level and the log or data information at the application level. It is the integration of various data of various components. The memory of the intelligent agent will not be lost due to system restart, but will be maintained throughout its life cycle. The life cycle of the intelligent agent and the continuous operation of the system are two different things and have no direct relationship. Among them, the memory of the intelligent agent can also be divided according to the identity ID of the intelligent agent. Even if different intelligent agents are running on the same physical system, their memory data is isolated from each other.

[0045] Multimodal AI refers to an artificial intelligence system that can process and understand data from multiple modalities at the same time. Modality usually refers to different forms of information, such as text, images, audio, video, etc. By integrating information from these modalities, multimodal AI can more accurately understand the complex real world, thereby achieving a higher level of intelligence.

[0046] In multimodal AI, each modal data (such as text data, image data, audio data, and video data, etc.) has its own unique characteristics and information expression methods.

[0047] Text data is one of the most common data types and contains written expressions in natural language. Text can be words, sentences, paragraphs, etc., and is usually used to describe, explain, or convey information. For example, news reports, social media posts, and user comments are all instances of text modality.

[0048] Image data is static visual information, containing rich features such as color, shape, and texture. Image modality is usually used for tasks such as object recognition, scene understanding, and image generation. For example, face recognition, road sign recognition in autonomous driving, etc.

[0049] Audio data includes sound signals such as human speech, music, and environmental sounds. Audio modalities can be used for applications such as speech recognition, sentiment analysis, and music recommendations. For example, voice assistants perform tasks by recognizing the user's voice commands.

[0050] Video data is a combination of dynamic visual and audio information, including information in the temporal dimension. Video modality can be used for tasks such as behavior recognition, video summarization, and sentiment analysis. For example, surveillance systems identify suspicious behavior by analyzing video streams.

[0051] For business interaction scenarios of embodied intelligence, such as the interactive question-and-answer process, the intelligent experience will involve time-related information, such as "What did you eat for dinner yesterday", which is similar to the situation where humans retrieve relevant information from their brain memory. Therefore, it is necessary to store and record the perception and cognitive data of embodied intelligence during its life cycle and support corresponding retrieval functions.

[0052] Specifically, corresponding question-and-answer scenarios can be generated to recall relevant Parse Graph (PG), Diff, and action-related data, such as: "Were there any tomatoes on the table at 3 pm yesterday?", "How many red blocks were on the table at 3 pm yesterday?", etc. You can also inquire about changes in the hunger level of the agent. In addition, humans can also ask the agent "Why is the tomato on the ground?", hoping to save the state "tomato:IsOn:floor" and the state before this state.

[0053] For the processing and storage of multimodal data of intelligent agents, the intelligent agent receives data input such as video and audio, and continuously interacts with the environment. At present, most data processing systems in the industry usually process single-type and single-modal data. For example, existing platforms such as TongOS and TongAI usually support joint interpretation and data storage of interpretation graphs. However, for the representation of intelligent agent memory, there are currently few engineering practices for modeling intelligent agent memory. Most of the existing ones are time series data records in the form of natural language (Generative Agents). In a multimodal environment, pure natural language form and unstructured memory data are incomplete, with high flexibility, but poor expressiveness and high difficulty in parsing. In addition, in terms of data accuracy and interpretability, question-answering systems based on memory data expressed in natural language and large language models have problems such as hallucinations, wrong answers, and poor interpretability. Moreover, in terms of data scalability and engineering implementation, the engineering of existing intelligent agent data storage is generally poor, mostly used for scientific research and local prototype development, and data storage is usually only for a single function and only in memory, making it difficult for the system to expand horizontally.

[0054] In view of at least one of the above problems, embodiments of the present invention aim to provide a multimodal data storage control device, method, intelligent agent, equipment, medium and product for an intelligent agent, so as to enable a data processing architecture for modeling, collection and storage of the intelligent agent's memory data, a comprehensive document database, a vector database and other functions, to realize the storage of the intelligent agent's multimodal perception and cognitive data, and to support various real-time question-and-answer scenarios with technologies such as retrieval enhancement, so that the embodied intelligence can process multimodal data more efficiently and smoothly, and enable the intelligent agent to accurately store and query multimodal data according to different interaction scenarios, thereby achieving a higher level of intelligence.

[0055] The following will be based on the above scenario. Figure 1~Figure 3 The multimodal data storage control device of the intelligent agent of the disclosed embodiment is described in detail.

[0056] Figure 1 The structure block diagram of the multimodal data storage control device of an intelligent agent according to an embodiment of the present invention is schematically shown.

[0057] like Figure 1 As shown, one aspect of an embodiment of the present invention provides a multimodal data storage control device 100 of an intelligent agent, which includes a data receiving layer 110, a data service layer 120 and a storage control layer 130.

[0058] The data receiving layer 110 is used to receive upstream input data, identify the upstream input data, and define business service logic that matches the data type of the upstream input data;

[0059] The data service layer 120 is used to perform processing services on the upstream input data received from the data receiving layer 110 according to the business service logic;

[0060] The storage control layer 130 is used to perform data storage control services according to the processing services of the data service layer 120 .

[0061] like Figure 1 As shown, the multimodal data storage control device 100 of the intelligent agent according to the embodiment of the present invention can be implemented based on a storage and retrieval architecture.

[0062] The multimodal data storage control device 100 mainly includes a data receiving layer 110, a data service layer 120 and a storage control layer 130, which can be understood as corresponding computer data processing modules and are used as the basic architecture of the multimodal data storage control device.

[0063] like Figure 1 As shown, the entire multimodal data storage control device 100 as a whole publishes the top-level microservice (such as ProxyServer) to the outside, and each data path can correspond to an independent vertical service, such as Figure 1 Shown are services 1 to 4. Among them, the services internally rely on and share horizontal modules and components.

[0064] For each module, its corresponding interaction relationship can be presented based on the service implementation process of data storage control.

[0065] The data receiving layer 110 can connect to the upstream data source module, expose the top-level microservices to the data source module, complete data reception and identification of the data type of the received data. Furthermore, the data receiving layer 110 can call the functional modules and services of the data service layer 120 of the next layer to implement the logic of specific data types and writing processes.

[0066] Figure 2A A structural block diagram schematically shows a data writing application scenario of a multimodal data storage control device of an intelligent agent according to an embodiment of the present invention.

[0067] like Figure 2A As shown, the upstream input data may be the upstream data provided by the data source module 200, and specifically may be the multimodal data actually obtained by the embodied intelligence in the Unreal Engine (or other world models), such as the image or video data "seen" by the visual sensor. The RGB, depth map and other data corresponding to these data may be processed by DAG (Directed Acyclic Graph), which may specifically include multiple modules such as CV, NLP, and PL, and may eventually enter the above-mentioned multimodal data storage control device 100 of the embodiment of the present invention in the form of an interpretation graph. Among them, the data source module 200 may be implemented according to different intelligent agent platforms (such as the TongOS platform), that is, the data receiving layer 110 may be connected to the intelligent agent platform, receive the upstream data released by the platform, and use the top-level microservice (such as Proxy) to process and distribute the data, and then the downstream specific data processing logic may be called to realize data storage control.

[0068] In the embodiment of the present invention, upstream input data of different data types correspond to data acquired by the agent in different application scenarios, such as Figure 2A Data types 1 to 3 are shown. For example, the upstream input data may include visual input data, self-motion data, triple difference data, and cognitive understanding data.

[0069] CVPG data can be a symbolic expression of the world that the agent objectively sees based on visual input. In other words, the image of the world seen by the agent is processed to obtain an interpretation map. These interpretation maps can represent the processing results of the agent on the world scene, including the identification, attributes and relative relationships of various objects in the scene, that is, visual input data; Belief-PG data can be a symbolic expression of the agent's cognition and understanding of the objective world, which can represent the agent's subjective understanding of the world, which may include self-perception but not in line with objective facts, that is, cognitive understanding data; U-action data can be a description of the actions and tasks performed by the agent itself, that is, self-action data; Status-change data can be a difference expression of the above-mentioned visual input data (CVPG). In an embodiment of the present invention, the format of the difference expression data can be triple data (SPO) and contain the required context information, that is, triple difference data. Among them, for the record of the agent action stream, CVPG-actions can also be included, which can be the description data of the actions that the agent sees humans or other agents.

[0070] For example, suppose the agent is in a set room with a table, a chair, and a cup on the table. The agent guesses that there is water in the cup and decides to take the cup to drink water. Among them, the visual input data can include elements such as the room, table, chair, cup, and the agent itself, as well as information such as the number and appearance (such as shape, color, etc.) of these objects. The cognitive understanding data can include elements such as the room, table, chair, cup, and the agent itself, and also include the data that the agent "thinks", such as the cognition that the agent speculates that there is water in the cup (there may be no water in the actual cup). The self-action data can include descriptions of a series of actions such as "'take' the cup" and "'drink' water". Among them, for the self-action data, its specific format is relatively complex, and these actions may have a hierarchical structure and a sequential relationship, etc., which will not be described in detail. In addition, because there are no other agents in the scene, there is no CVPG-Actions data. If there are other agents, the information describing the recognition and cognition of the actions of other agents by the agent can be used as its data content.

[0071] Furthermore, for the above-mentioned different types of multimodal data, the corresponding data expressions may also be different.

[0072] Among them, for visual input data and cognitive understanding data, the specific expressions can be as follows:

[0073] { "beliefPgId": "test_belief_pg",

[0074] "Table_BuildingBlocks_zone40":

[0075] {

[0076] "zoneIndex": [ 40 ],

[0077] "coordinateX": [ 50.339295196533 ],

[0078] "CVReliability": [ 1 ],

[0079] "isVirtual": [ false ],

[0080] "coordinateZ": [ 41.831382751465 ],

[0081] "rdf:type": [ "platformZone" ],

[0082] "coordinateY": [ 91.618560791016 ]

[0083] },

[0084] "face_0": {

[0085] "isPartOf": [ "head_0" ],

[0086] "hasPart": [ "ear_right_0", "nose_0", "eye_left_0", "ear_left_0", "cheek_right_0", "chin_0","cheek_left_0", "forehead_0", "mouth_0", "eye_right_0"],

[0087] "maxVertexX": [ 0 ],

[0088] "maxVertexY": [ 0 ],

[0089] "minVertexX": [ 0 ],

[0090] "minVertexZ": [ 0 ],

[0091] "isVirtual": [ false ],

[0092] "minVertexY": [ 0 ],

[0093] "rdf:type": [ "face" ],

[0094] "maxVertexZ": [ 0 ]

[0095] },

[0096] "Table_BuildingBlocks_zone14": {

[0097] "zoneIndex": [ 14 ],

[0098] "coordinateX": [ 76.008612823486 ],

[0099] "CVReliability": [ 1 ],

[0100] "isVirtual": [ false ],

[0101] "coordinateZ": [ 41.831382751465 ],

[0102] "rdf:type": [ "platformZone" ],

[0103] "coordinateY": [ 72.530044555664 ]

[0104] },

[0105] "Table_BuildingBlocks_zone4": {

[0106] "zoneIndex": [ 4 ],

[0107] "coordinateX": [ 76.008612823486 ],

[0108] "CVReliability": [ 1 ],

[0109] "isVirtual": [ false ],

[0110] "coordinateZ": [ 41.831382751465 ],

[0111] "rdf:type": [ "platformZone" ],

[0112] "coordinateY": [ 66.167205810547 ]

[0113] },…

[0114] }

[0115] Among them, for the self-action data, the specific expression can be as follows:

[0116] {

[0117] "belefPgId": "c22722d5-7e98-4bc2-98c0-6a45c861c1e2",

[0118] "u_info": {

[0119] "unique_id": "64310bfe-368c-42d7-813e-fca142079eda",

[0120] "parent_id": null,

[0121] "u_id": "relocate111111ec843da9e93dc92099",

[0122] "args": {

[0123] "properties": {

[0124] "obj2pick": "tomato_123",

[0125] "place2put": {

[0126] "properties": {

[0127] "object_id": "table_345",

[0128] "direction": "common_base#isOn"

[0129] }

[0130] }

[0131] }

[0132] },

[0133] "change_time": 1703830448389839,

[0134] "start_time": 1703830448389839,

[0135] "status_code": "EXECUTING"

[0136] }

[0137] }

[0138] Among them, for the triple difference data, the specific expression can be as follows: [

[0140] {

[0141] "change_time": "2021-01-01T00:00:00Z",

[0142] "subject": {

[0143] "rdf:type": "CommonBase#BuildingBlock",

[0144] "value": "toy_cube_1"

[0145] },

[0146] "predicate": {

[0147] "rdf:type": "CommonBase#isOn"

[0148] },

[0149] "object": {

[0150] "before": null,

[0151] "after": {

[0152] "value": "floor",

[0153] "rdf:type": "CommonBase#FloorBoard"

[0154] }

[0155] }

[0156] },

[0157] {

[0158] "change_time": "2021-01-01T00:00:00Z",

[0159] "subject": {

[0160] "rdf:type": "CommonBase#BuildingBlock",

[0161] "value": "toy_cube_1"

[0162] },

[0163] "predicate": {

[0164] "rdf:type": "CommonBase#isIn"

[0165] },

[0166] "object": {

[0167] "before": {

[0168] "value": "Box_345",

[0169] "rdf:type": "CommonBase#ToyBox"

[0170] },

[0171] "after": null

[0172] }

[0173] },

[0174] {

[0175] "change_time": "2021-01-01T00:00:00Z",

[0176] "subject": {

[0177] "rdf:type": "CommonBase#PaperBall",

[0178] "value": "paperball_567"

[0179] },

[0180] "predicate": {

[0181] "rdf:type": "CommonBase#hasColour"

[0182] },

[0183] "object": {

[0184] "before": {

[0185] "value": "red",

[0186] "rdf:type": null

[0187] },

[0188] "after": {

[0189] "value": "green",

[0190] "rdf:type": null

[0191] }

[0192] }

[0193] },

[0194] {

[0195] "change_time": "2021-01-01T00:00:00Z",

[0196] "subject": {

[0197] "rdf:type": "CommonBase#RobotAIBaby",

[0198] "value": "aibaby_brother"

[0199] },

[0200] "predicate": {

[0201] "rdf:type": "EXISTENCE"

[0202] },

[0203] "object": {

[0204] "before": null,

[0205] "after": {

[0206] "value": true,

[0207] "ontology": null

[0208] }

[0209] }

[0210] } ]

[0212] Therefore, the data source module as the data publishing source can input the above multimodal data as upstream input data to the data receiving layer 110. The data receiving layer 110 can perform preliminary analysis, data format conversion, data extraction and distribution and other processing services on the received multimodal data to realize the recognition of these upstream input data and obtain the corresponding data type. Furthermore, the data receiving layer 110 can also match different business service logics for different multimodal data according to the different data types of these upstream input data (such as cognitive understanding data and visual input data, etc.).

[0213] Business service logic is the subsequent data processing service process defined for different types of data in the upstream input data. For example, the write process corresponds to the write service logic, so the storage process of the corresponding data can be implemented according to these write service logics.

[0214] Furthermore, in an embodiment of the present invention, the data service layer 120 may have various types of data service modules and tools that are maintained modularly, and may also serve as a client for third-party calls to provide specific data processing and conversion functions. In the data service layer 120, the functions of various data processing services may be relatively independent, and the triggering logic of its processing services may be combined and connected in series by the business service logic defined by the data receiving layer 110. For example, the data writing service logic may be combined and connected in series respectively, so that after receiving the upstream input data that has been preliminarily processed by the data receiving layer 110, the data service layer 120 may perform data writing processing services on these upstream input data according to the data writing service logic defined by the data service layer 120, such as Figure 2A In the writing process 1 to the writing process 3 shown, each writing process can represent a writing process of a certain modality data, or a single writing process of multi-threaded parallel writing processing of a certain modality data. Among them, the various data processing service modules and tools included in the data service layer 120 can also be used for the data service process of a specific data type to realize flexible combination and arrangement. For details, please refer to the following about Figure 2A-Figure 3 Description of the data writing service and the data retrieval service are shown.

[0215] In addition, in the embodiment of the present invention, the specific business logic of the data service layer 120 belongs to business-related information. After processing and converting the upstream input data, the use of general storage units and external systems ultimately falls on the data control layer 130.

[0216] like Figure 2AAs shown, the data control layer 130 (such as Data Base Controller) may include various databases (such as Figure 2A The multimodal data storage control device of the embodiment of the present invention can be a data storage processing service architecture between the data source module 200 and the database, and can realize the data processing process between the corresponding intelligent platform (such as TongOS platform) and the corresponding multimodal database. Among them, the corresponding database can be MongoDB, Redis, Milvus, etc. for multimodal data storage processing.

[0217] Therefore, the multimodal data storage control device of the intelligent body according to the embodiment of the present invention can realize efficient recognition and precise storage control of various types of multimodal data, and can simultaneously support storage control services for multiple different types of multimodal data, ensuring that it can simultaneously support the access and parallel processing of multiple different types of multimodal data, and use triplet difference data to realize the difference expression of visual input data, support the storage and retrieval services of the corresponding difference data, and, through the data receiving layer 110 and the storage control layer 130, it can be linked with the algorithm module and support the calling of third-party modules, so that the overall device architecture is easier to expand and manage.

[0218] In summary, based on the multimodal data storage control device of the embodiment of the present invention, a data processing architecture with functions such as modeling, collection and storage of intelligent body memory data, comprehensive document database, vector database, etc. is constructed to realize the storage of multimodal data such as the intelligent body's multimodal perception and cognitive data, and support various real-time question and answer scenarios with technologies such as retrieval enhancement, so that the embodied intelligence can process multimodal data more efficiently and smoothly, and enable the intelligent body to accurately store and query multimodal data according to different interaction scenarios, thereby achieving a higher level of intelligence.

[0219] like Figure 1-2A As shown, according to an embodiment of the present invention, the data service layer 120 includes a data writing module 121. The data writing module 121 is used to perform writing services on received upstream input data according to different data types according to business service logic.

[0220] As mentioned above, in an embodiment of the present invention, the data service layer 120 can take over the business service logic defined by the data receiving layer 110 and further perform in-depth processing and handling on the upstream input data input by the data receiving layer 110, such as writing services and retrieval services.

[0221] In the embodiment of the present invention, the data writing service can be performed by the data writing module 121 according to the writing service logic defined by the data receiving layer 110 to execute the writing process. Among them, the data writing can be similar to the streaming process. After the intelligent agent is started, the system automatically completes the initialization and establishes the data link. Thereafter, the data generated by the intelligent agent during operation will be continuously written to the data through the multimodal data storage control device of the intelligent agent of the embodiment of the present invention. At this time, the data party does not need to actively call the corresponding data service protocol or interface (such as API interface, etc.).

[0222] Among them, different types of upstream input data will execute write service processes according to different write links, such as Figure 2A Shown are write process 1 - write process 3.

[0223] Therefore, the data writing module 121 can implement writing processing for multimodal data of different data types, thereby ensuring that multimodal data of different data types can be accurately and efficiently written according to a matching writing process.

[0224] like Figure 1-2A As shown, according to an embodiment of the present invention, the data writing module 121 includes a difference generation service unit 201 and a key-value pair data service unit 202. The difference generation service unit 201 is used to generate triple difference data according to the cognitive understanding data of the upstream input data to capture and describe the amount of change before and after the data; the key-value pair data service unit 202 is used to perform key-value pair data conversion services on the triple difference data and the cognitive understanding data of the upstream input data, the self-action data and the visual input data.

[0225] like Figure 2A As shown, in an embodiment of the present invention, the difference generation service unit 201 and the key-value pair data service unit 202 can serve as data processing service modules and tools of the above-mentioned data writing module 121, and realize the data writing processing flow through flexible combination and arrangement.

[0226] In an embodiment of the present invention, the difference generation service unit 201 (such as a diff generator) can process cognitive understanding data and generate triple difference data based on the cognitive understanding data. The triple difference data can be used as description data of the data change amount of cognitive understanding data, and there is a difference with the corresponding full data. For example, the difference generation service unit 201 can compare the difference description between two adjacent Belief PG data in the cognitive understanding data according to a predetermined field, thereby generating corresponding triple difference data.

[0227] In an embodiment of the present invention, the key-value pair data service unit 202 (such as a json data service unit) can implement key-value pair data processing, such as key-value pair data conversion services (such as json format data conversion) for data processing, conversion and storage of such data. Among them, triple difference data and cognitive understanding data of upstream input data, self-action data and visual input data can all be used as processing objects of the key-value pair data service unit 202. Through the key-value pair data service unit 202, the data writing process can be realized in json format for these multimodal data.

[0228] Therefore, compared with the situation that traditional data processing systems can only process single-modal data, through the flexible combination and service logic arrangement of the above-mentioned difference generation service unit 201 and the key-value pair data service unit 202, it is possible to realize parallel data writing of multiple multimodal data. The data writing process is more targeted and can support data joint interpretation, thereby reducing the difficulty of data writing, further ensuring that the intelligent question and answer system based on the above-mentioned multimodal data storage control device of the embodiment of the present invention is not prone to erroneous questions and answers, and has stronger interpretability. In addition, the flexible combination and service logic arrangement of multiple units also makes the data architecture more engineered and more powerful, and can well achieve horizontal expansion.

[0229] like Figure 1-2A As shown, according to an embodiment of the present invention, the data writing module 121 further includes a text vector service unit 203 and a vector data service unit 204. The text vector service unit 203 is used to perform data vectorization services on triple difference data, cognitive understanding data, and self-action data; the vector data service unit 204 is used to perform vector data writing on the vectorized data generated by the data vectorization service.

[0230] like Figure 2AAs shown, in an embodiment of the present invention, based on the combination and arrangement of the above-mentioned difference generation service unit 201 and the key-value pair data service unit 202, the text vector service unit 203 and the vector data service unit 204 can serve as data processing service modules and tools for the above-mentioned data writing module 121, and realize the data writing processing flow through flexible combination and arrangement.

[0231] In an embodiment of the present invention, the text vector service unit 203 (such as the info2text client) can implement data vectorization (Embedding) processing services on the input triple difference data, cognitive understanding data, self-action data, etc. according to a third-party service call request, and generate corresponding text description information and vectorized data.

[0232] In an embodiment of the present invention, the vector data service unit 204 can serve as a writing service module for the above-mentioned vectorized data, and execute a vectorized writing process for the vectorized data generated by the above-mentioned triple difference data, cognitive understanding data, self-action data, etc., so that it can be accurately written into the corresponding vector database through the storage control layer 130.

[0233] Therefore, based on the combined arrangement of the above-mentioned difference generation service unit 201 and the key-value pair data service unit 202, through the flexible combination and service logic arrangement of the above-mentioned text vector service unit 203 and the vector data service unit 204, it is possible to further realize parallel data writing of multiple multimodal data, and the data writing process is more targeted, which can better support data joint interpretation, thereby further reducing the difficulty of data writing, ensuring that the intelligent question and answer system is not prone to erroneous questions and answers, and has stronger interpretability. In addition, the flexible combination and service logic arrangement of multiple units also make the data architecture more engineered and more powerful, and can well achieve horizontal expansion.

[0234] Figure 2B Schematically shows a flow chart of a data writing application scenario of a multimodal data storage control device of an intelligent agent according to an embodiment of the present invention; Figure 2C Schematically shows a flow chart of another data writing application scenario of the multimodal data storage control device of an intelligent agent according to an embodiment of the present invention; Figure 2D A flowchart of another data writing application scenario of the multimodal data storage control device of an intelligent agent according to an embodiment of the present invention is schematically shown.

[0235] like Figure 2A and Figure 2BAs shown, in the writing link of cognitive understanding data, the data writing module 121 can implement the json data writing service of cognitive understanding data through the key-value pair data service unit 202. Specifically, the key-value pair data service unit 202 can expand the nested data structure of cognitive understanding data according to specific preset fields (such as instance_list and action_list, etc.), convert it into a json data array (such as json array), and finally store the data array in a designated database, such as MongoDB, according to the storage path of the designated database through the storage control layer 130, thereby completing the json data writing process of the cognitive understanding data.

[0236] Accordingly, if Figure 2B-Figure 2D As shown, the key-value pair data service unit 202 can also implement JSON data writing service for triple difference data, self-action data, visual input data and other data, and write them into a MongoDB database, which will not be described in detail.

[0237] like Figure 2A and Figure 2B-2C As shown, in the writing link of vectorized data, such as the writing process of cognitive understanding data, the data writing module 121 can implement the writing service of the vectorized data of cognitive understanding data through the text vector service unit 203 and the vector data service unit 204. Specifically, if and only if there is an action data field (such as action) in the cognitive understanding data, and its start time (startTime) and end time (endTime) both exist, the service request for data vectorization (Embedding) and the subsequent vector storage service will be triggered. Further, by requesting the text vector service unit 203 to execute the above-mentioned cognitive understanding data vectorization processing service process, the corresponding vectorized data and the corresponding natural language description information are generated. On this basis, the vectorized storage process can be executed on the above-mentioned vectorized data by calling the vector data service unit 204, thereby completing the service process of writing its vectorized data.

[0238] Accordingly, if Figure 2C As shown, the text vector service unit 203 and the matching vector data service unit 204 can also implement vectorized data writing services for self-action data (U-action) and other data, and write them into a MongoDB database, which will not be described in detail.

[0239] like Figure 2A and Figure 2BAs shown, before the triple difference data is written into the link, the data writing module 121 can generate triple difference data according to the cognitive understanding data through the difference generation service unit 201. Specifically, the difference generation service unit 201 can obtain the incremental data of the cognitive understanding data, and generate the corresponding triple difference data by performing data vectorization services such as spo_diff and embedding_diff on the incremental data. The data format of the vectorized data of the triple difference data corresponding to embedding_diff may be different from that of the above-mentioned vectorized data.

[0240] Furthermore, for the triple difference data write link (status change write), the data write module 121 can implement vectorized data writing of the triple difference data by calling the text vector service unit 203 and the vector data service unit 204, and can call the key-value pair data service unit 202 in parallel to implement json data conversion and writing of the triple difference data, thereby realizing its corresponding json data storage.

[0241] like Figure 2A and Figure 2D As shown, in the writing link of the visual input data, the data writing module 121 can realize the json data conversion and writing of the visual input data through the key-value pair data service unit 202. Specifically, the key-value pair data service unit 202 can expand the nested data structure of the visual input data according to the preset fields (such as instance_list and action_list, etc.), convert it into a json data array (such as json array), and finally store the data array into a specified database, such as a MongoDB database, according to the storage path of the specified database through the storage control layer 130.

[0242] It should be noted that, in the process of generating the triple difference data, the data service layer 120 of the embodiment of the present invention can calculate the cognitive understanding data transmitted by the top-level microservice separately through the difference generation service unit 201 (such as the status change server) to generate the corresponding difference data (such as the status change data). Specifically, the difference generation service unit 201 can output the difference data in the form of SPO according to the specific fields in the cognitive understanding data (such as isIn / isOn / hasColour / shape / CVReliability), that is, to realize the "subject-predicate-object" triple expression of the difference data, that is, the triple difference data.

[0243] For the triple difference data of the above embodiment of the present invention, the change of a specific field can represent the change of the state information of an object. For example, the change of the isOn field represents the change of the location of an object, and its triple difference data can be expressed as:

[0244] {

[0245] "change_time": "2021-01-01T00:00:00Z",

[0246] "subject": {

[0247] "rdf:type": "CommonBase#BuildingBlock",

[0248] "value": "toy_cube_1"

[0249] },

[0250] "predicate": {

[0251] "rdf:type": "CommonBase#isOn"

[0252] },

[0253] "object": {

[0254] "before": null,

[0255] "after": {

[0256] "value": "floor",

[0257] "rdf:type": "CommonBase#FloorBoard"

[0258] }

[0259] },

[0260] },

[0261] Therefore, according to the expression content of the above triple difference data, it can be understood that "the toy_cube_1 is put on the floor", and toy_cube_1 has no associated object before.

[0262] In addition, for triple difference data, predicate also includes EXISTENCE, which can represent the appearance or disappearance of an object, for example:

[0263] {

[0264] "change_time": "2021-01-01T00:00:00Z",

[0265] "subject": {

[0266] "rdf:type": "CommonBase#RobotAIBaby",

[0267] "value": "aibaby_brother"

[0268] },

[0269] "predicate": {

[0270] "rdf:type": "EXISTENCE"

[0271] },

[0272] "object": {

[0273] "before": null,

[0274] "after": {

[0275] "value": true,

[0276] "ontology": null

[0277] }

[0278] },

[0279] }

[0280] According to the triple difference data expression for the predicate expression above, it can be understood as the state change of "aibaby_brother appears", that is, status change.

[0281] Therefore, based on the combined arrangement of the above-mentioned difference generation service unit 201, key-value pair data service unit 202, text vector service unit 203 and vector data service unit 204, through the above-mentioned data writing module 121, efficient data writing of multiple multimodal data in parallel can be realized, and the data writing process is more targeted, and can better support data joint interpretation, thereby further reducing the difficulty of data writing, ensuring that the intelligent question and answer system is not prone to erroneous questions and answers, and has stronger interpretability. Moreover, the flexible combination of multiple units and the service logic arrangement also make the data architecture more engineered and more powerful, and can well achieve horizontal expansion.

[0282] Therefore, the multimodal data storage control device of the above-mentioned intelligent body in the embodiment of the present invention can realize the data processing architecture of the functions of modeling, collection and storage of the intelligent body's memory data, comprehensive document database, vector database, etc., and realize the writing control of multimodal data such as the intelligent body's multimodal perception and cognitive data, so as to support various real-time question-and-answer scenarios, make the embodied intelligence's processing of multimodal data more efficient and smooth, and enable the intelligent body to accurately realize the storage and query of multimodal data according to different interaction scenarios, thereby achieving a higher level of intelligence.

[0283] Figure 3 A structural block diagram schematically shows a data retrieval application scenario of a multimodal data storage control device of an intelligent agent according to an embodiment of the present invention.

[0284] like Figure 1-Figure 3 As shown, according to an embodiment of the present invention, the data service layer 120 further includes a data retrieval module 122. The data retrieval module 122 is used to respond to a data retrieval request and call the storage control layer 130 to perform retrieval services according to different data types.

[0285] The data retrieval request can be a computer instruction generated by a third-party service module directly calling the data service layer 120, which is used to control the data retrieval module 122 to execute the retrieval process. For example: the intelligent agent generates a data call instruction in response to a query received from a human being and sends it to the data service layer 120 of the multimodal data storage control device of the intelligent agent in an embodiment of the present invention. The data service layer 120 can use the data call instruction as a data retrieval request, generate corresponding data retrieval service logic, and call the storage control layer 130 accordingly to perform corresponding data retrieval.

[0286] Through the above-mentioned data retrieval module 122 of the embodiment of the present invention, it can be ensured that multimodal data of different data types can be accurately and efficiently queried and extracted according to the matching retrieval process. Therefore, the intelligent question-answering system based on the agent memory data expressed in natural language and the large language model can effectively avoid the occurrence of "hallucinations" and wrong answers, greatly improving its interpretability.

[0287] It should be noted that the data retrieval request can also be a computer instruction generated by a third-party service module directly calling the data receiving layer 110 to control its execution of the defined retrieval service logic. The data receiving layer 110 can send it and the corresponding retrieval service logic to the data service layer 120 to implement the corresponding data retrieval service process, which is not elaborated on in detail.

[0288] like Figure 1-Figure 3As shown, according to an embodiment of the present invention, the data retrieval module 122 includes a difference data retrieval service unit 301, a cognitive data retrieval service unit 302, an action data retrieval service unit 303 and a visual data retrieval service unit 304. The difference data retrieval service unit 301 is used to perform a query on the triple difference data stored by the data storage control service; the cognitive data retrieval service unit 302 is used to perform a query on the cognitive understanding data stored by the data storage control service; the action data retrieval service unit 303 is used to perform a query on the self-action data stored by the data storage control service; and the visual data retrieval service unit 304 is used to perform a query on the visual data stored by the data storage control service.

[0289] like Figure 3 As shown, the data retrieval module 122, through the combination and arrangement of service modules such as the difference data retrieval service unit 301, the cognitive data retrieval service unit 302, the action data retrieval service unit 303 and the visual data retrieval service unit 304, can execute data retrieval service calls to the storage control layer 130 according to data retrieval requests matching different data types, initiate a data query process for the corresponding data type, and finally complete the retrieval and extraction process of the corresponding data from the specified database through the storage control layer 130.

[0290] Among them, different data retrieval service units can implement retrieval services for different types of multimodal data, for example, the difference data retrieval service unit 301 can implement retrieval services for triple difference data (such as status change), the cognitive data retrieval service unit 302 can implement retrieval services for cognitive understanding data (such as Belief PG), the action data retrieval service unit 303 can implement retrieval services for self-action data (such as U-action), and the visual data retrieval service unit 304 can implement retrieval services for visual input data (such as CVPG). Therefore, parallel query of multimodal text data of multiple different data types can be realized at the same time, and the retrieval is more efficient and accurate.

[0291] It can be seen that based on the combined arrangement of the above-mentioned difference data retrieval service unit 301, cognitive data retrieval service unit 302, action data retrieval service unit 303 and visual data retrieval service unit 304, through the above-mentioned data retrieval module 122, efficient data query of various multimodal text data in parallel can be realized, the data retrieval process is more targeted, and can better support data joint interpretation, thereby further reducing the difficulty of data retrieval, ensuring that the intelligent question and answer system is not prone to erroneous questions and answers, and has stronger interpretability. Moreover, the flexible combination of multiple units and the arrangement of service logic also make the data architecture more engineered and more powerful, and can well achieve horizontal expansion.

[0292] like Figure 1-Figure 3 As shown, according to an embodiment of the present invention, the data retrieval module 122 further includes a vector retrieval service unit 305. The vector retrieval service unit 305 is used to perform vectorized retrieval on the vectorized data stored after the vector data service unit 204 performs vector data writing.

[0293] like Figure 3 As shown, in an embodiment of the present invention, the vector retrieval service unit 305 can execute a vectorized data retrieval service call to the storage control layer 130 according to a data retrieval request for vectorized data matching different data types, initiate a query process for vectorized data of the corresponding data type, and finally complete the retrieval and extraction process of the corresponding vectorized data from a specified database (such as a vectorized database) through the storage control layer 130.

[0294] For example, when the data retrieval request corresponds to the vectorized data retrieval of the own action data, a retrieval call control instruction corresponding to the storage control layer 130 can be generated, so that the storage control layer 130 can perform query services to the specified vectorized database. Finally, the storage control layer 130 can output the target vectorized data extracted from the corresponding vectorized database to the outside through the vector retrieval service unit 305.

[0295] It can be seen that based on the combined arrangement of the above-mentioned difference data retrieval service unit 301, cognitive data retrieval service unit 302, action data retrieval service unit 303 and visual data retrieval service unit 304, through the above-mentioned vector retrieval service unit 305, efficient retrieval of various multimodal vectorized data can be further realized, and efficient data query in parallel can be performed. The data retrieval process is more targeted and can better support data joint interpretation, thereby further reducing the difficulty of data retrieval, ensuring that the intelligent question and answer system is not prone to erroneous questions and answers, and has stronger interpretability. Moreover, the flexible combination of various units and the service logic arrangement also make the data architecture more engineered and more powerful, and can well achieve horizontal expansion.

[0296] It should be noted that the text retrieval process and the vectorized data retrieval process of the data retrieval module 122 require the implementation of a retrieval protocol between the intelligent multimodal data storage control device 100 of the embodiment of the present invention and a third party or a database, such as a corresponding retrieval API. The retrieval API can be used to define the storage and retrieval protocol of visual input data.

[0297] Specifically, if you perform a search on visual input data, you can use the preset fields corresponding to the corresponding search API: POST / cvpg / search?agent_id=<AGENT_ID> , initiate the query body (request body), such as:

[0298] {

[0299] "filter": {},

[0300] "limit": 0,

[0301] "projection": {},

[0302] "is_aggregate": true,

[0303] "aggregation": [

[0304] {}

[0305] ],

[0306] "unflatten": false

[0307] }

[0308] At the same time, the storage control layer 130 as the storage end can write the time field in the retrieval process as follows:

[0309] creation_time int, 16 bits cvpg outermost layer The top-level fields in Cvpg and belief-pg are used in the range range, which is consistent with the OScvpg time. start_time int, 16 bits cvpg-action Corresponding to common_base:startTime end_time int, 16 bits cvpg-action Corresponding to common_base:endTime infer_time int, 16 bits cvpg-action Corresponding to common_base:inferTime

[0310] The above time fields will eventually be fed back together with the retrieved visual input data. The retrieval APIs for other text data retrieval and vectorized data retrieval are not described in detail here.

[0311] In summary, according to the design of the above-mentioned retrieval API, for the retrieval scenarios of actual applications, corresponding queries can be generated to call relevant interface protocols. Specifically, in the text data retrieval service, the text retrieval service (such as text-search) can be called to search for retrieval data such as visual input data within a given time period, corresponding to POST / CVPG / search?agent_id=AGENT_ID, and filter the data through the filter (such as filter) retrieval field to extract the visual input data that meets the conditions in the database. In addition, for the recall of specific scenarios or related information, the vector retrieval service (such as vector-search) is accessed through the embedding vector data of the natural language, corresponding to POST / vector-search?agent_id=AGENT_ID, to obtain the matching vectorized information of semantic similarity.

[0312] It can be seen that mixing the above-mentioned text retrieval and vector retrieval can realize more complex intelligent agent query scenarios, in which the generation and triggering of queries can be triggered by modules such as NLP, and the downstream data processing module can perform information aggregation and natural language generation to jointly complete the end-to-end implementation of question answering.

[0313] like Figure 1-Figure 3 As shown, according to an embodiment of the present invention, the storage control layer 130 includes a plurality of storage control service units, each of which is used to perform storage control services on a matching database.

[0314] like Figure 1 , Figure 2A and Figure 3 As shown, the storage control layer 130 may include multiple constituent modules such as storage control service units 1-3, and each storage control service unit may perform data storage control services corresponding to different databases, including storage control processes such as data writing and data retrieval. For example, storage control service unit 1 may establish a corresponding storage control interaction relationship for database 1 to implement writing and retrieval of data in the database 1, that is, data reading and writing.

[0315] For example, the storage control layer 130 can connect to different data persistence layers, call the native SDK / API of components such as databases, and implement the data reading and writing process. Among them, the data persistence layer can be one of the basic setting layers, which can establish and deploy databases, implement database cluster management, and have functions such as port maintenance, system detection, and log recording. Accordingly, the database can include various databases such as MongoDB, Redis, and Milvus according to the reading and writing requirements of different text data and vectorized data.

[0316] Therefore, through the above-mentioned storage control layer 130, on the basis of the above-mentioned data service layer 120, further docking with an external database can be achieved to ensure that targeted reading and writing of data in the database can be achieved according to the data writing process or data retrieval process initiated by the data service layer 120. The data reading and writing process is more efficient and accurate, and can ensure the parallel processing of multiple multimodal data.

[0317] The multimodal data storage control device of the intelligent agent based on the embodiment of the present invention can enable the embodied intelligence to realize efficient and accurate data storage control process, and then realize question and answer support such as Tpgqa. The finally realized intelligent system can answer such questions, for example: "How many red building blocks were on the table one minute ago?", "How many red building blocks were on the table when you ate tomatoes?" and "When did you eat tomatoes?" inquiry scenarios.

[0318] More specifically, in order to further reflect the technical effect of the multimodal data storage control device of the above-mentioned intelligent agent in the embodiment of the present invention, the following application scenarios are provided for further explanation:

[0319] First, the embodied intelligence system is started (this is irrelevant to the operation of the multimodal data storage control device of the embodiment of the present invention): the intelligent agent is awakened, starts to look around, and while exploring the world, understands the world, starts to look for food in its environment, and obtains corresponding multimodal exploration data.

[0320] Secondly, the front-end data processing process is executed. Specifically, the front-end system and other algorithm modules can be used to enable the agent to convert the continuously input visual signals into multimodal data in a specific data format that is easy for the system to understand. For example, when the agent finds an apple on the table and decides to eat it, the decision-making system of the agent issues an instruction to eat it, which is still irrelevant to the operation of the multimodal data storage control device of the embodiment of the present invention.

[0321] After that, the data writing process is further performed. Specifically, the agent's understanding and modeling of the world in its mind (such as "there is a table in the room, there are apples on the table, apples are food, and 'I' am a little hungry"), this information is written into the Belief PG (i.e. cognitive understanding data) data writing process (such as Figure 2A-2D The data is written into the business logic shown in the figure. It goes through the corresponding writing steps of the key-value pair data service unit (such as JSON data service), difference generation service unit, vector data service unit (such as embedding and natural language description), and finally stored in different databases. In this process, the following details may also be involved relative to the intelligent agent (i.e., "I"): (a) The apple is picked up by "I", which corresponds to a state change, "the apple is no longer on the table", and this fact is reflected and described by a difference data (status change); (b) The apple is eaten, similarly, corresponding to "the apple no longer exists", corresponding to the description of another difference data (status change); (c) The apple disappeared because "I" issued an instruction to eat the apple, corresponding to the issuance of an action, which is reflected in the data of U-action (i.e., its own action data); (d) The current entire scene is described by Belief PG (i.e., cognitive understanding data), and the data is processed and stored by the Belief PG data write process; (e) Furthermore, the difference data and its own action data can generate vectors and natural language data through the info2text service (i.e., text vector service unit), and are also stored in the database.

[0322] Finally, humans as users can interact with intelligent agents in the following ways:

[0323] Example 1: A user asks the agent a question: "What were you doing just now?" The agent's front-end algorithm component converts this language description into a query, and implements retrieval interaction through the retrieval link API of the multimodal data storage control device of an embodiment of the present invention. The keywords here are "just now" and "doing" (the front-end component generates a filter corresponding to the time), and the agent's own action data stored is searched through text retrieval (text-search).

[0324] Example 2: The user asks the agent a question, "What red fruits were on the table just now?" The agent's front-end algorithm component can convert it into a query. A reasonable query idea is to parse several keywords "just now", "on the table", and "fruit" to correspond to the time, relative position relationship, and object attributes, generate data filtering conditions, and initiate text retrieval; at the same time, vector query (vector-search) can be performed through the vectorized data corresponding to "red fruit" to obtain relevant data that has contained similar to "red fruit" in historical scenes.

[0325] Therefore, it can be seen that the storage control device for multimodal data of the above-mentioned intelligent body of the embodiment of the present invention can meet the parallel storage requirements of multimodal data of various different data types, and can support file retrieval and vector retrieval at the time and field levels, and can further realize asynchronous writing and synchronous reading, greatly optimizing the write throughput, and can also generate differential expressions of corresponding data, support the storage and retrieval of differential data, and support semantic similarity retrieval through vectorization of multimodal data and generated natural language description. In addition, the above-mentioned device of the embodiment of the present invention can support the access of multiple data types, support docking with various embodied intelligent data platforms (such as TongOS), and realize the connection with interfaces such as grpc stream; in addition, it can also realize linkage with algorithm modules, support the calling of third-party modules, and the required device architecture is easy to expand and easy to manage.

[0326] In summary, based on the multimodal data storage control device of the embodiment of the present invention, a data processing architecture with functions such as modeling, collection and storage of the intelligent body's memory data, comprehensive document database, vector database, etc. is constructed to realize the storage of the intelligent body's multimodal perception and cognitive data, and support various real-time question-and-answer scenarios with technologies such as retrieval enhancement, so that the embodied intelligence can process multimodal data more efficiently and smoothly, and enable the intelligent body to accurately store and query multimodal data according to different interaction scenarios, thereby achieving a higher level of intelligence.

[0327] Another aspect of the embodiments of the present invention provides an intelligent agent, which includes the multimodal data storage control device of the intelligent agent described above. The intelligent agent of the embodiments of the present invention can be a humanoid embodied intelligence, which can realize intelligent interaction with real humans, and enable users to have a better intelligent level experience. In addition, the intelligent agent can also be a common intelligent robot, an intelligent drone, and an intelligent car, etc., which will not be described in detail.

[0328] To further reflect the improvements of the multimodal data storage control device of the above-mentioned intelligent body in the embodiment of the present invention, correspondingly, another aspect of the embodiment of the present invention provides a multimodal data storage control method of an intelligent body, which is implemented based on the multimodal data storage control device of the above-mentioned intelligent body, and includes: identifying the received upstream input data and defining the business service logic that matches the data type of the upstream input data; performing processing services on the received upstream input data according to the business service logic; and performing data storage control services according to the processing services.

[0329] Figure 4 The application scenario diagram of the multimodal data storage control method, device, equipment, medium and program product of an intelligent agent according to an embodiment of the present invention is schematically shown.

[0330] like Figure 4 As shown, the application scenario 400 according to this embodiment may include terminal devices 401, 402, 403, a network 404 and a server 405. The network 404 is used to provide a medium for a communication link between the terminal devices 401, 402, 403 and the server 405. The network 404 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0331] Users can use terminal devices 401, 402, 403 to interact with server 405 through network 404 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 401, 402, 403, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only for example).

[0332] The terminal devices 401 , 402 , and 403 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0333] The server 405 may be a server that provides various services, such as a background management server (only an example) that provides support for websites browsed by users using the terminal devices 401, 402, and 403. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0334] It should be noted that the multimodal data storage control method of the intelligent body provided in the embodiment of the present invention can generally be executed by the server 405. Accordingly, the multimodal data storage control device of the intelligent body provided in the embodiment of the present invention can generally be set in the server 405. The multimodal data storage control method of the intelligent body provided in the embodiment of the present invention can also be executed by a server or server cluster that is different from the server 405 and can communicate with the terminal devices 401, 402, 403 and / or the server 405. Correspondingly, the multimodal data storage control device of the intelligent body provided in the embodiment of the present invention can also be set in a server or server cluster that is different from the server 405 and can communicate with the terminal devices 401, 402, 403 and / or the server 405.

[0335] It should be understood that Figure 4 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0336] The following will be based on Figure 4 The scene described by Figure 4~Figure 5 The multimodal data storage control method of the intelligent agent of the disclosed embodiment is described in detail.

[0337] like Figure 5 As shown, another aspect of an embodiment of the present invention provides a multimodal data storage control method of an intelligent agent, which is implemented based on the multimodal data storage control device of the intelligent agent mentioned above, and includes operations S501-S503.

[0338] In operation S501, the received upstream input data is identified, and a business service logic matching the data type of the upstream input data is defined;

[0339] In operation S502, a processing service is performed on the received upstream input data according to the business service logic;

[0340] In operation S503, the data storage control service is performed according to the processing service.

[0341] Among them, the technical effects that can be achieved by the multimodal data storage control method for an intelligent body in an embodiment of the present invention can refer to the technical effect description of the multimodal data storage control device for an intelligent body mentioned above, and will not be repeated here.

[0342] Based on the above-mentioned multimodal data storage control method of the intelligent agent, the following will be combined with Figure 1 The multimodal data storage control device of the intelligent agent is further described.

[0343] Figure 1 The structure block diagram of the multimodal data storage control device of an intelligent agent according to an embodiment of the present invention is schematically shown.

[0344] like Figure 1 As shown, the multimodal data storage control device 100 of the intelligent agent in this embodiment includes a data receiving layer 110 , a data service layer 120 and a storage control layer 130 .

[0345] The data receiving layer 110 is used to receive upstream input data, identify the upstream input data, and define business service logic matching the data type of the upstream input data. In one embodiment, the data receiving layer 110 can be used to perform the operation S501 described above, which will not be described in detail here.

[0346] The data service layer 120 is used to perform processing services according to the business service logic on the upstream input data received by the data receiving layer 110. In one embodiment, the data service layer 120 can be used to perform the operation S502 described above, which will not be described in detail here.

[0347] The storage control layer 130 is used to perform data storage control services according to the processing services of the data service layer 120. In one embodiment, the storage control layer 130 can be used to perform the operation S503 described above, which will not be described in detail here.

[0348] According to an embodiment of the present invention, any multiple modules in the data receiving layer 110, the data service layer 120 and the storage control layer 130 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present invention, at least one of the data receiving layer 110, the data service layer 120 and the storage control layer 130 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware and firmware or in any appropriate combination of any of them. Alternatively, at least one of the data receiving layer 110, the data service layer 120 and the storage control layer 130 can be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding function can be executed.

[0349] Figure 6 A block diagram of an electronic device suitable for implementing a multimodal data storage control method for an intelligent agent according to an embodiment of the present invention is schematically shown.

[0350] The above-mentioned electronic device provided by an embodiment of the present invention includes one or more processors and a memory, and the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the above-mentioned multimodal data storage control method of the intelligent body.

[0351] like Figure 6 As shown, the electronic device 600 according to an embodiment of the present invention includes a processor 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 to a random access memory (RAM) 603. The processor 601 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 601 may also include an onboard memory for caching purposes. The processor 601 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0352] In RAM 603, various programs and data required for the operation of electronic device 600 are stored. Processor 601, ROM 602 and RAM 603 are connected to each other via bus 604. Processor 601 performs various operations of the method flow according to the embodiment of the present invention by executing the program in ROM 602 and / or RAM 603. It should be noted that the program can also be stored in one or more memories other than ROM 602 and RAM 603. Processor 601 can also perform various operations of the method flow according to the embodiment of the present invention by executing the program stored in the one or more memories.

[0353] According to an embodiment of the present invention, the electronic device 600 may further include an input / output (I / O) interface 605, which is also connected to the bus 604. The electronic device 600 may further include one or more of the following components connected to the I / O interface 605: an input portion 606 including a keyboard, a mouse, etc.; an output portion 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 608 including a hard disk, etc.; and a communication portion 609 including a network interface card such as a LAN card, a modem, etc. The communication portion 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage portion 608 as needed.

[0354] The present invention also provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to execute the multimodal data storage control method of the above-mentioned intelligent agent.

[0355] The computer-readable storage medium may be included in the device / apparatus / system described in the above embodiment; or it may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present invention is implemented.

[0356] According to an embodiment of the present invention, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present invention, the computer-readable storage medium may include the ROM 602 and / or RAM 603 described above and / or one or more memories other than ROM 602 and RAM 603.

[0357] An embodiment of the present invention also includes a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the multimodal data storage control method of the above-mentioned intelligent agent.

[0358] The computer program includes program codes for executing the method shown in the flow chart. When the computer program product is run in a computer system, the program codes are used to enable the computer system to implement the method provided by the embodiment of the present invention.

[0359] The computer program executes the above functions defined in the system / device of the embodiment of the present invention when it is executed by the processor 601. According to the embodiment of the present invention, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0360] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 609, and / or installed from a removable medium 611. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0361] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the processor 601, the above functions defined in the system of the embodiment of the present invention are performed. According to the embodiment of the present invention, the system, device, means, module, unit, etc. described above can be implemented by a computer program module.

[0362] According to an embodiment of the present invention, the program code for executing the computer program provided by the embodiment of the present invention can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0363] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0364] It will be appreciated by those skilled in the art that the features described in the various embodiments and / or claims of the present invention may be combined and / or combined in various ways, even if such combinations and / or combinations are not explicitly described in the present invention. In particular, the features described in the various embodiments and / or claims of the present invention may be combined and / or combined in various ways without departing from the spirit and teachings of the present invention. All of these combinations and / or combinations fall within the scope of the present invention.

[0365] The embodiments of the present invention are described above. However, these embodiments are only for the purpose of illustration, and are not intended to limit the scope of the present invention. Although each embodiment is described above, it does not mean that the measures in each embodiment cannot be used in combination. The scope of the present invention is defined by the attached claims and their equivalents. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

[0366] All actions of acquiring information, signals or data in the present invention are carried out in compliance with the relevant data protection laws, regulations and policies of the country where they are located, and with the authorization given by the owner of the corresponding device.

Claims

1. A multimodal data storage control device for an intelligent agent, characterized in that: include: The data receiving layer is used to receive upstream input data, identify the upstream input data, and define business service logic that matches the data type of the upstream input data; wherein the data receiving layer is connected to the intelligent agent platform, receives the upstream data released by the intelligent agent platform, and uses the top-level microservices to process and distribute the data, and then calls the downstream specific data processing logic to realize data storage control; the upstream input data of different data types correspond to the multimodal data obtained by the intelligent agent in different application scenarios, including visual input data, self-action data, triple difference data, and cognitive understanding data; a data service layer, configured to perform processing services on the upstream input data received by the data receiving layer according to the business service logic; and The storage control layer is used to perform data storage control services according to the processing services of the data service layer; wherein the storage control layer includes docking with various databases, and implements storage control of upstream input data according to the database path and format defined between the storage control layer and the corresponding database by executing read and write service requests on the docked database; The data service layer includes a data writing module and a data retrieval module, and the data writing module is used to perform writing services on the received upstream input data according to different data types according to the business service logic; Wherein, the data writing module includes a difference generation service unit, a key-value pair data service unit, a text vector service unit and a vector data service unit: the difference generation service unit is used to generate triple difference data according to the cognitive understanding data of the upstream input data to capture and describe the amount of change before and after the data; the key-value pair data service unit is used to perform key-value pair data conversion services on the triple difference data and the cognitive understanding data of the upstream input data, the self-action data and the visual input data; the text vector service unit is used to perform data vectorization services on the triple difference data, cognitive understanding data and self-action data; the vector data service unit is used to perform vector data writing on the vectorized data generated by the data vectorization service; The data retrieval module is used to respond to a data retrieval request and call the storage control layer to perform retrieval services according to different data types.

2. The device according to claim 1, characterized in that The data retrieval module comprises: A difference data retrieval service unit, configured to perform a query on the triple difference data stored by the data storage control service; a cognitive data retrieval service unit, configured to perform a query on the cognitive understanding data stored by the data storage control service; An action data retrieval service unit, used to perform a query on the own action data stored by the data storage control service; The visual data retrieval service unit is used to perform queries on the visual data stored by the data storage control service.

3. The device according to claim 1, characterized in that The data retrieval module also includes: The vector retrieval service unit is used to perform vectorized retrieval on the vectorized data stored after the vector data service unit performs vector data writing.

4. The device according to claim 1, characterized in that The storage control layer includes: A plurality of storage control service units, each of the plurality of storage control service units is used to perform storage control services on a matching database.

5. An intelligent agent, characterized in that: A multimodal data storage control device comprising an intelligent agent according to any one of claims 1-4.

6. A method for controlling multimodal data storage of an intelligent agent, implemented based on the multimodal data storage control device of an intelligent agent according to any one of claims 1 to 4, characterized in that: include: Identify received upstream input data and define business service logic matching the data type of the upstream input data; Execute processing services on the received upstream input data according to the business service logic; as well as The data storage control service is executed according to the processing service.

7. An electronic device comprising: one or more processors; a memory for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors are caused to execute the method of claim 6.

8. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to execute the method of claim 6.

9. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method according to claim 6 is implemented.

Images