A method for constructing a universal artificial spatiotemporal intelligence large model

Abstract

The present invention discloses a method for constructing a universal artificial space-time intelligent large model. Based on the object expression and relationship model of zero-dimensional points, one-dimensional lines, two-dimensional surfaces, and three-dimensional bodies, a unified structured topological expression and storage of various types of multimedia information such as text, graphics, images, audio, video, etc. in the real world of human society is established. A multimodal large model training method with three-dimensional representation is adopted to construct an artificial space-time large model that can uniformly express and process all objects in human society. Combined with space-time information processing, space-time relationship analysis, knowledge graph and business process logical reasoning, it provides the capabilities of space-time relationship expression, semantic understanding, environmental perception, space-time reasoning and spatial decision-making, solves the problem that the current large language model has a single expression, lacks topological relationships, and is difficult to process the space-time relationships in the three-dimensional real world. It can be fully applied to multimedia information processing, theoretical model derivation, automatic driving, intelligent robots, industrial engineering and other fields.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method for constructing a universal artificial spatiotemporal intelligence large model. Background Art

[0002] In recent years, artificial intelligence technology has developed rapidly. Large Language Models (LLMs) have achieved remarkable results in many fields such as natural language processing, speech recognition, and computer vision due to their powerful representation learning and generalization capabilities. They have become a hot direction for research and development in the field of artificial intelligence and have been widely used in literature, art, education, finance and other industries, having a profound impact on the development of human society.

[0003] Currently, the number of parameters in large language models has reached billions or even hundreds of billions, which places extremely high demands on computing resources and training technology. In order to train such a large model, a large amount of data is required for training. Acquiring, cleaning and labeling this data is an expensive and time-consuming process, and is about to face bottlenecks in computing resources and data resources.

[0004] In addition, the essence of the large language model is to process language text data. Although it can also process multimedia data such as images and videos through encoding, it is still based on a one-dimensional data structure, with a relatively simple structure, a tokenized text sequence, and lacks the expression of spatial relationships. In addition, since the large model is mainly trained based on text data, it lacks direct experience of interaction with the physical world and cannot learn spatial, temporal and physical interactions through its own experience like humans. As a result, it cannot effectively integrate multi-dimensional information about time and space in the real world, and it is difficult to support the perception, understanding and operation of three-dimensional space. It cannot be directly used in industrial production, engineering construction and other fields that need to process multi-dimensional spatiotemporal information interaction. For example, in the application scenarios of autonomous driving and robots, since it is necessary to accurately model and predict the spatiotemporal changes of each object in the environment, the processing method of the large language model is based on the processing of one-dimensional data structures, which requires extremely high computing power support, lacks a unified analysis framework for time and space dimensions, and is difficult to provide accurate and consistent spatiotemporal correlation analysis results.

[0005] At present, large language models are mainly applicable to intelligent processing in the field of liberal arts, which is "liberal arts intelligence", such as text, images and videos. They are not suitable for processing data in science and engineering, especially the processing and interpretation of spatiotemporal data in the field of human social engineering, such as the intelligent processing of multi-dimensional design graphics of various types of engineering. Graphic design and application in the field of engineering are problems that must be solved in an industrialized and intelligent society. Otherwise, the application scenarios of artificial intelligence will be partial and incomplete.

[0006] The real world of human beings is composed of three-dimensional physical objects or multimedia information objects with time and space characteristics. The three-dimensional world follows the laws of physics and has inherent structures, attributes and time-space relationships. Faced with the inherent limitations of the existing large language models' one-dimensional data structure principles and data modeling methods, there is an urgent need to find a unified and universal large model expression and processing method, and to develop a large model of expression, perception, reasoning and interactive decision-making that is more suitable for the real space world. It can comprehensively process multimedia data such as "text, graphics, images, audio, video" faced by human society, and can handle all professional field problems such as "liberal arts, science, engineering", etc. Summary of the invention

[0007] In view of the above problems, the present invention proposes a method for constructing a universal artificial spatiotemporal intelligence large model.

[0008] The present invention discloses a method for constructing a universal artificial spatiotemporal intelligence large model, comprising:

[0009] Adopting multi-dimensional object abstract expression, various objects and spatial relationships in the real world are processed, and a spatio-temporal intelligent large model is established to uniformly process structured spatio-temporal object expression and data storage. The multi-dimensional object abstraction refers to: zero-dimensional point, one-dimensional line, two-dimensional surface, three-dimensional body;

[0010] The spatiotemporal object is processed using a multimodal spatiotemporal coding algorithm to convert the spatiotemporal data corresponding to the spatiotemporal object into coded data for subsequent spatiotemporal intelligent large model training and application;

[0011] Based on the multimodal spatiotemporal coding data, a multimodal spatiotemporal training data set is constructed, and a spatiotemporal multimodal training method with three-dimensional representation is adopted to train or fine-tune the spatiotemporal intelligent large model;

[0012] The trained spatiotemporal intelligence big model is published as a spatiotemporal intelligence big model reasoning service to provide spatiotemporal intelligence capabilities, which include: spatiotemporal relationship expression, semantic understanding, environmental perception, spatiotemporal reasoning and spatial decision-making.

[0013] Optionally, a multi-dimensional object abstract expression is used to process various objects and spatial relationships in the real world, including:

[0014] Collect data of various objects in the real world, including but not limited to multimedia information formats, identify and analyze the spatiotemporal information features of various objects in the multimedia information formats, and generate multimodal, structured spatiotemporal object expressions of zero-dimensional points, one-dimensional lines, two-dimensional surfaces, and three-dimensional bodies in combination with the environment, scene or context, relative space, and knowledge graph association and spatial relationship of various objects. The multimedia information formats include: text format, graphic format, image format, audio format, and video format. The spatiotemporal information features include: position, form, state, and tense.

[0015] Among them, the expression of spatiotemporal objects has the characteristics of position, tense and spatial relationship at the same time, ensuring the consistency and correlation of objects in time and space dimensions; the spatiotemporal objects are expressed through topological relationships, orientation or order relationships, and metric relationships. The topological relationships include: association, adjacency, and inclusion. The orientation or order relationships include: up, down, left, right, east, west, south, and north. The metric relationships include: distance, angle, or near.

[0016] Optionally, the various objects in the real world include: real geographic space objects in the real world, virtual geographic space objects, and multimedia type objects in the multimedia context's own relative space, and the multimedia type objects include: text, graphics, images, audio, and video types;

[0017] The zero-dimensional point is an object-oriented modeling expression of the basic unit of the real world; the object of the basic unit refers to various physical objects with spatial positions in the real world, as well as multimedia information objects, and the multimedia information objects include: text, graphics, images, audio, and video; for the various physical objects, they are expressed through the spatial position, environment, and spatial relationship of the objects; for the multimedia information objects, they are expressed through the digital features, scenes or contexts, and relative spaces of the objects, including but not limited to the sequence position of the text in the paragraph and the correlation relationship with other texts, graphic vector nodes, and the sequence position of the image frame in the video and the correlation relationship with other frames;

[0018] The one-dimensional line is a modeling expression of the change and evolution process of real-world objects in the time dimension, including: expressing the continuous change process or trajectory of the object in the time and space dimensions as a one-dimensional line, or expressing the data sequence of zero-dimensional points, multiple zero-dimensional point objects of the same type as a one-dimensional line, recording their flow process, logical relationship and topological adjacency relationship in time and space; wherein the data sequence includes but is not limited to a coordinate sequence, a character sequence, a text paragraph, a mathematical formula, a graphic vector arc segment, an image set, and a video frame;

[0019] The two-dimensional surface is a modeling expression of the adjacency and aggregation of real-world objects, including: expressing similar one-dimensional line objects as two-dimensional surfaces, recording their adjacency relationships in time and space, and using them for data analysis and association modeling on a larger scale; when processing the adjacency and aggregation of real-world objects, completely recording the basic unit's own characteristics and the complex derivation process or the combined association characteristics of multiple data according to the object's position, type or change process in time or space dimensions;

[0020] The three-dimensional body is a modeling expression of the overall spatiotemporal evolution of large-scale data sets or complex systems in real-world objects, including: representing large data sets of multi-dimensional zero-dimensional points, one-dimensional lines, and two-dimensional surfaces as complex bodies or complex processes, and recording their overall relationships or evolution processes in space and time; combining objects and events in multiple space-time dimensions to generate an overall space-time body structure to support comprehensive space-time analysis and prediction of complex systems.

[0021] Optionally, a multimodal spatiotemporal coding algorithm is used for processing to convert the spatiotemporal data corresponding to the spatiotemporal object into coded data for subsequent spatiotemporal intelligent large model training and application, including:

[0022] Step T1: Based on the spatiotemporal object, each basic object in the spatiotemporal data is regarded as a node of a spatiotemporal topological graph, and the spatiotemporal adjacency between each node is used as an edge in the graph. By calculating the spatiotemporal distance or correlation between objects, the edge weights between nodes are defined to construct a spatiotemporal topological graph structure; wherein the correlation between objects is determined according to the similarity, proximity or time sequence in the spatiotemporal data, and the spatiotemporal data refers to multimodal spatiotemporal multidimensional data;

[0023] Step T2: Encode data based on graph neural network, construct pre-training tasks through spatial association, and express the spatiotemporal topological graph structure in high dimension.

[0024] Optionally, data encoding is performed based on a graph neural network, a pre-training task is constructed through spatial association, and the spatiotemporal topological graph structure is expressed in high dimensions, including:

[0025] By aggregating information of nodes and neighboring nodes in the spatiotemporal topological graph, the spatiotemporal features of each node are updated to capture the spatial and temporal associations between the multimodal spatiotemporal multidimensional data;

[0026] The multimodal spatiotemporal multidimensional data is deeply learned through a multi-layer graph neural network to generate a coding representation that enhances spatiotemporal correlation for further model training.

[0027] Optionally, a spatiotemporal multimodal training method with three-dimensional representation is used to train or fine-tune the spatiotemporal intelligence large model, including:

[0028] Step N1: Based on the multimodal spatiotemporal coding data, combined with specific application scenarios and task requirements, data containing spatiotemporal information is collected or generated to construct a multimodal spatiotemporal training data set; the training data set contains labels, time and space attribute information of the multimodal data, guides the adaptability and accuracy of the spatiotemporal intelligent big model in specific tasks, and trains the spatiotemporal correlation ability of the spatiotemporal intelligent big model;

[0029] Step N2: Use the constructed multimodal spatiotemporal training dataset as input, adopt the multidimensional spatiotemporal attention network model, and train the spatiotemporal intelligent big model through the spatiotemporal multimodal training method to capture the time series relationship and spatial features in the data marked with zero-dimensional points, one-dimensional lines, two-dimensional surfaces, and three-dimensional bodies.

[0030] Optionally, the spatiotemporal multimodal training method refers to: three-dimensional representation of zero-dimensional points, one-dimensional lines, two-dimensional surfaces, and three-dimensional bodies + spatiotemporal multimodal training of time dimension. The underlying layer of the spatiotemporal intelligent big model adopts a three-dimensional representation method, based on three-dimensional spatial expression and time dimension, and combines the time series and spatial characteristics of the multimodal spatiotemporal multidimensional data for modeling, so as to more accurately capture the complex relationship between time and space, reflect the dynamic changes of space and time in the real world, and improve the performance of the spatiotemporal intelligent big model in space-time reasoning.

[0031] Optionally, when constructing the multimodal spatiotemporal training dataset, the diversity and coverage of the data are mainly obtained to ensure the robustness and generalization ability of the spatiotemporal intelligent model in different situations; wherein, the text data includes various types of content, including but not limited to natural language descriptions, commands, and dialogues, covering a wide range of semantics and different complexities; the image data includes but is not limited to static and dynamic images from different perspectives, different environments, and different time periods; the geographic spatial data includes but is not limited to formatted spatial data of different industries and different precisions, and constructs spatial relationships related to the spatiotemporal data.

[0032] Optionally, the spatiotemporal intelligence capability provided by the spatiotemporal intelligence big model reasoning service is based on the spatial relationship expressed by the spatiotemporal objects. During the spatiotemporal intelligence big model reasoning process, the topological relationship between objects recorded by the spatiotemporal objects is used and output, including object-related knowledge graphs and business process logic, to reduce the computational complexity of the spatiotemporal intelligence big model reasoning and efficiently complete the reasoning task.

[0033] Optionally, the spatiotemporal relationship expression includes a topological spatial relationship, a sequential spatial relationship, a metric relationship, and a time tag or timestamp, wherein the topological spatial relationship includes: association, adjacency, and inclusion, the sequential spatial relationship includes: orientation, front, back, left, and right, and the metric relationship includes: length and angle;

[0034] The semantic understanding capability is to provide analysis and recognition of spatiotemporal objects based on visual information processing or digital input, and output spatiotemporal objects with zero-dimensional point, one-dimensional line, two-dimensional surface, three-dimensional body features, as well as spatial relationships between objects and objects, and between objects and the environment;

[0035] The ability of environmental perception is based on the expression of spatiotemporal objects, understanding and predicting the relative relationship and change trend between spatiotemporal objects, and providing support for further spatiotemporal relationship reasoning applications;

[0036] The ability of spatiotemporal reasoning is based on the ability of environmental perception, understanding the spatial and temporal environment of objects in the real world or the contextual scene of multimedia information and virtual environment, and reasoning and analyzing the association relationship between objects to generate spatial object features and association relationship trends of future temporal features;

[0037] The spatial decision-making capability is based on the semantic understanding, the environmental perception, and the spatiotemporal reasoning, combined with specific spatiotemporal scenarios and business processes, to achieve spatial decision-making applications in spatiotemporal scenarios through spatiotemporal reasoning analysis or business process learning.

[0038] The method for constructing a universal artificial spatiotemporal intelligent big model of the present invention first uses a multi-dimensional object abstract expression method to process various objects and spatial relationships in the real world, and establishes a structured spatiotemporal object expression and data storage that can be uniformly processed by the spatiotemporal intelligent big model; then a multimodal spatiotemporal coding algorithm is used to process the spatiotemporal objects, and the spatiotemporal data corresponding to the spatiotemporal objects are converted into coded data for subsequent spatiotemporal intelligent big model training and application; then based on the multimodal spatiotemporal coding data, a multimodal spatiotemporal training data set is constructed, and a three-dimensional representation of the spatiotemporal multimodal training method is used to train or fine-tune the spatiotemporal intelligent big model; finally, the trained spatiotemporal intelligent big model is published as a spatiotemporal intelligent big model inference service to provide spatiotemporal intelligence capabilities.

[0039] The method for constructing a general artificial spatiotemporal intelligence big model (which can be defined as: Artificial General SpatioTemporal Intelligence, referred to as AGSTI) proposed in the present invention is based on the object expression and spatial relationship model in the form of spatial zero-dimensional points, one-dimensional lines, two-dimensional surfaces, and three-dimensional bodies, to establish a unified structured topological expression and storage of various types of multimedia information such as text, graphics, images, audio, video, etc. in the real world of human society, and adopts a multimodal big model training method with three-dimensional representation to construct an artificial spatiotemporal intelligence big model that can uniformly express and process all objects in human society. Combined with spatiotemporal information processing, spatiotemporal relationship analysis, knowledge graph and business process logical reasoning, it provides spatiotemporal relationship expression, semantic understanding, environmental perception, spatiotemporal reasoning and spatial decision-making capabilities, solves the problem that the current large language model has a single expression, lacks topological relationships, and is difficult to process the spatiotemporal relationships in the three-dimensional real world. The general artificial spatiotemporal intelligence big model constructed based on the present invention can greatly improve the decision-making accuracy and adaptability of intelligent systems in complex scenarios, and can be fully applied to application fields such as multimedia information processing, theoretical model derivation, autonomous driving, intelligent robots, industrial engineering, etc., providing new technical support for these application fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0041] Figure 1 It is a flow chart of a method for constructing a general artificial spatiotemporal intelligence large model according to an embodiment of the present invention;

[0042] Figure 2 It is a schematic diagram of an exemplary abstract expression method of real-world objects in an embodiment of the present invention. DETAILED DESCRIPTION

[0043] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention, are only part of the embodiments of the present invention, not all of the embodiments, and are not used to limit the present invention.

[0044] The method for constructing the general artificial spatiotemporal intelligence large model proposed in the present invention refers to Figure 1 A flowchart of a method for constructing a general artificial spatiotemporal intelligence large model is shown, and the method comprises:

[0045] Step 101: Use a multi-dimensional object abstraction expression method to process various objects and spatial relationships in the real world, and establish a structured spatio-temporal object expression and data storage that can be uniformly processed by the spatio-temporal intelligent large model. Multi-dimensional object abstraction refers to: zero-dimensional points, one-dimensional lines, two-dimensional planes, and three-dimensional bodies.

[0046] The present invention creatively proposes the theory of multi-dimensional object abstraction, which includes: zero-dimensional points, one-dimensional lines, two-dimensional planes, and three-dimensional bodies. To better understand the technical solution of the present invention, the theory of this multi-dimensional object abstraction will be elaborated in detail first.

[0047] The multi-dimensional object abstraction proposed by the present invention analyzes and abstracts all kinds of real objects in human society, as well as data such as text, graphics, images, audio, and video obtained from the cognition of various objects, into structured zero-dimensional points, one-dimensional lines, two-dimensional planes, and three-dimensional bodies. By adopting a three-dimensional representation model expression and training method, it solves the limitation problem of the large language model based on text expression from the data model expression and training level, and optimizes the computing power requirements for model training and reasoning.

[0048] The so-called various objects in the real world include: real geographical space objects, virtual geographical space objects, and multimedia type objects with relative space in multimedia context. The multimedia type objects include: various types such as text, graphics, images, audio, and video. To better understand the above theory, some specific examples are used for understanding:

[0049] (1) Zero-dimensional points: They are the basic units of the real world, such as vector nodes, the number "1", the letter "A", the Chinese character "Mao", physical entities such as "a tree, a rose, a person, a tiger, a picture, a snapshot, auditory and olfactory information at a certain time point, a part or a building component", and mathematical operation symbols such as "the product symbol ∏, the summation symbol ∑, or the sine symbol sin". These all participate in the expression and training of the model in the form of zero-dimensional points in the general artificial spatio-temporal intelligent large model of the present invention.

[0050] Combined with the above examples, the zero-dimensional point can be understood as an object-oriented modeling expression of the basic unit of the real world. Among them, the object of the basic unit refers to various physical objects with spatial positions in the real world, as well as multimedia information objects, and multimedia information objects include: text, graphics, images, audio, video, etc. For the spatial position, it includes the absolute coordinates and local coordinates of various objects, as well as the relative sequence number and page number and relative coordinates of the scene and context where the object is located; for various physical objects, it is expressed through the spatial position, environment and spatial relationship of the object. For multimedia information objects, it is expressed through the digital features, scene or context, and relative space of the object, including but not limited to the serial position of the text in the paragraph, the page number in the chapter and the correlation relationship with other texts, as well as the serial position of the image frame in the video and the correlation relationship with other frames.

[0051] (2) One-dimensional lines: such as vector arc segments, a string of numbers or characters, a phrase or a sentence, physical entities such as "a row of trees, a row of roses, a row of people or a single person walking or running, a row of tigers or a single tiger walking or running, a video, a piece of auditory or olfactory information", etc., subsystems formed by multiple parts or building components, local calculation formulas, etc. In the general artificial spatiotemporal intelligence large model of the present invention, all of these participate in the expression and training of the model in the form of one-dimensional lines.

[0052] Combined with the above examples, a one-dimensional line can be understood as a modeling expression of the change and evolution process of real-world objects in the time dimension, including: expressing the continuous change process or trajectory of an object in the time and space dimensions as a one-dimensional line, or expressing a data sequence of zero-dimensional points or multiple zero-dimensional point objects of the same type as a one-dimensional line, and combining topological node features to record its flow process, logical relationship, and topological adjacency in time and space. Among them, data sequences include but are not limited to coordinate sequences, character sequences, text paragraphs, mathematical formulas, image sets, video frames, etc.

[0053] (3) Two-dimensional surfaces: such as vector polygons, multiple or multiple data or character strings, physical entities such as "a forest of trees, a rose bush, a crowd of people, a herd of tigers, a complete film narrative or a complete TV video, a set of staged auditory and olfactory information", a machine or building with complete functions, a complete calculation or deduction process (a combination of multiple formulas), etc.

[0054] Combined with the above examples, the two-dimensional surface can be understood as a modeling expression of the adjacency and aggregation of real-world objects, including: expressing multiple zero-dimensional points and one-dimensional line objects of the same type as two-dimensional surfaces, recording their adjacency relationships in time and space, and using them for larger-scale data analysis and spatiotemporal object modeling; when processing real-world object associations and object aggregations, the basic unit's own characteristics and complex derivation processes or combined association characteristics of multiple data are fully recorded according to the object's position, type or change process in time or space dimensions.

[0055] (4) Three-dimensional objects: such as vector polyhedrons, data sets or text collections, parks or communities, zoos, physical entities such as "a piece of farmland or forest or grassland, a movie or TV series", workshops or factories, buildings, a book on mathematics, physics and chemistry, etc.

[0056] Combined with the above examples, the three-dimensional volume can be understood as a modeling expression of the overall spatiotemporal evolution of large-scale data sets or complex systems in real-world objects, including: representing large data sets of multi-dimensional zero-dimensional points, one-dimensional lines, and two-dimensional surfaces as a complex body or composite process, and recording their overall relationship or evolution process in space and time. Combining objects and events in multiple space-time dimensions to generate an overall space-time volume structure supports comprehensive space-time analysis and prediction of complex systems.

[0057] The above theory of multidimensional object abstraction can be combined with Figure 2 A schematic diagram of an exemplary abstract expression of a real-world object is shown to provide a more intuitive understanding: a physical object in a real-world object ( Figure 2 The left side shows the number "1", ... a string of numbers, ... multiple segments of data, ... a data set or a text set...), depending on their type, they can be expressed as a unified spatiotemporal expression: zero-dimensional points, one-dimensional lines, two-dimensional surfaces, three-dimensional bodies, corresponding to vector nodes, arcs, polygons, polyhedrons, with temporal attributes, and multiple relationships such as spatial relationships, topology, order, and measurement.

[0058] The above is a detailed explanation and description of the theory of multi-dimensional object abstraction. The construction of the general artificial spatiotemporal intelligence model (which can be defined as: Artificial General SpatioTemporal Intelligence, referred to as AGSTI) proposed in the present invention first needs to use a multi-dimensional object abstract expression method to process various objects and spatial relationships in the real world, and establish a structured spatiotemporal object expression and data storage that can be uniformly processed by the spatiotemporal intelligence model. The preferred methods include:

[0059] First, data of various objects in the real world are collected, including but not limited to multimedia information formats; then the spatiotemporal information features of various objects in the multimedia information formats are identified and analyzed; then, the environment, scene or context, relative space, and knowledge graph association and spatial relationship of various objects are combined to generate multimodal, structured spatiotemporal object expressions of zero-dimensional points, one-dimensional lines, two-dimensional surfaces, and three-dimensional bodies. Among them, multimedia information formats include: text format, graphic format, image format, audio format, and video format; spatiotemporal information features include: position, form, state, and tense.

[0060] The so-called spatiotemporal object expression has the characteristics of location, temporal state and topological relationship at the same time, which can ensure the consistency and relevance of objects in time and space. Spatiotemporal objects can be expressed through topological relationship, orientation or order relationship and metric relationship, among which topological relationship includes: association, adjacency and inclusion; orientation or order relationship includes: up and down, left and right, east and west, south and north; metric relationship includes: distance, angle or near and far.

[0061] Step 102: The spatiotemporal object is processed using a multimodal spatiotemporal coding algorithm to convert the spatiotemporal data corresponding to the spatiotemporal object into multimodal spatiotemporal coding data for subsequent spatiotemporal intelligent large model training and application.

[0062] After establishing a structured spatiotemporal object expression and data storage that can be uniformly processed by the spatiotemporal intelligent big model, it is necessary to use a multimodal spatiotemporal coding algorithm to process the spatiotemporal objects, so as to convert the spatiotemporal data corresponding to the spatiotemporal objects into multimodal spatiotemporal coding data for subsequent spatiotemporal intelligent big model training and application. The better methods include:

[0063] Step T1: Based on the spatiotemporal objects, each basic object in the spatiotemporal data is regarded as a node of the spatiotemporal topological graph, and the spatiotemporal adjacency between the nodes is regarded as the edge in the graph. By calculating the spatiotemporal distance or correlation between the objects, the edge weights between the nodes are defined to construct the spatiotemporal topological graph structure; the correlation between the objects is determined according to the similarity, proximity or time sequence in the spatiotemporal data, and the spatiotemporal data refers to multimodal spatiotemporal multidimensional data;

[0064] Step T2: Encode data based on graph neural network, build pre-training tasks through spatial association, and express the spatiotemporal topological graph structure in high dimensions.

[0065] For high-dimensional expression, a better method includes: first, by aggregating the information of spatiotemporal topological graph nodes and neighboring nodes, updating the spatiotemporal features of each node, and capturing the spatial and temporal correlations between the multimodal spatiotemporal multidimensional data; then, performing deep learning on the multimodal spatiotemporal multidimensional data through a multi-layer graph neural network to generate a coded representation with enhanced spatiotemporal correlation for further model training.

[0066] Step 103: Based on the multimodal spatiotemporal coding data, a multimodal spatiotemporal training data set is constructed, and a spatiotemporal multimodal training method with three-dimensional representation is adopted to train or fine-tune a large spatiotemporal intelligent model.

[0067] After converting the spatiotemporal data into multimodal spatiotemporal coding data for subsequent training and application of the spatiotemporal intelligent big model, a multimodal spatiotemporal training data set can be constructed based on the multimodal spatiotemporal coding data, and then based on this multimodal spatiotemporal training data set, a three-dimensional representation of the spatiotemporal multimodal training method is used to train or fine-tune to form a spatiotemporal intelligent big model.

[0068] A better method for training or fine-tuning a large spatiotemporal intelligence model includes:

[0069] Step N1: Based on the multimodal spatiotemporal coding data, combined with specific application scenarios and task requirements, data containing spatiotemporal information is collected or generated to construct a multimodal spatiotemporal training data set; the training data set contains the labels, time and space attribute information of the multimodal data, so as to guide the adaptability and accuracy of the spatiotemporal intelligent big model in specific tasks, so as to train the spatiotemporal correlation ability of the spatiotemporal intelligent big model;

[0070] Step N2: Use the constructed multimodal spatiotemporal training dataset as input, adopt the multidimensional spatiotemporal attention network model, and train the spatiotemporal intelligent big model through the spatiotemporal multimodal training method to capture the time series relationship and spatial features in the data marked with zero-dimensional points, one-dimensional lines, two-dimensional surfaces, and three-dimensional bodies.

[0071] The spatiotemporal multimodal training method in step N2 refers to: 3D representation of zero-dimensional points, one-dimensional lines, two-dimensional surfaces, and three-dimensional bodies + spatiotemporal multimodal training in the time dimension, which is different from the training of traditional language models or multimodal language models. The bottom layer of traditional model training is usually based on sequence length and attention mechanism, and uses tokenized one-dimensional representation. The two are completely different training ideas and architectures.

[0072] The underlying layer of the spatiotemporal intelligent big model proposed in the present invention adopts a three-dimensional representation method, based on three-dimensional spatial expression and time dimension, and combines the time series and spatial characteristics of multimodal spatiotemporal multidimensional data for modeling, so as to more accurately capture the complex relationship between time and space, reflect the dynamic changes of spatiotemporal in the real world, and improve the performance of the spatiotemporal intelligent big model in spatiotemporal reasoning.

[0073] When constructing a multimodal spatiotemporal training dataset, the main focus is on obtaining data diversity and coverage to ensure the robustness and generalization of the spatiotemporal intelligence model in different situations. Among them, text data includes various types of content, including but not limited to natural language descriptions, commands, dialogues, etc., which cover a wide range of semantics and different complexities; image data includes but is not limited to static and dynamic images from different perspectives, different environments, and different time periods; geospatial data includes but is not limited to formatted spatial data of different industries and different precisions, etc., and these geospatial data also construct spatial relationships related to spatiotemporal data.

[0074] Step 104: Publish the trained spatiotemporal intelligence big model as a spatiotemporal intelligence big model reasoning service to provide spatiotemporal intelligence capabilities, which include: spatiotemporal relationship expression, semantic understanding, environmental perception, spatiotemporal reasoning, and spatial decision-making.

[0075] After training or fine-tuning to form a spatiotemporal intelligence big model, that is, after the spatiotemporal intelligence big model is trained, it can be released as a spatiotemporal intelligence big model reasoning service, thereby providing spatiotemporal intelligence capabilities. The so-called spatiotemporal intelligence capabilities include: spatiotemporal relationship expression, semantic understanding, environmental perception, spatiotemporal reasoning and spatial decision-making.

[0076] The spatiotemporal intelligence capability provided by the spatiotemporal intelligence big model reasoning service is based on the spatial relationship expressed by the spatiotemporal objects. In the spatiotemporal intelligence big model reasoning process, the topological relationship between objects recorded by the spatiotemporal objects is used and output, including the object-related knowledge graph and business process logic, which can reduce the computational complexity of the spatiotemporal intelligence big model reasoning, thereby efficiently completing the reasoning task. Among them:

[0077] The so-called spatiotemporal relationship expressions include: topological spatial relationships, sequential spatial relationships, metric relationships, and time tags or timestamps. Topological spatial relationships include: association, adjacency, inclusion, etc., sequential spatial relationships include: orientation, front, back, left, right, etc., and metric relationships include: length, angle, etc.

[0078] The so-called semantic understanding ability is to provide analysis and recognition of spatiotemporal objects based on visual information processing or digital input, and output spatiotemporal objects with zero-dimensional point, one-dimensional line, two-dimensional surface, and three-dimensional body features, as well as the spatial relationship between objects and objects, and between objects and the environment.

[0079] The so-called environmental perception ability is based on the expression of spatiotemporal objects, understanding and predicting the relative relationship and change trend between spatiotemporal objects, providing support for further spatiotemporal relationship reasoning applications.

[0080] The so-called spatiotemporal reasoning ability is based on the aforementioned environmental perception ability, understanding the spatial and temporal environment of objects in the real world or the contextual scenes and virtual environments of multimedia information, and reasoning and analyzing the relationships between objects to generate spatial object features and relationship trends of future temporal characteristics.

[0081] The so-called spatial decision-making capability is based on the aforementioned semantic understanding, environmental perception, and spatiotemporal reasoning, combined with specific spatiotemporal scenarios and business processes, and through spatiotemporal reasoning analysis or business process learning, to realize spatial decision-making applications in spatiotemporal scenarios.

[0082] The general artificial spatiotemporal intelligence model proposed in the present invention has a wide range of specific applications, and can be used in all neighborhoods that require the above-mentioned spatiotemporal intelligence capabilities, for example:

[0083] In multimedia information processing, multimedia data is converted into the spatial expression of the spatiotemporal intelligent big model to generate semantic understanding and reasoning results, and achieve answers that are consistent with the context and have strong relevance. That is, in multimedia information processing application scenarios, the problem is understood and semantically parsed based on spatiotemporal data, and the multimedia input is converted into a spatiotemporal object representation with spatiotemporal associations. The spatiotemporal intelligent big model is used to analyze the temporal and spatial characteristics of the problem and generate answers that conform to the spatiotemporal context. Combined with multimodal data, reasoning results with spatiotemporal accuracy are provided during the question-answering process to ensure the consistency of answers in time and space. For question-answering tasks with time dependence, the spatiotemporal intelligent big model can infer possible future scenarios based on historical spatiotemporal data and provide predictive answers.

[0084] For autonomous driving scenarios, the spatiotemporal data of each object in the environment is processed and analyzed in real time based on the spatiotemporal intelligent big model; the motion trajectory of surrounding objects is predicted through the spatiotemporal intelligent big model, and path planning and driving decisions are generated in combination with the current spatiotemporal information. Multimodal data is used to perceive the vehicle's surrounding environment in all directions, forming a driving scenario analysis with spatiotemporal correlation, supporting autonomous driving decisions in complex road conditions and dynamic environments. During the autonomous driving process, potential dangers that are about to occur can also be predicted, and driving safety and response capabilities can be improved by adjusting driving strategies in real time. Applied in the autonomous driving system, by analyzing and processing the spatiotemporal data in the environment, the motion trajectory and state changes of surrounding objects can be identified and predicted to support driving decisions.

[0085] For intelligent robot scenarios, the multimodal spatiotemporal data in the robot's environment is processed in real time based on the spatiotemporal intelligent big model to improve the robot's perception of environmental changes. The spatiotemporal intelligent big model is used to plan the robot's path and make autonomous decisions, so that it can accurately judge the spatiotemporal relationship of objects in complex environments and make reasonable action responses. The spatiotemporal intelligent big model is used to predict behavior, helping the robot understand the spatiotemporal change trends of surrounding objects and support the execution and dynamic adjustment of tasks. When performing tasks, the robot makes real-time adjustments through the spatiotemporal intelligent big model to ensure that its decision-making process conforms to the temporal and spatial characteristics of objects in the environment, improving autonomy and the accuracy of task completion. By processing and analyzing the multimodal data of the environment in real time, the robot's perception and autonomous decision-making capabilities of complex spatiotemporal environments are improved.

[0086] In industrial engineering application scenarios, the spatiotemporal intelligent big model can intelligently process multi-dimensional engineering thematic graphics and documents, etc. In short, the general artificial spatiotemporal intelligent big model proposed in the present invention can be well applied to all fields and scenarios that require the use of the above-mentioned spatiotemporal intelligent capabilities.

[0087] To summarize, the method for constructing a general artificial spatiotemporal intelligent big model proposed in the present invention is based on the object expression and spatial relationship model in the form of spatial zero-dimensional points, one-dimensional lines, two-dimensional surfaces, and three-dimensional bodies, to establish a unified structured topological expression and storage of various types of multimedia information such as text, graphics, images, audio, video, etc. in the real world of human society, and adopts a three-dimensional representation multimodal big model training method to construct an artificial spatiotemporal intelligent big model that can uniformly express and process all objects in human society. It combines spatiotemporal information processing, spatiotemporal relationship analysis, knowledge graphs, and business process logical reasoning to provide spatiotemporal relationship expression, semantic understanding, environmental perception, spatiotemporal reasoning, and spatial decision-making capabilities, and solves the problem that the current large language model has a single expression, lacks topological relationships, and is difficult to process the spatiotemporal relationships of the three-dimensional real world. It can be fully applied to multimedia information processing, theoretical model derivation, autonomous driving, intelligent robots, industrial engineering and other fields, and has good practicality.

[0088] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0089] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0090] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation modes, which are merely illustrative rather than restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are within the protection of the present invention.

Claims

1. A method for constructing a general artificial spatiotemporal intelligence large model, characterized in that: include: Adopting multi-dimensional object abstract expression, various objects and spatial relationships in the real world are processed, and a spatio-temporal intelligent large model is established to uniformly process structured spatio-temporal object expression and data storage. The multi-dimensional object abstraction refers to: zero-dimensional point, one-dimensional line, two-dimensional surface, three-dimensional body; The spatiotemporal object is processed using a multimodal spatiotemporal coding algorithm to convert the spatiotemporal data corresponding to the spatiotemporal object into multimodal spatiotemporal coding data for subsequent spatiotemporal intelligent large model training and application; Based on the multimodal spatiotemporal coding data, a multimodal spatiotemporal training data set is constructed, and a spatiotemporal multimodal training method with three-dimensional representation is adopted to train or fine-tune the spatiotemporal intelligent large model; The trained spatiotemporal intelligence big model is published as a spatiotemporal intelligence big model reasoning service to provide spatiotemporal intelligence capabilities, including: spatiotemporal relationship expression, semantic understanding, environmental perception, spatiotemporal reasoning, and spatial decision-making; The various objects in the real world include: real geographic space objects in the real world, virtual geographic space objects, and multimedia type objects in the multimedia context's own relative space, and the multimedia type objects include: text, graphics, images, audio, and video types; The zero-dimensional point is an object-oriented modeling expression of the basic unit of the real world; the object of the basic unit refers to various physical objects with spatial positions in the real world, as well as multimedia information objects, and the multimedia information objects include: text, graphics, images, audio, and video; for the various physical objects, they are expressed through the spatial position, environment, and spatial relationship of the objects; for the multimedia information objects, they are expressed through the digital features, scenes or contexts, and relative spaces of the objects, including but not limited to the sequence position of the text in the paragraph and the correlation relationship with other texts, graphic vector nodes, and the sequence position of the image frame in the video and the correlation relationship with other frames; The one-dimensional line is a modeling expression of the change and evolution process of real-world objects in the time dimension, including: expressing the continuous change process or trajectory of the object in the time and space dimensions as a one-dimensional line, or expressing the data sequence of zero-dimensional points, multiple zero-dimensional point objects of the same type as a one-dimensional line, recording their flow process, logical relationship and topological adjacency relationship in time and space; wherein the data sequence includes but is not limited to a coordinate sequence, a character sequence, a text paragraph, a mathematical formula, a graphic vector arc segment, an image set, and a video frame; The two-dimensional surface is a modeling expression of the adjacency and aggregation of real-world objects, including: expressing similar one-dimensional line objects as two-dimensional surfaces, recording their adjacency relationships in time and space, and using them for data analysis and association modeling on a larger scale; when processing the adjacency and aggregation of real-world objects, completely recording the basic unit's own characteristics and the complex derivation process or the combined association characteristics of multiple data according to the object's position, type or change process in time or space dimensions; The three-dimensional body is a modeling expression of the overall spatiotemporal evolution of large-scale data sets or complex systems in real-world objects, including: representing large data sets of multi-dimensional zero-dimensional points, one-dimensional lines, and two-dimensional surfaces as complex bodies or complex processes, and recording their overall relationships or evolution processes in space and time; combining objects and events in multiple space-time dimensions to generate an overall space-time body structure to support comprehensive space-time analysis and prediction of complex systems.

2. The construction method according to claim 1, characterized in that: Use multi-dimensional object abstract expression to process various objects and spatial relationships in the real world, including: Collect data of various objects in the real world, including but not limited to multimedia information formats, identify and analyze the spatiotemporal information features of various objects in the multimedia information formats, and generate multimodal, structured spatiotemporal object expressions of zero-dimensional points, one-dimensional lines, two-dimensional surfaces, and three-dimensional bodies in combination with the environment, scene or context, relative space, and knowledge graphs and spatial relationships of various objects. The multimedia information formats include: text format, graphic format, image format, audio format, and video format. The spatiotemporal information features include: position, form, state, and tense. Among them, the expression of spatiotemporal objects has the characteristics of position, tense and spatial relationship at the same time, ensuring the consistency and correlation of objects in time and space dimensions; the spatiotemporal objects are expressed through topological relationships, orientation or order relationships, and metric relationships. The topological relationships include: association, adjacency, and inclusion. The orientation or order relationships include: up, down, left, right, east, west, south, and north. The metric relationships include: distance, angle, or near.

3. The construction method according to claim 1, characterized in that: The multimodal spatiotemporal coding algorithm is used for processing to convert the spatiotemporal data corresponding to the spatiotemporal object into multimodal spatiotemporal coding data for subsequent spatiotemporal intelligent large model training and application, including: Step T1: Based on the spatiotemporal object, each basic object in the spatiotemporal data is regarded as a node of a spatiotemporal topological graph, and the spatiotemporal adjacency between each node is used as an edge in the graph. By calculating the spatiotemporal distance or correlation between objects, the edge weights between nodes are defined to construct a spatiotemporal topological graph structure; wherein the correlation between objects is determined according to the similarity, proximity or time sequence in the spatiotemporal data, and the spatiotemporal data refers to multimodal spatiotemporal multidimensional data; Step T2: Encode data based on graph neural network, construct pre-training tasks through spatial association, and express the spatiotemporal topological graph structure in high dimension.

4. The construction method according to claim 3, characterized in that: Data encoding is performed based on graph neural networks, pre-training tasks are constructed through spatial association, and the spatiotemporal topological graph structure is expressed in high dimensions, including: By aggregating information of nodes and neighboring nodes in the spatiotemporal topological graph, the spatiotemporal features of each node are updated to capture the spatial and temporal associations between the multimodal spatiotemporal multidimensional data; The multimodal spatiotemporal multidimensional data is deeply learned through a multi-layer graph neural network to generate a coding representation that enhances spatiotemporal correlation for further model training.

5. The construction method according to claim 1, characterized in that: The spatiotemporal multimodal training method with three-dimensional representation is used to train or fine-tune the spatiotemporal intelligent large model, including: Step N1: Based on the multimodal spatiotemporal coding data, combined with specific application scenarios and task requirements, data containing spatiotemporal information is collected or generated to construct a multimodal spatiotemporal training data set; the training data set contains labels, time and space attribute information of the multimodal data, guides the adaptability and accuracy of the spatiotemporal intelligent big model in specific tasks, and trains the spatiotemporal correlation ability of the spatiotemporal intelligent big model; Step N2: Use the constructed multimodal spatiotemporal training dataset as input, adopt the multidimensional spatiotemporal attention network model, and train the spatiotemporal intelligent big model through the spatiotemporal multimodal training method to capture the time series relationship and spatial features in the data marked with zero-dimensional points, one-dimensional lines, two-dimensional surfaces, and three-dimensional bodies.

6. The construction method according to claim 5, characterized in that: The spatiotemporal multimodal training method refers to: three-dimensional representation of zero-dimensional points, one-dimensional lines, two-dimensional surfaces, and three-dimensional bodies + spatiotemporal multimodal training of time dimension. The spatiotemporal intelligent big model adopts a three-dimensional representation method at the bottom layer, based on three-dimensional spatial expression and time dimension, and combines the time series and spatial characteristics of multimodal spatiotemporal multidimensional data for modeling, so as to more accurately capture the complex relationship between time and space, reflect the dynamic changes of space and time in the real world, and improve the performance of the spatiotemporal intelligent big model in space-time reasoning.

7. The construction method according to claim 5, characterized in that: When constructing the multimodal spatiotemporal training dataset, the main focus is on obtaining data diversity and coverage to ensure the robustness and generalization ability of the spatiotemporal intelligence model in different situations; among them, text data includes various types of content, including but not limited to natural language descriptions, commands, and dialogues, covering a wide range of semantics and different complexities; image data includes but is not limited to static and dynamic images from different perspectives, different environments, and different time periods; geographic spatial data includes but is not limited to formatted spatial data of different industries and different precisions, and constructs spatial relationships related to spatiotemporal data.

8. The construction method according to claim 1, characterized in that: The spatiotemporal intelligence capability provided by the spatiotemporal intelligence big model reasoning service is based on the spatial relationship expressed by the spatiotemporal objects. During the spatiotemporal intelligence big model reasoning process, the topological association relationship between objects recorded by the spatiotemporal objects is used and output, including object-related knowledge graphs and business process logic, to reduce the computational complexity of the spatiotemporal intelligence big model reasoning and efficiently complete the reasoning task.

9. The construction method according to claim 8, characterized in that: The spatiotemporal relationship expression includes topological spatial relationship, sequential spatial relationship, metric relationship, and time mark or timestamp. The topological spatial relationship includes: association, adjacency, and inclusion. The sequential spatial relationship includes: orientation, front, back, left, and right. The metric relationship includes: length and angle. The semantic understanding capability is to provide analysis and recognition of spatiotemporal objects based on visual information processing or digital input, and output spatiotemporal objects with zero-dimensional point, one-dimensional line, two-dimensional surface, three-dimensional body features, as well as spatial relationships between objects and objects, and between objects and the environment; The ability of environmental perception is based on the expression of spatiotemporal objects, understanding and predicting the relative relationship and change trend between spatiotemporal objects, and providing support for further spatiotemporal relationship reasoning applications; The ability of spatiotemporal reasoning is based on the ability of environmental perception, understanding the spatial and temporal environment of objects in the real world or the contextual scene of multimedia information and virtual environment, and reasoning and analyzing the association relationship between objects to generate spatial object features and association relationship trends of future temporal features; The spatial decision-making capability is based on the semantic understanding, the environmental perception, and the spatiotemporal reasoning, combined with specific spatiotemporal scenarios and business processes, to achieve spatial decision-making applications in spatiotemporal scenarios through spatiotemporal reasoning analysis or business process learning.

Images