A spatiotemporal context extraction method and system for real-world model training

Abstract

The present invention relates to the field of artificial intelligence technology, and in particular to a spatiotemporal context extraction method and system for real-world model training. The method extracts all image information and text semantic information through content feature vectors, extracts structural information such as the spatial layout of the scene in the captured image, the relative positions of people and objects through spatial structure feature vectors, extracts feature information such as the appearance, posture, and movement of the target object through entity feature vectors, and obtains time series feature vectors based on the content feature vectors, spatial structure feature vectors, and entity feature vectors within multiple preset time periods. The method captures the multi-dimensional dynamic changes of the target physical place over a period of time, analyzes the extracted vectors to obtain scene analysis feature data, and integrates all feature information to obtain spatiotemporal context, thereby improving the richness and comprehensiveness of the representation of the target physical place and ensuring that the real-world model obtained through subsequent training has a better representation degree and adaptability to the target physical place.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a spatiotemporal context extraction method and system for real-world model training. Background Art

[0002] Building real-world models is crucial for promoting the application of artificial intelligence in areas such as smart cities, autonomous driving, and intelligent security. By digitally simulating real-world scenarios, it can provide accurate decision-making for urban planning, traffic management, and safety control, effectively improving social efficiency and the rationality of resource allocation. Multimodal data fusion, a core technology for building real-world models, integrates multidimensional information such as images and text to fully recreate real scenes, making it the key to building high-precision models.

[0003] However, existing technologies have many shortcomings in multimodal data processing and real-world model construction. At the data collection level, it focuses on the acquisition of single or a few modal data, such as relying solely on image data or text data, which cannot fully cover the information dimensions of the real scene, resulting in a lack of sufficient data support for model construction and difficulty in truly reflecting the complex situations in the real world. In the data processing link, traditional feature extraction methods fail to fully combine the characteristics of different modal data. For image data, it is difficult to accurately extract subtle features in complex scenes, resulting in a low degree of representation of the scene and physical objects such as people and objects in the scene; for text data, there are limitations in semantic understanding and key information extraction, which makes the extracted features unable to effectively represent the essence of the data, reducing the depth of the model's understanding of the scene and physical objects such as people and objects in the scene. In terms of data fusion, most existing technologies adopt simple splicing or shallow fusion strategies, failing to deeply explore the spatiotemporal correlations and semantic connections between multimodal data, and unable to obtain spatiotemporal context with a high degree of scene representation. As a result, the constructed real-world models have a weak understanding of scenes and physical objects. During the model training stage, they cannot deeply consider the representation of scenes and physical objects, and are difficult to adapt to complex changes in different scenes and at different times. They cannot meet the requirements of practical applications for high accuracy and strong adaptability of models, limiting the widespread application and development of real-world models in various fields.

[0004] Therefore, how to improve the representation of the physical place by the extracted spatiotemporal context and thus improve the accuracy and adaptability of the real-world model has become an urgent problem to be solved. Summary of the Invention

[0005] In response to the above technical problems, the technical solution adopted by the present invention is a spatiotemporal context extraction method for real-world model training, which includes the following steps:

[0006] S1, obtain a number of collected data corresponding to the target physical place within M preset time periods, wherein each collected data can be a collected image or a collected text, and M is a positive integer.

[0007] S2. For any preset time period, the collected data corresponding to the current preset time period is encoded according to the preset content encoder, the preset spatial structure encoder and the preset character entity encoder to obtain the content feature vector corresponding to each collected data, the spatial structure feature vector corresponding to each collected image, and the entity feature vector corresponding to each target object in each collected image.

[0008] S3. Based on all content feature vectors, spatial structure feature vectors and entity feature vectors corresponding to the target physical place in the current preset time period and the N preset time periods before the current preset time period, obtain the time series feature vector corresponding to the target physical place in the current preset time period, where N is an integer greater than 0.

[0009] S4, input the content feature vector, spatial structure feature vector and entity feature vector corresponding to the target physical place in the current preset time period into the initial world model, and obtain the scene parsing feature data corresponding to the target physical place in the current preset time period, wherein the scene parsing feature data includes scene description text, question answer text and relationship feature vector list.

[0010] S5, all content feature vectors, spatial structure feature vectors, entity feature vectors, time series feature vectors and scene analysis feature data corresponding to the target physical place in all preset time periods are used as the spatiotemporal context corresponding to the target physical place, wherein the spatiotemporal context is used to perform model distillation on the cloud world model in the cloud to obtain the real-world model corresponding to the target physical place.

[0011] The present invention also provides a spatiotemporal context extraction system for real-world model training, the spatiotemporal context extraction system for real-world model training comprising:

[0012] The acquisition data acquisition module is used to acquire a number of acquisition data corresponding to the target physical place within M preset time periods, wherein each acquisition data can be an acquisition image or an acquisition text, and M is a positive integer.

[0013] The first feature encoding module is used to encode the collected data corresponding to the current preset time period according to the preset content encoder, the preset spatial structure encoder and the preset character entity encoder for any preset time period, and obtain the content feature vector corresponding to each collected data, the spatial structure feature vector corresponding to each collected image, and the entity feature vector corresponding to each target object in each collected image.

[0014] The second feature encoding module is used to obtain the time series feature vector corresponding to the target physical place in the current preset time period based on all content feature vectors, spatial structure feature vectors and entity feature vectors corresponding to the target physical place in the current preset time period and N preset time periods before the current preset time period, where N is an integer greater than 0.

[0015] The scene parsing module is used to input the content feature vector, spatial structure feature vector and entity feature vector corresponding to the target physical place in the current preset time period into the initial world model, and obtain the scene parsing feature data corresponding to the target physical place in the current preset time period, wherein the scene parsing feature data includes scene description text, question answer text and relationship feature vector list.

[0016] The spatiotemporal context extraction module is used to take all content feature vectors, spatial structure feature vectors, entity feature vectors, time series feature vectors and scene analysis feature data corresponding to the target physical place in all preset time periods as the spatiotemporal context corresponding to the target physical place, wherein the spatiotemporal context is used to perform model distillation on the cloud world model in the cloud to obtain the real-world model corresponding to the target physical place.

[0017] The present invention has at least the following beneficial effects: by collecting data in two modes, image and text, the state information of the target physical place is covered from the two dimensions of visual information and semantic information, which makes up for the limitations of single-modality data, and by collecting data within multiple preset time periods, it can record the dynamic changes of the target physical place over time. The preset content encoder converts the original collected image and collected text into content feature vectors, realizing the transformation of data from original form to feature and abstraction, removing redundant information in the collected data and retaining key features, extracting structural information such as the spatial layout of the scene in the collected image, the relative positions of people and objects, etc. by encoding the spatial structure feature vector, characterizing the static layout features of the target physical place, and encoding the target object by encoding the entity feature vector It encodes the appearance, posture, and movement of the target physical place, further integrates the overall character state characteristics, all image information and text semantic information, and the spatial structure information between all target objects in the target physical place within multiple preset time periods to obtain a temporal feature vector, captures the multi-dimensional dynamic changes of the target physical place in terms of content modality, spatial structure modality, and entity modality over a period of time, and parses the extracted vectors to obtain scene parsing feature data, and integrates all feature information to obtain the spatiotemporal context, which improves the richness and comprehensiveness of the representation of the target physical place, so that the real-world model can dynamically adapt to the multi-dimensional feature vectors and dynamic changes of the target physical place, and improves the representation degree and adaptability of the trained real-world model to the target physical place. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0019] Figure 1 A flowchart of a spatiotemporal context extraction method for real-world model training provided in Example 1 of the present invention;

[0020] Figure 2 A flowchart of a real-world model training method based on a local physical environment provided in the second embodiment of the present invention;

[0021] Figure 3 A schematic diagram of a spatiotemporal context extraction system for real-world model training provided in Example 3 of the present invention. DETAILED DESCRIPTION

[0022] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making any creative efforts shall fall within the scope of protection of the present invention.

[0023] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It is understood that, where appropriate, the above-mentioned terms used to distinguish similar objects can be interchanged so that the present invention can also implement other embodiments other than the above-mentioned illustrated embodiments or described embodiments. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0024] Example 1

[0025] This embodiment provides a spatiotemporal context extraction method for real-world model training, such as Figure 1 As shown in FIG, the spatiotemporal context extraction method for real-world model training includes the following steps:

[0026] S1, obtain a number of collected data corresponding to the target physical place within M preset time periods, wherein each collected data can be a collected image or a collected text, and M is a positive integer.

[0027] Among them, the target physical place is an actual three-dimensional spatial area with a specific spatial structure and functional use. It is the object of research and modeling, such as shopping malls, campuses, factories and other specific spatial areas.

[0028] The preset time period is an artificially set time interval used to divide the data collection phase in order to analyze the changing patterns of the target physical location at different time phases.

[0029] The target physical location corresponds to a number of collected data within each preset time period. Specifically, each collected data can be a collected image in image mode or a collected text in text mode. The collected image is visual information of the target physical location acquired by an image acquisition device such as a camera. It exists in the form of a pixel matrix and can intuitively present information such as the appearance, location, and form of target objects within the location, such as the distribution of customers in a shopping mall or the display layout of stores. The collected text can be derived from text information such as signs, display screens, and receipts within the target location, or it can be converted into text through methods such as voice recognition, and is used to represent semantic information related to the location and the people, objects, and other entities within the location.

[0030] Each piece of collected data represents the state of several target objects within the target physical location during the current time period. Target objects are entities within the target physical location that are of research value, including people, objects, robots, and so on. Target objects exhibit different states over different time periods. By setting multiple time periods, you can capture characteristics of the target physical location, such as periodic changes and long-term trends.

[0031] In one embodiment, S1 includes the following steps:

[0032] S11, obtain several initial data corresponding to the target physical place in each preset time period, wherein each initial data can be a captured image, a captured video or a captured text, and each initial data corresponds to the status of several target objects in the target physical place in the current preset time period.

[0033] S12, sampling each captured video, and taking each image frame obtained by sampling and each captured image in the initial data as the corresponding captured image of the target physical place within the corresponding preset time period.

[0034] Captured images present visual information of the target physical location in static images. Captured videos are continuous visual information of the target physical location recorded by video capture devices such as cameras. They can dynamically record the behavior and changes of target objects within the location, such as customer routes within a shopping mall or store operations. Captured text conveys semantic information about the location through text.

[0035] An image frame is the basic unit of video capture. Each frame is a static image that records the visual information of the captured video at a specific moment, including the state of the target physical location and object at that moment. Sampling rules can be based on time intervals (e.g., selecting a frame every fixed time interval) or frame intervals (e.g., selecting a frame every fixed number of frames). This allows for preserving the key information of the video while reducing the data volume and improving the efficiency of subsequent data processing. The fixed time or fixed frame number for sampling can be set by the implementer based on the actual conditions of the video capture equipment.

[0036] As mentioned above, by collecting data in both image and text modes, the status information of the target physical place is covered from the two dimensions of visual information and semantic information, which makes up for the limitations of single-modality data. In addition, by collecting data within multiple preset time periods, it can record the dynamic changes of the target physical place over time, and provide time series data support for subsequent analysis of the changing laws of the place and prediction of future status.

[0037] S2. For any preset time period, the collected data corresponding to the current preset time period is encoded according to the preset content encoder, the preset spatial structure encoder and the preset character entity encoder to obtain the content feature vector corresponding to each collected data, the spatial structure feature vector corresponding to each collected image, and the entity feature vector corresponding to each target object in each collected image.

[0038] Among them, the preset content encoder is a model component used to extract features and encode collected data. The structure and parameters of the preset content encoder are pre-set and can be constructed based on deep learning architectures such as convolutional neural networks and Transformer. It can automatically learn and extract effective features in the collected data according to the characteristics of the collected data, and convert the original collected data into a more representative vector form.

[0039] Correspondingly, the content feature vector is the output result of encoding the collected data by a preset content encoder, and represents the core content features of the collected data in the form of a vector.

[0040] The spatial structure encoder analyzes the captured image as a whole, extracts structural information such as the spatial layout of the scene, the relative positions of people and objects in the captured image, characterizes the static layout characteristics of the target physical place, and generates the corresponding spatial structure feature vector.

[0041] The character entity encoder encodes the appearance, posture, action, etc. of the target object in the captured image and generates an entity feature vector for each target object.

[0042] For example, in a shopping mall scenario, the spatial structure encoder will analyze information such as store layout and aisle location, while the human entity encoder will extract features such as customers' clothing, posture, movement trajectory, and the color, category, and shape of the items on sale.

[0043] In one embodiment, S2 includes the following steps:

[0044] S21 , for any preset time period, encoding each collected data corresponding to the current preset time period according to a preset content encoder, and obtaining a content feature vector corresponding to each collected data.

[0045] S22, encode each acquired image corresponding to the current preset time period according to the preset spatial structure encoder and the preset character entity encoder, and obtain the spatial structure feature vector corresponding to each acquired image and the entity feature vector corresponding to each target object in each acquired image.

[0046] In a specific embodiment, the preset content encoder includes a preset image encoder and a preset text encoder, and S21 includes the following steps:

[0047] S211 : For any collected data, if the current collected data is a collected image, encode the current collected data according to a preset image encoder to obtain a content feature vector corresponding to the current collected data.

[0048] S212: If the current collected data is collected text, encode the current collected data according to a preset text encoder to obtain a content feature vector corresponding to the current collected data.

[0049] Among them, the preset image encoder can automatically extract the features of the acquired image through components such as convolutional layers, pooling layers, and fully connected layers based on the convolutional neural network architecture in deep learning. Specifically, the convolution kernel in the convolution layer slides on the acquired image, and extracts local features such as edges and textures of the acquired image through convolution operations. The pooling layer is used to compress data, reduce the amount of calculation while retaining the main features. After stacking multiple convolution and pooling layers, higher-level and more abstract image features are gradually extracted. Finally, the extracted image features are mapped to a content feature vector of a fixed length through the fully connected layer. For the content feature vector of the acquired image, each dimension can represent information such as edge feature intensity, texture pattern, color distribution, etc. in the acquired image.

[0050] The text encoder can be based on word embedding technology used in natural language processing and recurrent neural networks or their variants, such as long short-term memory networks and gated recurrent units. First, a word embedding layer captures the semantic and syntactic information of each word in the collected text, converting each word into a low-dimensional word vector representation. Then, a recurrent neural network or its variant processes the word vectors sequentially according to the order of the collected text, capturing the contextual information and long-term dependencies in the collected text. Ultimately, the entire text sequence is encoded into a fixed-length content feature vector. For the content feature vector of the collected text, each dimension can represent the weight of different semantic words, the distribution of text topics, and other semantic and grammatical information.

[0051] As mentioned above, the preset content encoder converts the original collected images and collected texts into content feature vectors, realizing the transformation of data from its original form to a featured and abstract form, removing redundant information in the collected data while retaining key features, and providing high-quality input data for multimodal data fusion and model training.

[0052] In one embodiment, S22 includes the following steps:

[0053] S221 , for any collected image corresponding to the current preset time period, perform dimension conversion on the current collected image to obtain a three-dimensional image corresponding to the current collected image.

[0054] S222: semantically annotate the three-dimensional image corresponding to the current acquired image to obtain an annotated image corresponding to the current acquired image.

[0055] S223 , encoding the annotated image corresponding to the current captured image according to a preset spatial structure encoder, and obtaining a spatial structure feature vector corresponding to the current captured image.

[0056] S224 , performing entity extraction on the annotated image corresponding to the current captured image, and obtaining an entity image corresponding to each target object in the current captured image.

[0057] S225 , encoding the entity image corresponding to each target object in the current captured image according to a preset human entity encoder, and obtaining an entity feature vector corresponding to each target object in the current captured image.

[0058] Among them, the two-dimensional acquired image is converted into a three-dimensional image according to the preset image dimension conversion method, which solves the problem of missing depth information in the two-dimensional image and provides richer geometric information for the subsequent extraction of spatial structural features. Those skilled in the art know that any image dimension conversion method in the prior art falls within the protection scope of the present invention. For example, the preset image dimension conversion method can be to use a deep learning model to predict the depth value of each pixel in the acquired image, generate a pseudo depth map, and then combine it with the original RGB image to form an RGB-D three-dimensional representation, or it can be for multi-angle acquired images, and reconstruct a three-dimensional point cloud through structured light motion or multi-view stereo vision technology to obtain a three-dimensional image. Specifically, the internal and external parameters of the acquisition device corresponding to the acquired image can be obtained, and a conversion matrix from 2D image pixel coordinates to 3D coordinates can be generated according to the internal and external parameters, and the current acquired image can be converted into the corresponding three-dimensional image according to the conversion matrix.

[0059] Perform pixel-level semantic segmentation on the 3D image, labeling different categories of objects, such as people and objects, to provide clear semantic guidance for the spatial structure encoder and the person entity encoder to focus on meaningful person entities and their spatial relationships. In this embodiment, a model such as Mask R-CNN can be used to identify the categories and instance boundaries of person entities in the 3D image, labeling each pixel with the corresponding semantic category. For example, background pixels are labeled as 0, pixels of person A are labeled as 1, pixels of person B are labeled as 2, pixels of object C are labeled as 3, pixels of object D are labeled as 4, pixels of robot E are labeled as 5, and so on.

[0060] By using the masks of people and objects in the semantic segmentation results, the areas where people and objects are located are extracted from the annotated image, and a pure entity image corresponding to each target object is generated, providing high-quality input for the subsequent person entity encoder and reducing background noise interference.

[0061] In this embodiment, the preset spatial structure encoder can adopt a graph neural network architecture, mainly including a multi-scale feature extraction layer, a semantic enhancement module, a spatial relationship modeling layer, and a geometric constraint module. The multi-scale feature extraction layer is used to extract visual features at different levels. For example, low-level features retain spatial details for accurate region positioning, while high-level features capture semantic information to assist in region semantic classification. The semantic enhancement module is used to integrate semantic annotation information, using semantic masks as weights in the attention mechanism to strengthen the understanding of the functional areas of the scene and focus on meaningful spatial regions. The spatial relationship modeling layer is used to construct the scene topology structure and learn the spatial relationships between regions. Specifically, the physical space is converted into a graph structure, with regions as nodes and spatial relationships as edges. The node representation is updated through convolutional propagation to capture spatial dependencies. The geometric constraint module is used to introduce three-dimensional geometric prior knowledge and standardize feature learning through a geometric constraint loss function to ensure the rationality of spatial relationships. The geometric constraints include distance constraints, angle constraints, and inclusion constraints. Specifically, the distance constraint is used to penalize unreasonable distance predictions between regions, the angle constraint is used to ensure that the angle relationship between regions conforms to physical common sense, and the inclusion constraint is used to enforce the learning of the correct region inclusion hierarchy. Finally, all node information is integrated through graph pooling operation to generate the final spatial structure feature vector.

[0062] The pre-set human entity encoder can include a multimodal input fusion layer, an appearance feature extraction branch, a posture and motion analysis branch, and an attribute classification and behavior prediction branch. The multimodal input fusion layer performs pre-processing on the entity image, such as scaling and normalization, before inputting it into the convolutional layer. The convolution operation encodes the coordinates of the joints or corners of people and objects into a heat map or vector representation. The appearance feature extraction branch uses a convolutional layer to extract visual features of the human region and a pooling layer and a fully connected layer to generate the human appearance vector. The posture and motion analysis branch calculates geometric features such as joint length and edge length based on the joints or corners of people and objects. The attribute classification and behavior prediction branch is used to predict and analyze the basic attributes and behavioral states of people and objects. Finally, a gated fusion mechanism is used to integrate the features of multiple branches to obtain the entity feature vector.

[0063] As mentioned above, by encoding the spatial structure feature vector, the structural information such as the spatial layout of the scene, the relative positions of people and objects in the captured image is extracted to characterize the static layout characteristics of the target physical place. By encoding the entity feature vector, the appearance, posture, movement, etc. of the target object are encoded, which complements the content feature vector and jointly constructs a more comprehensive multimodal feature representation.

[0064] In a specific embodiment, the preset content encoder, the preset spatial structure encoder and the preset character entity encoder need to be trained in coordination with the multimodal base model and the adaptation layer to achieve alignment of different modal data in the latent space, thereby supporting the understanding of entities, positional relationships and events in the physical space.

[0065] Collaborative training is divided into two phases, relying on cold-start data and data augmentation strategies. Cold-start data consists of semantically annotated 3D point cloud datasets (such as ScanNet and ShapeNet). This data is combined with multimodal models (such as CLIP and ViT) and specialized models (such as 3D object detection models) for scene simulation and annotation, resulting in a multimodal dataset containing content (images / point clouds), spatial structure (geometric coordinates / topological relationships), and human entities (semantic labels / attributes). Furthermore, the large model generates different scenario scenarios (such as "office meetings" and "kitchen cooking") and replaces objects / characters in the scenes (such as replacing table and chair styles or character clothing) for data augmentation, thereby expanding data diversity and serving as the data foundation for collaborative training.

[0066] In the first stage, the encoder is aligned with the latent space of the multimodal base model, so that the output features of the content encoder, spatial structure encoder and character entity encoder are compatible with the latent space of the multimodal base model and a unified semantic representation is established. The content encoder extracts visual features (such as edges, textures, object categories, and text content) through a convolutional neural network (CNN) or Transformer, and aligns the extracted content feature vectors with the latent vectors output by the base model, minimizing feature distribution differences. This allows the content encoder to map visual information to the latent space of the base model and support cross-modal semantic associations. The spatial structure encoder uses a graph neural network (GNN) or a geometric deep learning model (such as PointNet) to encode spatial coordinates and structural relationships, generate a spatial structure feature vector, and match the spatial structure feature vector with the predefined spatial latent vector in the base model. Using methods such as triplet loss to constrain the features of similar spatial relationships to be close, the spatial structure encoder can represent the positional relationships between people and objects and unify them with the spatial semantics of the base model. The person entity encoder uses a text encoder (such as BERT) or an attribute embedding layer to convert discrete labels into continuous entity feature vectors, compare the entity feature vectors with the entity prototypes stored in the base model, and optimize the feature distribution through cross-entropy loss or metric learning. This allows the person entity encoder to distinguish different identities, actions, and attributes, and align them with the entity semantics of the base model.

[0067] The adaptation layer adds a fully connected layer or attention layer between the encoder and the base module to adjust the feature dimension and distribution to ensure cross-module compatibility.

[0068] The second phase conducts task-oriented training of the overall model. Based on the aligned encoder, the model is trained to complete physical space understanding tasks, such as entity recognition, text answering, relational reasoning, and event comprehension. Specifically, the encoder parameters aligned in the first phase are fixed. Based on cold start data and data augmentation strategies, combined with more scenario scripts and dynamic interaction data, positive and negative samples (such as reasonable spatial layout and unreasonable spatial layout) are constructed. A contrastive loss is used to enhance the model's sensitivity to subtle differences, ultimately achieving a unified representation and reasoning of multi-dimensional information in physical space, completing the task of understanding physical space.

[0069] As mentioned above, by aligning the latent space of the content encoder, spatial structure encoder, and character entity encoder with the multimodal base model, heterogeneous data such as visual content, spatial structure, and semantics are mapped to a unified feature space, solving the problem of traditional models' fragmented processing of multimodal data. This improves the deep understanding and generalization capabilities of physical space, thereby meeting the requirements of practical applications for high accuracy and strong adaptability of the model.

[0070] S3. Based on all content feature vectors, spatial structure feature vectors and entity feature vectors corresponding to the target physical place in the current preset time period and the N preset time periods before the current preset time period, obtain the time series feature vector corresponding to the target physical place in the current preset time period, where N is an integer greater than 0.

[0071] In a specific embodiment, S3 includes the following steps:

[0072] S31, for any preset time period, all content feature vectors, spatial structure feature vectors and entity feature vectors corresponding to the target physical place in the current preset time period are spliced ​​to obtain the scene status feature vector corresponding to the target physical place in the current preset time period.

[0073] S32, splicing and compressing the scene status feature vectors corresponding to the target physical place in the current preset time period and N preset time periods before the current preset time period, and obtaining the time series feature vector corresponding to the target physical place in the current preset time period.

[0074] Among them, all feature vectors of the target physical place within the current preset time period are spliced ​​and integrated according to specific logic, and feature information of different modalities and dimensions are integrated into a complete unified feature representation, that is, the scene status feature vector, providing a comprehensive data foundation for the subsequent construction of the real-world model.

[0075] Among them, for the target physical place from the current preset time period and the N preset time periods before the current preset time period, the scene status feature vectors within the N+1 preset time periods are further spliced ​​and compressed to generate a time series feature vector, which captures the multi-dimensional dynamic changes of the content mode, spatial structure mode and entity mode of the target physical place over a period of time, and provides a feature basis for constructing a time series model that reflects the laws of change in the real world.

[0076] The scene status feature vector before the first preset time period is the preset feature vector to ensure that starting from the first preset time period, the feature vector splicing logic of each preset time period is consistent, that is, the features of N+1 periods are spliced ​​together to avoid destroying the uniformity of timing processing due to special boundary conditions.

[0077] In a specific embodiment, the preset feature vector can be an all-0 vector, indicating "no historical information" or "initial state", which will not introduce false historical feature data and avoid the timing feature vector learning incorrect timing dependencies due to default values ​​such as random values ​​or non-zero constants.

[0078] In another specific embodiment, a spatiotemporal database can be configured to store the collection information corresponding to each historical time period or the feature information including content feature vectors, spatial structure feature vectors and entity feature vectors. Before obtaining the time series feature vectors corresponding to the first preset time period to the Nth preset time period, the collection information of the corresponding historical time period stored can be extracted from the spatiotemporal database to obtain the scene status feature vectors corresponding to each historical time period, or the feature information of the corresponding historical time period stored can be extracted from the spatiotemporal database to further obtain the scene status feature vectors corresponding to each historical time period.

[0079] As described above, loading real historical features into the spatiotemporal database can ensure that valid collected data or feature data are used starting from the first preset time period, thereby enhancing the time series representation capability of the spatiotemporal feature vector.

[0080] In a specific embodiment, S31 includes the following steps:

[0081] S311 , for any target object, concatenate all entity feature vectors corresponding to the current target object to obtain an intermediate feature vector corresponding to the current target object.

[0082] S312: splice the intermediate feature vectors corresponding to all target objects corresponding to the target physical place within the current preset time period to obtain a first target vector corresponding to the target physical place within the current preset time period.

[0083] S313: Concatenate all content feature vectors corresponding to the target physical place within the current preset time period to obtain a second target vector corresponding to the target physical place within the current preset time period.

[0084] S314: All spatial structure feature vectors corresponding to the target physical place within the current preset time period are spliced ​​to obtain a third target vector corresponding to the target physical place within the current preset time period.

[0085] S315, splicing the first target vector, the second target vector and the third target vector corresponding to the target physical place in the current preset time period according to the preset splicing order, and obtaining the scene status feature vector corresponding to the target physical place in the current preset time period.

[0086] Here, all entity feature vectors corresponding to the current target object are spliced ​​to integrate multi-view or multi-time features of the same target object to form a more comprehensive individual representation. The splicing method in this embodiment is serial splicing.

[0087] Collect the intermediate feature vectors of all target objects within the current time period and concatenate them according to preset rules to obtain the first target vector, which integrates the overall status characteristics of the people in the target physical location. For example, sort the target objects by their ID and then concatenate them in series.

[0088] The second target vector integrates all image information and text semantic information corresponding to the target physical place. The third target vector integrates the spatial structure information between all target objects in the target physical place. The scene status feature vector further integrates the overall character status characteristics of the target physical place, all image information and text semantic information, and the spatial structure information between all target objects, forming a unified multimodal feature representation, providing a complete and rich feature foundation for subsequent model training.

[0089] As mentioned above, the spatial structure information between all target objects in the target physical place is integrated. The scene status feature vector further integrates the overall character status characteristics of the target physical place, all image information and text semantic information, and the spatial structure information between all target objects, unifying the visual, semantic and spatial information, improving the richness and comprehensiveness of the representation of the target physical place, and providing a complete and rich feature foundation for subsequent model training.

[0090] In a specific embodiment, S32 includes the following steps:

[0091] S321, in order from first to last, splice the scene status feature vectors corresponding to the target physical place in the current preset time period and the N preset time periods before the current preset time period, and obtain the reference feature vector corresponding to the target physical place in the current preset time period.

[0092] S322: compress the reference feature vector corresponding to the target physical place within the current preset time period to obtain the time series feature vector corresponding to the target physical place within the current preset time period.

[0093] The scene status feature vectors for N+1 preset time periods are arranged in chronological order and concatenated according to their dimensions to form a reference feature vector, fully preserving all feature information within the time window. If the dimension of a single scene status feature vector is K, the dimension of the reference feature vector is (N+1)×K.

[0094] Then, a dimensionality reduction algorithm is used to compress the reference feature vector to a fixed dimension to reduce the feature dimension and computational complexity, while extracting key patterns in the time series and removing redundant information. The dimensionality reduction algorithm in this embodiment can be feature mapping through a fully connected layer.

[0095] The specific value of the time window length N can be set by the implementer based on actual conditions. For example, feature concatenation can be performed based on the mean value of the time window length in historical experience, or it can be set based on the actual requirements of downstream application tasks.

[0096] As described above, the N+1 scene status feature vectors corresponding to the current preset time period and the N preset time periods before the current preset time period are further spliced ​​and compressed into a time series feature vector, capturing the multi-dimensional dynamic changes of the target physical place in terms of content mode, spatial structure mode and entity mode over a period of time.

[0097] S4, input the content feature vector, spatial structure feature vector and entity feature vector corresponding to the target physical place in the current preset time period into the initial world model, and obtain the scene parsing feature data corresponding to the target physical place in the current preset time period, wherein the scene parsing feature data includes scene description text, question answer text and relationship feature vector list.

[0098] Among them, the initial world model is a pre-trained model that has basic multimodal understanding capabilities before model distillation. It is usually trained based on general scene data (such as public images, text datasets, and 3D point cloud libraries) and has cross-domain generalization capabilities, but is not optimized for specific target physical locations.

[0099] The initial world model receives the content feature vector, spatial structure feature vector, and entity feature vector corresponding to the target physical place, and performs basic semantic understanding and multimodal feature analysis to generate scene parsing feature data, which serves as a supervisory signal for the cloud-based world model distillation. It provides specific feature information corresponding to the target physical place for subsequent model distillation, allowing the real-world model to dynamically adapt to the multi-dimensional feature vectors and dynamic changes of the target physical place.

[0100] Specifically, the scene description text is a comprehensive summary of the scene of the target physical place, and the question answer text is the answer to the question contained in the collected text in the collected data. The scene description text and the question answer text are used to provide semantic supervision signals during model distillation; the relationship feature vector list includes several relationship feature vectors, each of which is used to represent the corresponding interactions, locations, and other relationships between people, objects, and people and objects, and is used to train the reasoning ability of real-world models.

[0101] As mentioned above, the initial world model is used to parse and generate multimodal features, provide supervision signals for model distillation, enhance the world model's generalization ability for spatiotemporal context, and guide the world model to learn reasoning logic that is closer to the target physical place, thereby enhancing the real-world model's semantic understanding, spatiotemporal reasoning, and multi-task adaptation capabilities for the target physical place.

[0102] S5, all content feature vectors, spatial structure feature vectors, entity feature vectors, time series feature vectors and scene analysis feature data corresponding to the target physical place in all preset time periods are used as the spatiotemporal context corresponding to the target physical place, wherein the spatiotemporal context is used to perform model distillation on the cloud world model in the cloud to obtain the real-world model corresponding to the target physical place.

[0103] Among them, the cloud world model is a pre-trained general model with broad generalization capabilities but lacks place-specific knowledge.

[0104] The cloud world model is used as the teacher model and the real-world model is used as the student model. By constructing a knowledge distillation loss including content feature vector loss, spatial structure feature vector loss, entity feature vector loss and temporal feature vector loss, it is used to constrain the feature extraction results of the student model in content modality, spatial structure modality, person modality and temporal modality to be close to the teacher model. Combined with task loss for training, the parameters of the student model are updated by minimizing the total loss obtained by task loss + knowledge distillation loss to obtain the real-world model corresponding to the target physical place.

[0105] Among them, the real-world model is used to perform the target application task according to the multimodal features corresponding to the target physical place, and the task loss is calculated according to the model output and the label corresponding to the target application task. The target application task can be set by the implementer according to the actual situation. For example, the target application task can be a task for predicting the flow of people in a smart retail scenario. Correspondingly, the output of the student model can be the predicted value of the flow of people in the future cycle time, and the corresponding label can be the actual value of the flow of people in the future cycle time obtained by counting the deployed cameras. The task loss is calculated based on the predicted value of the flow of people and the corresponding actual value of the flow of people, which serves as the loss basis for updating the parameters of the student model. Specifically, the task loss can be the mean square error loss.

[0106] As described above, by distilling the cloud world model in the cloud based on the multi-dimensional feature vectors and dynamic changes of the content modality, spatial structure modality and entity modality of the target physical place over a period of time, the real-world model corresponding to the target physical place is obtained, so that the real-world model can dynamically adapt to the multi-dimensional feature vectors and dynamic changes of the target physical place, thereby improving the representation degree and adaptability of the trained real-world model to the target physical place.

[0107] As mentioned above, by collecting data in two modes, image and text, the state information of the target physical place is covered from the two dimensions of visual information and semantic information, which makes up for the limitations of single-modality data. In addition, by collecting data within multiple preset time periods, it is possible to record the dynamic changes of the target physical place over time. The preset content encoder converts the original collected images and collected texts into content feature vectors, realizing the transformation of data from original form to feature and abstraction, removing redundant information in the collected data and retaining key features. By encoding the spatial structure feature vector, the spatial layout of the scene in the collected image, the relative positions of people and objects and other structural information are extracted to characterize the static layout features of the target physical place. By encoding the entity feature vector, the appearance, It encodes posture, action, etc., further integrates the overall character state characteristics of the target physical place within multiple preset time periods, all image information and text semantic information, and the spatial structure information between all target objects to obtain a temporal feature vector, captures the multi-dimensional dynamic changes of the target physical place in terms of content modality, spatial structure modality and entity modality over a period of time, and parses the extracted vectors to obtain scene parsing feature data, and integrates all feature information to obtain the spatiotemporal context, which improves the richness and comprehensiveness of the representation of the target physical place, so that the real-world model can dynamically adapt to the multi-dimensional feature vectors and dynamic changes of the target physical place, and improves the representation degree and adaptability of the trained real-world model to the target physical place.

[0108] Example 2

[0109] This second embodiment provides a method for constructing a real-world model based on multimodal features, such as Figure 2 As shown, the real-world model extraction method based on multimodal features includes the following steps:

[0110] S10, obtain the local regularity knowledge base corresponding to the target physical place before the target time period and the first modal features and second modal features corresponding to the target physical place within the target time period, wherein the first modal features include content feature vectors, spatial structure feature vectors and time series feature vectors, and the second modal features include entity feature vectors and relationship feature vectors.

[0111] Among them, the target physical place is an actual three-dimensional spatial area with a specific spatial structure and functional use. It is the object of research and modeling, such as shopping malls, campuses, factories and other specific spatial areas.

[0112] The target time period is a manually set time interval, and in this embodiment, it can be the period between two distillation trainings of the world model on the cloud.

[0113] The local law knowledge base is a database that stores the historical data laws of the target physical place. It can be trained or extracted based on the second modal feature data of the reference physical place over a long period of time in the past. It serves as the basis for distinguishing the known feature data from the unknown feature data in the second modal features corresponding to the target physical place in the target time period, thereby guiding the cloud-based world model to learn the local characteristics of the target physical place during the distillation training process, thereby enhancing the real-world model's ability to fine-tune the representation of the target physical place.

[0114] In one embodiment, S10 includes the following steps:

[0115] S101, obtain the characteristic vector of each second mode for each reference physical place within M preset time periods before the target time period, wherein the second mode includes a time series mode and a relationship mode, the characteristic vector for the time series mode is a time series characteristic vector, and the characteristic vector for the relationship mode is a relationship characteristic vector, and M is a positive integer.

[0116] S102: For any second modality, cluster all feature vectors corresponding to the current second modality for all reference physical places to obtain a plurality of feature cluster sets corresponding to the current second modality.

[0117] S103: Store all feature cluster sets corresponding to the current second modality into a local rule knowledge base corresponding to the target physical place.

[0118] The reference physical place is another physical environment that is similar or related to the target physical place in terms of physical characteristics, functional attributes, or scene types. By analyzing the feature data of multiple reference physical places in entity mode and relational mode, common patterns (such as crowd flow patterns, object placement rules, and interaction patterns between people and objects) are extracted. This provides cross-scene prior knowledge for the target physical place, and extracts features beyond the common patterns of the target physical place's offset reference. These features are used to specifically characterize the target physical place's personalized characteristics, thereby guiding the cloud-based world model to learn the local characteristics of the target physical place during the distillation training process, thereby enhancing the real-world model's ability to fine-tune the target physical place's representation.

[0119] Furthermore, spatial structural features (such as room layout and object location) usually have strong stability and domain regularity, and their regularity has been pre-modeled by referring to the common laws of physical places. If there is a temporary adjustment to the spatial structure (such as rearranging the tables and chairs in the conference room), the change will be indirectly reflected through the temporal features, and there is no need to judge the regularity separately; while entity features (such as human appearance and robot model) are highly personalized, and the definition of regularity is vague (such as different people wearing different colors of clothes is normal). Forcibly judging regularity may introduce misjudgment; temporal features are the changing trajectories of content modality, spatial structure modality, and entity modality over time. Their regularity needs to be based on the regular judgment results of the previous modality. Therefore, the content feature vector, spatial structure feature vector, and temporal feature vector corresponding to the first modal feature are all uploaded to the cloud as the basis for training the cloud world model.

[0120] The relationship feature vector describes the interaction logic between people and objects (such as conversations between people, contact between people and objects, and the position association between objects), and directly reflects information such as the dynamic laws and social rules of the physical place. The content feature vector, including the image feature vector and the text feature vector, can represent the scene semantic characteristics of the image and text corresponding to the physical place, thereby reflecting the type and function of the physical place. Dynamic interactions are prone to deviate from conventional common laws due to time, events or human factors (such as sudden gatherings, abnormal placement of items, etc.). The unconventionality of content features can reflect the particularity of the type or function of the physical place itself (such as the appearance of fire drill props in the office). Therefore, it is necessary to identify abnormal patterns under entity modality and relationship modality through common laws, and characterize the personalized characteristics of the target physical place in terms of static scene semantics and dynamic interaction situations, thereby enhancing the world model's deep understanding of the target physical place.

[0121] The M preset time periods before the target time period currently to be analyzed are used to extract historical feature data of the reference physical place.

[0122] All feature vectors corresponding to each second modality are clustered using a clustering algorithm such as K-means or DBSCAN to obtain several feature cluster sets corresponding to the current second modality. Each feature cluster set clusters several feature vectors with a high degree of similarity. Those skilled in the art will appreciate that any clustering algorithm in the prior art falls within the scope of protection of the present invention and will not be further described herein.

[0123] As described above, through the process of historical data collection, modal clustering and knowledge storage, the complex second modal characteristics of the target physical place in the historical time are converted into structured regular knowledge, and reusable prior knowledge is extracted from the historical data, which provides a basis for extracting personalized characteristics beyond the common regularities of the offset reference of the target physical place.

[0124] In one specific embodiment, S101 includes the following steps:

[0125] S1011, obtain a number of historical collection data corresponding to each reference physical place within M preset time periods, wherein the historical collection data is a historical collection image or a historical collection text, and each historical collection data represents the status of a number of target objects in the corresponding reference physical place within the current time period.

[0126] S1012, for any preset time period corresponding to any reference physical place, encode each historical collected data corresponding to the current reference physical place in the current preset time period according to a preset content encoder, and obtain a content feature vector corresponding to each historical collected data.

[0127] S1013, encode each historically collected image corresponding to the current reference physical place in the current preset time period according to the preset spatial structure encoder and the preset person entity encoder, and obtain the spatial structure feature vector corresponding to each historically collected image, and the entity feature vector corresponding to each target object in each historically collected image.

[0128] S1014, concatenate all content feature vectors, spatial structure feature vectors and entity feature vectors corresponding to the current reference physical place within the current preset time period to obtain the scene status feature vector corresponding to the current reference physical place within the current preset time period.

[0129] S1015, splicing and compressing the scene status feature vectors corresponding to the current reference physical place in the current preset time period and N preset time periods before the current preset time period, and obtaining the time series feature vector corresponding to the current reference physical place in the current preset time period, where N+1 is the preset number of vector splicing, and the scene status feature vector before the first preset time period is the preset feature vector.

[0130] S1016, input the content feature vector, spatial structure feature vector and entity feature vector corresponding to the current reference physical place in the current preset time period into the first initial world model, and obtain several relationship feature vectors corresponding to the current reference physical place in the current preset time period.

[0131] Among them, the first initial world model is a pre-trained model that has basic multimodal understanding capabilities before model distillation. It is usually trained based on general scene data (such as public images, text datasets, and 3D point cloud libraries) and has cross-domain generalization capabilities, but is not optimized for specific reference physical places.

[0132] In one embodiment, S10 includes the following steps:

[0133] S110, obtaining a number of target acquisition data corresponding to the target physical place within the target time period, wherein the target acquisition data is a target acquisition image or a target acquisition text, and each target acquisition data corresponds to the status of a number of target objects in the target physical place within the target time period.

[0134] S120 , encoding each target collected data corresponding to the target time period according to a preset content encoder, and obtaining a content feature vector corresponding to each target collected data.

[0135] S130, encoding each target acquisition image according to a preset spatial structure encoder and a preset person entity encoder, obtaining a spatial structure feature vector corresponding to each target acquisition image, and an entity feature vector corresponding to each target object in each target acquisition image.

[0136] S140, concatenating all content feature vectors, spatial structure feature vectors, and entity feature vectors corresponding to the target physical place within the target time period, to obtain a scene status feature vector corresponding to the target physical place within the target time period.

[0137] S150, splicing and compressing the scene status feature vector corresponding to the target physical place within the target time period and the scene status feature vector corresponding to the N-M+1 to M-th preset time periods before the target time period, and obtaining the time series feature vector corresponding to the target physical place within the target time period.

[0138] S160, input the content feature vector, spatial structure feature vector and entity feature vector corresponding to the target physical place in the target time period into the second initial world model, and obtain several relationship feature vectors corresponding to the target physical place in the target time period.

[0139] Among them, the second initial world model is obtained by training and optimization based on the first initial model and the first modal features and second modal features corresponding to the reference physical place. It has the ability to generalize across domains and across various reference physical places, but is not optimized for a specific target physical place.

[0140] The content feature vector, spatial structure feature vector, entity feature vector, scene status feature vector and timing feature vector corresponding to the target physical place within the target time period, as well as the method for obtaining the content feature vector, spatial structure feature vector, entity feature vector, scene status feature vector and timing feature vector corresponding to the target physical place within each preset time period, can refer to the method for obtaining the content feature vector, spatial structure feature vector, entity feature vector, scene status feature vector and timing feature vector in Example 1.

[0141] As described above, by comprehensively acquiring the target collected data of the target physical place within the target time period, and using the preset content encoder, spatial structure encoder and character entity encoder to perform targeted encoding on the target collected data, it is possible to fully mine the effective information in the target collected data, obtain comprehensive and representative content feature vectors, spatial structure feature vectors, entity feature vectors, time sequence feature vectors and relationship feature vectors, and improve the ability to understand the target physical place and target object in content mode, spatial structure mode, entity mode, time sequence mode and relationship mode.

[0142] S20, classifying the second modal features within the target time period according to the local rule knowledge base, and obtaining a target classification result corresponding to each feature vector in the second modal features, wherein the target classification result includes known feature data and unknown feature data.

[0143] Among them, the second modal features within the target time period and the historical laws represented by the local law knowledge base are analyzed to determine whether each eigenvector in the second modal features conforms to the known laws, so as to filter out unknown feature data, and then label each eigenvector in the second modal features. The corresponding target classification results are added to the training set of the cloud world model to optimize the model parameters, thereby improving the representation degree and adaptability of the real-world model to the target physical place.

[0144] In one embodiment, S20 includes the following steps:

[0145] S201, for any second modality, taking any feature vector corresponding to the current second modality of the target physical place within a target time period as a target feature vector.

[0146] S202 : Calculate a first distance between the target feature vector and each feature cluster set corresponding to the current second modality based on the target feature vector and each feature cluster set corresponding to the current second modality.

[0147] S203: Obtain a target classification result corresponding to the target feature vector based on a first distance between the target feature vector and each feature cluster set corresponding to the current second modality.

[0148] S204, traverse all feature vectors corresponding to all second modes of the target physical place within the target time period, and obtain the target classification result corresponding to each feature vector in the second modal features corresponding to the target physical place within the target time period.

[0149] In one specific embodiment, S203 includes the following steps:

[0150] S2031: Obtain a preset first distance threshold.

[0151] S2032: If the first distance between the target feature vector and each feature cluster set corresponding to the current second modality is less than or equal to a preset first distance threshold, determine that the target classification result corresponding to the target feature vector is known feature data.

[0152] S2033: If the first distances between the target feature vector and all feature cluster sets corresponding to the current second modality are greater than a preset first distance threshold, it is determined that the target classification result corresponding to the target feature vector is unknown feature data.

[0153] Among them, the first distance is obtained by the Euclidean distance and other distance calculation methods, which is used to measure the similarity between the target feature vector and each feature cluster set. The smaller the first distance, the higher the similarity between the target feature vector and each feature cluster set.

[0154] For any feature cluster set corresponding to the current second modality, in this embodiment, the average value of the distance between the target feature vector and all feature vectors in the current feature cluster set can be used as the first distance, or the distance between the target feature vector and the feature vector corresponding to the cluster center of the current feature cluster set can be used as the first distance.

[0155] Further, a preset first distance threshold corresponding to the current feature cluster set is obtained. If the first distance between the target feature vector and the current feature cluster set is less than or equal to the preset first distance threshold, it indicates that the similarity between the target feature vector and the current feature cluster set is high, and the target feature vector can be classified into the current feature cluster set, and the target classification result corresponding to the target feature vector is determined to be known feature data. The preset first distance threshold can be set according to time conditions. For example, the preset first distance threshold can be the maximum value or average value of the distance between the feature vector corresponding to the cluster center in the current feature cluster set and other feature vectors in the set.

[0156] If the first distance between the target feature vector and all feature cluster sets is greater than the preset first distance threshold, it means that the similarity between the target feature vector and all feature cluster sets corresponding to the current second modality is low, and the target feature vector cannot be classified into the feature cluster set corresponding to the current second modality. Then, the target classification result corresponding to the target feature vector is determined to be unknown feature data, indicating that the local rule knowledge base cannot understand the target feature vector based on the currently stored local rule information. Correspondingly, the target feature vector can be regarded as the personalized feature of the target physical place.

[0157] As described above, the target feature vector is classified based on the feature clustering set to obtain the target classification result, which is used as the personalized feature of the target physical place. After category labeling, it is input into the cloud for model training, thereby improving the representation degree and adaptability of the trained real-world model to the target physical place.

[0158] S30, based on the local regularity knowledge base, the first modal features, the second modal features, and the target classification results corresponding to each feature vector in the second modal features, the cloud world model in the cloud is subjected to model distillation to obtain a real-world model corresponding to the target physical place, wherein the real-world model is used to execute the target application task according to the multimodal features corresponding to the target physical place.

[0159] The cloud world model is a pre-trained general model with broad generalization capabilities but lacks location-specific knowledge. The feature vectors in the feature clustering set in the local regularity knowledge base, the first modal features, the second modal features, and the target classification results corresponding to each feature vector in the second modal features are aligned according to the time period, and a mixed training set is constructed according to a preset ratio.

[0160] The cloud world model is used as the teacher model, and the real-world model is used as the student model. By constructing a knowledge distillation loss including feature matching loss and classification consistency loss, combined with task loss for training, the parameters of the student model are updated by minimizing the total loss obtained by task loss + knowledge distillation loss to obtain the real-world model corresponding to the target physical place.

[0161] Among them, the real-world model is used to perform the target application task according to the multimodal features corresponding to the target physical place, and the task loss is calculated according to the model output and the label corresponding to the target application task. The target application task can be set by the implementer according to the actual situation. For example, the target application task can be a task for predicting the flow of people in a smart retail scenario. Correspondingly, the output of the student model can be the predicted value of the flow of people in the future cycle time, and the corresponding label can be the actual value of the flow of people in the future cycle time obtained by counting the deployed cameras. The task loss is calculated based on the predicted value of the flow of people and the corresponding actual value of the flow of people, which serves as the loss basis for updating the parameters of the student model. Specifically, the task loss can be the mean square error loss.

[0162] The feature matching loss is used to constrain the student model's extraction of known features to be close to the teacher model. Specifically, the feature matching loss can be the mean square error loss corresponding to the features under each modality. The classification consistency loss is used to constrain the student model's clustering results for unknown features to be aligned with the teacher model's soft labels. Specifically, the temporal consistency loss can be the KL divergence loss.

[0163] In a specific embodiment, the number of parameters corresponding to the real-world model is smaller than the number of parameters corresponding to the cloud-world model.

[0164] Among them, through distillation, the student model does not need to replicate all the parameters and computational complexity of the teacher model, but only needs to capture its core features and decision-making logic, thereby significantly reducing the number of parameters, reducing the computational complexity and memory usage of the model, and ensuring that the model can be inferred in real time at the edge, meeting the low-latency requirements in various scenarios, while reducing the overall computing cost and facilitating large-scale application.

[0165] As mentioned above, by deploying regular processing of newly emerging unknown feature data and triggering incremental distillation training, the real-world model can be adjusted according to the changes in the target physical place, so that the real-world model can dynamically adapt to the personalized laws and dynamic changes of the target physical place, thereby improving the execution accuracy of the task when executing the target application task.

[0166] In a specific embodiment, S10 further includes the following steps:

[0167] S104: Analyze several feature vectors in each feature cluster set corresponding to the current second modality to obtain feature change trend data corresponding to each feature cluster set of the current second modality for all reference physical places.

[0168] S105: Store all feature cluster sets corresponding to the current second modality and feature change trend data corresponding to each feature cluster set into a local rule knowledge base corresponding to the target physical place.

[0169] Among them, for each feature cluster set, the changes of several feature vectors in each feature cluster set over time are analyzed, and the corresponding feature change trend data are obtained to characterize the dynamic changes of each second mode of the target physical place, which serves as the basis for predicting the feature vector situation under the corresponding second mode in the future time period, and then the unconventional features are identified and classified through the predicted feature vectors.

[0170] The feature cluster set can be used as the static typical state of the reference physical place, and the feature change trend data can be used as the dynamic evolution law of the reference physical place. The feature cluster set and feature change trend data corresponding to each second mode are stored in the local law knowledge base corresponding to the target physical place, which is convenient for combining the second mode characteristics within the target time period and judging whether the second mode characteristics of the target physical place within the target time period conform to the common historical laws based on the local law knowledge base, so as to further improve the accuracy of extracting personalized features beyond the common laws of the offset reference of the target physical place.

[0171] In one specific embodiment, S104 includes the following steps:

[0172] S1041: For any feature cluster set corresponding to the current second modality, obtain a preset time period corresponding to each feature vector in the current feature cluster set.

[0173] S1042, constructing a time series prediction model corresponding to the current feature cluster set based on all feature vectors in the current feature cluster set and the preset time period corresponding to each feature vector, wherein the time series prediction model is used to characterize the feature change trend of the current feature cluster set for the current second modality of all reference physical places.

[0174] S1043: traverse all feature cluster sets corresponding to the current second modality, and obtain a time series prediction model corresponding to each feature cluster set corresponding to the current second modality.

[0175] S1044: traverse all second modalities and obtain the time series prediction model corresponding to each feature cluster set corresponding to each second modality.

[0176] S1045, storing the time series prediction model corresponding to each feature cluster set corresponding to each second modality into the local rule knowledge base corresponding to the target physical place.

[0177] The time series prediction model can be based on a long-short-term memory network, capturing long-term dependencies in time series through gating mechanisms such as input gates, forget gates, and output gates. Furthermore, by fine-tuning all feature vectors in the current feature cluster set and the preset time period corresponding to each feature vector, a time series prediction model corresponding to the current feature cluster set is constructed. This model is used to characterize the temporal evolution of the feature cluster sets corresponding to all reference physical locations, and thus predict the feature vectors corresponding to the target physical location for the current second modality in various future time periods.

[0178] As described above, through the process of feature clustering, time series modeling, and knowledge storage, implicit spatiotemporal trends are converted into executable model parameters, which improves the interpretability of scene understanding in the reference physical place. In this way, independent modeling is performed for different second modes and feature clustering sets, accurately adapting to the personalized laws of the target physical place, thereby enhancing scene adaptability.

[0179] In a specific embodiment, S20 further includes the following steps:

[0180] S205 , obtaining a first classification result corresponding to the target feature vector based on a first distance between the target feature vector and each feature cluster set corresponding to the current second modality, wherein the first classification result is an outlier feature and a non-outlier feature.

[0181] S206: If the first classification result corresponding to the target feature vector is an outlier feature, the target classification result corresponding to the target feature vector is determined as unknown feature data.

[0182] S207, if the first classification result corresponding to the target feature vector is a non-outlier feature, then the target classification result corresponding to the target feature vector is obtained according to the target time period corresponding to the target feature vector and the time series prediction model corresponding to each feature cluster set corresponding to the current second modality.

[0183] S208, traverse all second modes to obtain the target classification result corresponding to each feature vector in the multimodal features corresponding to the target physical place within the target time period.

[0184] Among them, if the first distance between the target feature vector and the current feature cluster set is less than or equal to the preset first distance threshold, it means that the similarity between the target feature vector and the current feature cluster set is high, and the target feature vector can be classified into the current feature cluster set, then the first classification result corresponding to the target feature vector is determined to be a non-outlier feature.

[0185] After the first round of classification of the target feature vector based on the feature cluster set, the target feature vector is further judged based on the time series prediction model corresponding to each feature cluster set corresponding to the current second modality to determine whether it conforms to the corresponding feature change trend, thereby performing a secondary classification on the target feature vector to determine whether the target feature vector belongs to unknown feature data, thereby improving the accuracy of the target classification results through two rounds of classification operations.

[0186] If the first distance between the target feature vector and all feature cluster sets is greater than the preset first distance threshold, it means that the similarity between the target feature vector and all feature cluster sets corresponding to the current second modality is low, and the target feature vector cannot be classified into the feature cluster set corresponding to the current second modality. The first classification result corresponding to the target feature vector is determined to be an outlier feature, and the target classification result corresponding to the target feature vector is further determined to be unknown feature data, indicating that the local rule knowledge base cannot understand the target feature vector based on the currently stored local rule information, thereby clarifying the personalized characteristics corresponding to the target physical place through category labeling.

[0187] In a specific embodiment, S207 includes the following steps:

[0188] S2071, for a time series prediction model corresponding to any feature cluster set corresponding to the current second modality, obtain a prediction feature vector corresponding to the current second modality of the target physical place within a target time period.

[0189] S2072: Calculate and obtain a second distance between the target feature vector and the corresponding predicted feature vector.

[0190] S2073, traverse the time series prediction models corresponding to all feature cluster sets corresponding to the current second modality, and obtain all second distances corresponding to the target feature vector.

[0191] S2074 , obtaining a target classification result corresponding to the target feature vector according to all second distances corresponding to the target feature vector.

[0192] Among them, the corresponding predicted feature vector is obtained through the time series prediction model, and the second distance is obtained by the Euclidean distance and other distance calculation methods to characterize the similarity between the target feature vector and the predicted feature vector. Correspondingly, the smaller the second distance, the higher the similarity between the target feature vector and the predicted feature vector, that is, the target feature vector is more consistent with the feature change trend.

[0193] The preset second distance threshold corresponding to the current second mode is obtained. If any second distance corresponding to the target feature vector is less than or equal to the preset second distance threshold, it means that the similarity between the target feature vector and the corresponding feature change trend is high, and the target classification result corresponding to the target feature vector is determined to be known feature data.

[0194] If all the second distances corresponding to the target feature vector are greater than the preset second distance threshold, it means that the similarity between the target feature vector and the corresponding feature change trend is low, and the target classification result corresponding to the target feature vector is determined to be unknown feature data, indicating that the local rule knowledge base cannot understand the target feature vector based on the currently stored local rule information, thereby clarifying the personalized characteristics corresponding to the target physical place through category labeling, and transmitting it to the cloud to participate in model training, so as to improve the representation degree and adaptability of the trained real-world model to the target physical place.

[0195] The preset second distance threshold may be set according to time conditions. For example, the preset second distance threshold may be the standard deviation of all second distances obtained in historical operations.

[0196] In the above, the target feature vector is first classified in the first round based on the feature cluster set to quickly exclude obviously abnormal outlier features and reduce the amount of subsequent calculations. The time series prediction model corresponding to each feature cluster set corresponding to the current second modality is used to determine whether the non-outlier features conform to the corresponding feature change trend, thereby obtaining the target classification result and improving the accuracy and reliability of the target classification result.

[0197] In the above, by obtaining the local law knowledge base corresponding to the target physical place before the target time period, the common laws corresponding to the second modal features of different reference physical places in the historical time period are provided, and the content feature vector, spatial structure feature vector, entity feature vector, time series feature vector and relationship feature vector of the target physical place in the target time period are obtained, which improves the understanding and representation capabilities of the target physical place and target object in the content mode, spatial structure mode, entity mode, time series mode and relationship mode, and further classifies the second modal features in the target time period based on the local law knowledge base, obtains the target classification results corresponding to each feature vector in the second modal features, and thus Each eigenvector in the two-modal features is labeled with a category to clarify the personalized features corresponding to the target physical place. Finally, the local law knowledge base, the first modal features, the second modal features, and the target classification results corresponding to each eigenvector in the second modal features are added to the training set of the cloud world model. The cloud world model is jointly distilled to adjust the real-world model according to the various modal features and personalized features of the target physical place, so that the real-world model can dynamically adapt to the personalized laws and dynamic changes of the target physical place, thereby improving the representation degree and adaptability of the trained real-world model to the target physical place, and thus improving the execution accuracy of the task when performing the target application task.

[0198] Example 3

[0199] This embodiment 3 provides a spatiotemporal context extraction system for real-world model training, such as Figure 3 As shown in Figure 1, the spatiotemporal context extraction system for real-world model training includes:

[0200] The collected data acquisition module 31 is used to obtain a number of collected data corresponding to the target physical place within M preset time periods, wherein each collected data can be a collected image or a collected text, and M is a positive integer.

[0201] The first feature encoding module 32 is used to encode the collected data corresponding to the current preset time period according to the preset content encoder, the preset spatial structure encoder and the preset character entity encoder for any preset time period, and obtain the content feature vector corresponding to each collected data, the spatial structure feature vector corresponding to each collected image, and the entity feature vector corresponding to each target object in each collected image.

[0202] The second feature encoding module 33 is used to obtain the time series feature vector corresponding to the target physical place in the current preset time period based on all content feature vectors, spatial structure feature vectors and entity feature vectors corresponding to the target physical place in the current preset time period and N preset time periods before the current preset time period, where N is an integer greater than 0.

[0203] The scene parsing module 34 is used to input the content feature vector, spatial structure feature vector and entity feature vector corresponding to the target physical place in the current preset time period into the initial world model, and obtain the scene parsing feature data corresponding to the target physical place in the current preset time period, wherein the scene parsing feature data includes scene description text, question answer text and relationship feature vector list.

[0204] The spatiotemporal context extraction module 35 is used to take all content feature vectors, spatial structure feature vectors, entity feature vectors, time series feature vectors and scene analysis feature data corresponding to the target physical place in all preset time periods as the spatiotemporal context corresponding to the target physical place, wherein the spatiotemporal context is used to perform model distillation on the cloud world model in the cloud to obtain the real world corresponding to the target physical place.

[0205] In a specific embodiment, the data acquisition module 31 includes:

[0206] The initial data acquisition submodule is used to obtain a number of initial data corresponding to the target physical place within each preset time period, wherein each initial data can be a captured image, a captured video or a captured text, and each initial data corresponds to the status of several target objects in the target physical place within the current preset time period.

[0207] The image sampling submodule is used to sample each captured video and use each sampled image frame and each captured image in the initial data as the corresponding captured image of the target physical place within the corresponding preset time period.

[0208] In a specific embodiment, the first feature encoding module 32 includes:

[0209] The first feature encoding submodule is configured to encode each collected data corresponding to the current preset time period according to a preset content encoder for any preset time period, and obtain a content feature vector corresponding to each collected data.

[0210] The second feature encoding submodule is used to encode each acquired image corresponding to the current preset time period according to the preset spatial structure encoder and the preset character entity encoder, and obtain the spatial structure feature vector corresponding to each acquired image, and the entity feature vector corresponding to each target object in each acquired image.

[0211] In a specific embodiment, the preset content encoder includes a preset image encoder and a preset text encoder, and the first feature encoding submodule includes:

[0212] The image encoding unit is used to encode any collected data according to a preset image encoder if the current collected data is a collected image, and obtain a content feature vector corresponding to the current collected data.

[0213] The text encoding unit is used to encode the current collected data according to a preset text encoder if the current collected data is collected text, and obtain a content feature vector corresponding to the current collected data.

[0214] In a specific embodiment, the second feature encoding submodule includes:

[0215] The image dimension conversion unit is used to perform dimension conversion on any collected image corresponding to the current preset time period to obtain a three-dimensional image corresponding to the current collected image.

[0216] The semantic annotation unit is used to perform semantic annotation on the three-dimensional image corresponding to the current collected image, and obtain the annotated image corresponding to the current collected image.

[0217] The spatial structure feature encoding unit is used to encode the annotated image corresponding to the current collected image according to a preset spatial structure encoder to obtain the spatial structure feature vector corresponding to the current collected image.

[0218] The entity extraction unit is used to perform entity extraction on the annotated image corresponding to the current collected image, and obtain the entity image corresponding to each target object in the current collected image.

[0219] The entity feature encoding unit is used to encode the entity image corresponding to each target object in the current captured image according to a preset person entity encoder, and obtain the entity feature vector corresponding to each target object in the current captured image.

[0220] In a specific embodiment, the second feature encoding module 33 includes:

[0221] The first feature splicing submodule is used to splice all content feature vectors, spatial structure feature vectors and entity feature vectors corresponding to the target physical place in the current preset time period for any preset time period, and obtain the scene status feature vector corresponding to the target physical place in the current preset time period.

[0222] The second feature splicing submodule is used to splice and compress the scene status feature vectors corresponding to the target physical place in the current preset time period and the N preset time periods before the current preset time period, and obtain the time series feature vector corresponding to the target physical place in the current preset time period.

[0223] In a specific embodiment, the first feature splicing submodule includes:

[0224] The entity feature vector splicing unit is used to splice all entity feature vectors corresponding to any target object to obtain an intermediate feature vector corresponding to the current target object.

[0225] The intermediate feature vector splicing unit is used to splice the intermediate feature vectors corresponding to all target objects corresponding to the target physical place within the current preset time period, and obtain the first target vector corresponding to the target physical place within the current preset time period.

[0226] The content feature vector splicing unit is used to splice all content feature vectors corresponding to the target physical place within the current preset time period to obtain a second target vector corresponding to the target physical place within the current preset time period.

[0227] The spatial structure feature vector splicing unit is used to splice all spatial structure feature vectors corresponding to the target physical place within the current preset time period to obtain a third target vector corresponding to the target physical place within the current preset time period.

[0228] The scene status feature vector acquisition unit is used to splice the first target vector, the second target vector and the third target vector corresponding to the target physical place within the current preset time period in a preset splicing order to obtain the scene status feature vector corresponding to the target physical place within the current preset time period.

[0229] In a specific embodiment, the second feature splicing submodule includes:

[0230] The reference feature vector acquisition unit is used to splice the scene status feature vectors corresponding to the target physical place in the current preset time period and the N preset time periods before the current preset time period in order from first to last time, and obtain the reference feature vector corresponding to the target physical place in the current preset time period.

[0231] The time series feature vector acquisition unit is used to compress the reference feature vector corresponding to the target physical place within the current preset time period, and obtain the time series feature vector corresponding to the target physical place within the current preset time period.

[0232] It should be noted that the information interaction, execution process and other contents between the above modules are based on the same concept as the embodiment of the method of the present invention. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0233] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention in any form. Although the present invention has been disclosed as above in terms of preferred embodiments, they are not intended to limit the present invention. Any technician familiar with this profession can make some changes or modifications to equivalent embodiments of equivalent changes using the technical contents disclosed above without departing from the scope of the technical solution of the present invention. However, any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention are still within the scope of the technical solution of the present invention.

Claims

1. A spatiotemporal context extraction method for real-world model training, characterized in that The extraction method comprises the following steps: S1, obtaining a plurality of collected data corresponding to the target physical location within M preset time periods, wherein each collected data is a collected image or collected text, and M is a positive integer; S2, for any preset time period, encoding the collected data corresponding to the current preset time period according to a preset content encoder, a preset spatial structure encoder, and a preset person entity encoder, to obtain a content feature vector corresponding to each collected data, a spatial structure feature vector corresponding to each collected image, and an entity feature vector corresponding to each target object in each collected image; S3, obtaining a time series feature vector corresponding to the target physical place in the current preset time period based on all content feature vectors, spatial structure feature vectors, and entity feature vectors corresponding to the target physical place in the current preset time period and N preset time periods before the current preset time period, where N is an integer greater than 0; S4, inputting the content feature vector, spatial structure feature vector, and entity feature vector corresponding to the target physical place in the current preset time period into the initial world model, and obtaining scene parsing feature data corresponding to the target physical place in the current preset time period, wherein the scene parsing feature data includes scene description text, question answer text, and a list of relationship feature vectors; S5, taking all content feature vectors, spatial structure feature vectors, entity feature vectors, time series feature vectors and scene analysis feature data corresponding to the target physical place in all preset time periods as the spatiotemporal context corresponding to the target physical place, wherein the spatiotemporal context is used to perform model distillation on the cloud world model in the cloud to obtain the real-world model corresponding to the target physical place.

2. The spatiotemporal context extraction method for real-world model training according to claim 1, characterized in that S1 includes the following steps: S11, obtaining a plurality of initial data corresponding to the target physical place in each preset time period, wherein each initial data is a captured image, a captured video, or a captured text, and each initial data corresponds to a state of a plurality of target objects in the target physical place in the current preset time period; S12, sampling each captured video, and taking each image frame obtained by sampling and each captured image in the initial data as the corresponding captured image of the target physical place within the corresponding preset time period.

3. The spatiotemporal context extraction method for real-world model training according to claim 1, characterized in that S2 includes the following steps: S21, for any preset time period, encoding each collected data corresponding to the current preset time period according to a preset content encoder, and obtaining a content feature vector corresponding to each collected data; S22, encode each acquired image corresponding to the current preset time period according to the preset spatial structure encoder and the preset character entity encoder, and obtain the spatial structure feature vector corresponding to each acquired image and the entity feature vector corresponding to each target object in each acquired image.

4. The spatiotemporal context extraction method for real-world model training according to claim 3, characterized in that The preset content encoder includes a preset image encoder and a preset text encoder. S21 includes the following steps: S211 , for any collected data, if the current collected data is a collected image, encode the current collected data according to a preset image encoder to obtain a content feature vector corresponding to the current collected data; S212: If the current collected data is collected text, encode the current collected data according to a preset text encoder to obtain a content feature vector corresponding to the current collected data.

5. The spatiotemporal context extraction method for real-world model training according to claim 3, characterized in that S22 includes the following steps: S221, for any collected image corresponding to the current preset time period, performing dimension conversion on the current collected image to obtain a three-dimensional image corresponding to the current collected image; S222, performing semantic annotation on the three-dimensional image corresponding to the current acquired image to obtain an annotated image corresponding to the current acquired image; S223, encoding the annotated image corresponding to the current acquired image according to the preset spatial structure encoder to obtain a spatial structure feature vector corresponding to the current acquired image; S224, performing entity extraction on the annotated image corresponding to the current captured image to obtain an entity image corresponding to each target object in the current captured image; S225 , encoding the entity image corresponding to each target object in the current captured image according to the preset human entity encoder, and obtaining the entity feature vector corresponding to each target object in the current captured image.

6. The spatiotemporal context extraction method for real-world model training according to claim 1, characterized in that S3 includes the following steps: S31, for any preset time period, concatenating all content feature vectors, spatial structure feature vectors, and entity feature vectors corresponding to the target physical place within the current preset time period to obtain a scene status feature vector corresponding to the target physical place within the current preset time period; S32, splicing and compressing the scene status feature vectors corresponding to the target physical place in the current preset time period and N preset time periods before the current preset time period, and obtaining the time series feature vector corresponding to the target physical place in the current preset time period.

7. The spatiotemporal context extraction method for real-world model training according to claim 6, characterized in that S31 includes the following steps: S311, for any target object, concatenate all entity feature vectors corresponding to the current target object to obtain an intermediate feature vector corresponding to the current target object; S312: Concatenate intermediate feature vectors corresponding to all target objects corresponding to the target physical place within the current preset time period to obtain a first target vector corresponding to the target physical place within the current preset time period; S313: Concatenate all content feature vectors corresponding to the target physical place within the current preset time period to obtain a second target vector corresponding to the target physical place within the current preset time period; S314: Concatenate all spatial structure feature vectors corresponding to the target physical place within the current preset time period to obtain a third target vector corresponding to the target physical place within the current preset time period; S315, splicing the first target vector, the second target vector and the third target vector corresponding to the target physical place in the current preset time period in a preset splicing order, and obtaining the scene status feature vector corresponding to the target physical place in the current preset time period.

8. The spatiotemporal context extraction method for real-world model training according to claim 6, characterized in that S32 includes the following steps: S321: Concatenate, in chronological order, scene status feature vectors corresponding to the target physical place in the current preset time period and N preset time periods before the current preset time period to obtain a reference feature vector corresponding to the target physical place in the current preset time period; S322: compress the reference feature vector corresponding to the target physical place within the current preset time period to obtain the time series feature vector corresponding to the target physical place within the current preset time period.

9. A spatiotemporal context extraction system for real-world model training, characterized in that The spatiotemporal context extraction system for real-world model training includes: A collection data acquisition module is used to acquire a plurality of collection data corresponding to the target physical location within M preset time periods, wherein each collection data is a collection image or a collection text, and M is a positive integer; A first feature encoding module is configured to encode, for any preset time period, the collected data corresponding to the current preset time period according to a preset content encoder, a preset spatial structure encoder, and a preset person entity encoder, to obtain a content feature vector corresponding to each collected data, a spatial structure feature vector corresponding to each collected image, and an entity feature vector corresponding to each target object in each collected image; a second feature encoding module, configured to obtain a time series feature vector corresponding to the target physical place within the current preset time period based on all content feature vectors, spatial structure feature vectors, and entity feature vectors corresponding to the target physical place within the current preset time period and N preset time periods before the current preset time period, where N is an integer greater than 0; A scene parsing module is configured to input the content feature vector, spatial structure feature vector, and entity feature vector corresponding to the target physical place in the current preset time period into the initial world model, and obtain scene parsing feature data corresponding to the target physical place in the current preset time period, wherein the scene parsing feature data includes scene description text, question answer text, and a list of relationship feature vectors; The spatiotemporal context extraction module is used to use all content feature vectors, spatial structure feature vectors, entity feature vectors, time series feature vectors and scene analysis feature data corresponding to the target physical place in all preset time periods as the spatiotemporal context corresponding to the target physical place, wherein the spatiotemporal context is used to perform model distillation on the cloud world model in the cloud to obtain the real-world model corresponding to the target physical place.

10. The spatiotemporal context extraction system for real-world model training according to claim 9, characterized in that The data acquisition module includes: An initial data acquisition submodule is used to acquire a plurality of initial data corresponding to the target physical place within each preset time period, wherein each initial data is a captured image, a captured video, or a captured text, and each initial data corresponds to the state of a plurality of target objects in the target physical place within the current preset time period; The image sampling submodule is used to sample each captured video and use each sampled image frame and each captured image in the initial data as the corresponding captured image of the target physical place within the corresponding preset time period.

Images