Cross-modal meta-semantic understanding method and system for complex large-scale scenes

Abstract

The present invention discloses a method and system for cross-modal meta-semantic understanding of complex large scenes, the method comprising: performing image meta-semantic interpretation on complex large scene images, obtaining the position distribution and scale distribution of multiple types of objects in the scene, and establishing cross-scale co-occurrence relationships between multiple types of objects; performing text meta-semantic interpretation on demand text based on the position distribution and object scale distribution of multiple types of objects in the complex scene, obtaining a phrase list whose information volume corresponds to the complex scene in descending order, and extracting text features; performing adaptive processing on complex scene images according to the co-occurrence relationship between the phrase list and the objects, obtaining a local image to be further understood, and extracting image features; embedding text features and image features into a cross-modal feature space for cross-modal meta-semantic fusion and mapping, and obtaining an image understanding result of the corresponding text. The present invention can realize the interpretation of complex scene meta-semantics, and thus realize intelligent understanding of complex scenes.

Description

Technical Field

[0001] The present invention relates to the field of semantic understanding technology, and in particular to a method and system for cross-modal meta-semantic understanding in complex large-scale scenarios. Background Art

[0002] Complex scene semantic understanding aims to model the semantics of images and videos and use the representation to analyze visual content. It is a core common technology that supports downstream tasks such as video summary generation, video retrieval, and video question answering. Taking video semantic retrieval as an example, visual semantic understanding requires interpreting the user's natural language description and retrieving and locating the relevant areas in the image and video, which is a typical cross-modal problem.

[0003] Existing scene semantic understanding technologies mainly perform modal feature fusion and matching based on image-level visual implicit representation (or local region visual implicit representation) and sentence-level language implicit representation to achieve visual semantic understanding. However, such technologies can only support visual understanding of simple scenes.

[0004] With the rapid development of imaging technology in recent years, the perception flux has exceeded one billion pixels, and video content presents the characteristics of large scenes, multiple objects, and complex relationships. Image-level implicit representation or local region implicit representation cannot describe the semantics of images and videos, and cannot establish the complex relationship between the global and local in complex scenes. Therefore, it is necessary to design a new method to transform the "whole" into "meta", construct a meta-semantic representation of complex large scenes, and establish a new mechanism for cross-modal semantic understanding. Summary of the invention

[0005] The present invention aims to solve one of the technical problems in the related art at least to a certain extent.

[0006] To this end, the present invention proposes a cross-modal meta-semantic understanding method for complex large scenes, aiming to overcome the problem that traditional cross-modal representation and processing technologies cannot support the understanding of complex scenes, to achieve the interpretation of the meta-semantics of complex scenes, and then to achieve intelligent understanding of complex scenes.

[0007] Another object of the present invention is to propose a complex large-scene cross-modal meta-semantic understanding system.

[0008] To achieve the above objectives, the present invention proposes a complex large-scale cross-modal meta-semantic understanding method, comprising:

[0009] Perform image meta-semantic interpretation on complex large scene images to obtain the location distribution and scale distribution of multiple types of objects in the scene, and establish cross-scale co-occurrence relationships between multiple types of objects;

[0010] Based on the position distribution and scale distribution of multiple types of objects in the scene, the demand text of the complex large scene image is interpreted by text meta-semantics to obtain a phrase list whose information content corresponds to the complex scene in descending order;

[0011] Adaptively processing a complex large scene image with a wide field of view and high resolution according to the co-occurrence relationship between the phrase list and the objects to obtain a local image to be understood;

[0012] The text features extracted based on the phrase list and the image features extracted based on the local image to be understood are embedded in the cross-modal feature space for cross-modal meta-semantic fusion and mapping to obtain the image understanding results of the corresponding text.

[0013] The complex large-scenario cross-modal meta-semantic understanding method of the embodiment of the present invention may also have the following additional technical features:

[0014] The complex large-scenario cross-modal meta-semantic understanding method of the embodiment of the present invention also has the following additional technical features:

[0015] In one embodiment of the present invention, image meta-semantics are interpreted for complex large scene images to obtain the position distribution and scale distribution of multiple types of objects in the scene, and cross-scale co-occurrence relationships between multiple types of objects are established, including:

[0016] Perform multi-target detection on complex large scenes to obtain detection results, and perform statistical analysis on the detection results to obtain the position distribution and scale distribution of multiple types of objects in the scene;

[0017] A complex relationship recognition model is trained based on the position distribution and scale distribution of multiple types of objects in the scene, and a cross-scale co-occurrence relationship between multiple objects is established, wherein the co-occurrence relationship includes the spatial correlation distance between the objects.

[0018] In one embodiment of the present invention, based on the position distribution and scale distribution of multiple types of objects in the scene, the text meta-semantic interpretation of the demand text of the complex large scene image with wide field of view and high resolution is performed to obtain a phrase list whose information content corresponds to the complex scene in descending order, including:

[0019] Segmenting the demand text of the complex large scene image to obtain a phrase set, and establishing a semantic structure between the phrase sets through syntactic analysis;

[0020] Based on the semantic structure and the position distribution and scale distribution of multiple types of objects in the scene, the phrases are arranged in descending order of information based on the complex large scene image with wide field of view and high resolution to obtain a list of phrases to be understood.

[0021] In one embodiment of the present invention, adaptively processing a complex large scene image according to the co-occurrence relationship between the phrase list and the objects to obtain a local image to be understood includes:

[0022] Perform scale-adaptive downsampling operations on complex large scene images according to the scale of the object to be understood;

[0023] According to the co-occurrence relationship between multiple types of objects, the local image to be understood after the downsampling operation is sampled, enlarged and expanded.

[0024] In one embodiment of the present invention, text features extracted based on a phrase list and image features extracted based on a local image to be understood are embedded into a cross-modal feature space for cross-modal meta-semantic fusion and mapping, and an image understanding result of the corresponding text is obtained, including:

[0025] The local image to be understood is divided into multiple small blocks, each small block is flattened, the time features of the small blocks are extracted using a fully connected layer, the positional relationship between the small blocks is established using position encoding, and the visual attention mechanism is used to extract the visual features between the small blocks to obtain the visual features of the local image;

[0026] Transform the text to be understood into a list of word vectors and use the attention mechanism to extract text features;

[0027] The visual fully connected layer and the text fully connected layer are used respectively to transform the visual features and text features into a unified cross-modal space of vision and text for fusion and mapping, so as to calculate in the cross-modal space to obtain the semantic understanding results.

[0028] To achieve the above objectives, the present invention proposes, on the other hand, a complex large-scene cross-modal meta-semantic understanding system, comprising:

[0029] Image meta-semantic interpretation module, used to interpret image meta-semantics of complex large scene images, obtain the position distribution and scale distribution of multiple types of objects in the scene, and establish cross-scale co-occurrence relationships between multiple types of objects;

[0030] A text meta-semantic interpretation module, used to perform text meta-semantic interpretation on the demand text of the complex large scene image based on the position distribution and scale distribution of multiple types of objects in the scene, and obtain a phrase list whose information content corresponds to the complex scene in descending order;

[0031] A wide-field high-resolution image adaptive processing module, used for adaptively processing a wide-field high-resolution complex large-scene image according to the co-occurrence relationship between the phrase list and the object to obtain local image information to be understood;

[0032] The cross-modal meta-semantic fusion and mapping module is used to embed the text features extracted based on the phrase list and the image features extracted based on the local image to be understood into the cross-modal feature space for cross-modal meta-semantic fusion and mapping, and obtain the image understanding result of the corresponding text. The complex large-scene cross-modal meta-semantic understanding method and system of the embodiment of the present invention establishes the complex relationship between the global and the local in the complex scene, and continuously deepens the understanding of the complex scene.

[0033] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0035] Figure 1 is a flowchart of a complex large-scene cross-modal meta-semantic understanding method according to an embodiment of the present invention;

[0036] Figure 2 This is an example diagram of a billion-pixel large-scene multi-object complex relationship image video data in a macroscopic scene according to an embodiment of the present invention;

[0037] Figure 3 is a flowchart of text meta-semantic interpretation according to an embodiment of the present invention;

[0038] Figure 4 is a flow chart of visual meta-semantic interpretation and cross-modal semantic fusion and mapping according to an embodiment of the present invention;

[0039] Figure 5 is a schematic diagram of a neural network forward process according to an embodiment of the present invention;

[0040] Figure 6 It is a structural diagram of a complex large-scenario cross-modal meta-semantic understanding system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0041] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0042] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0043] The following describes a complex large-scene cross-modal meta-semantic understanding method and system proposed according to an embodiment of the present invention with reference to the accompanying drawings.

[0044] Figure 1 It is a flowchart of a complex large-scene cross-modal meta-semantic understanding method according to an embodiment of the present invention.

[0045] like Figure 1 As shown, the method includes but is not limited to the following steps:

[0046] S1, perform image meta-semantic interpretation on complex large scene images to obtain the location distribution and scale distribution of multiple types of objects in the scene, and establish the cross-scale co-occurrence relationship between multiple types of objects;

[0047] S2, based on the location distribution and scale distribution of multiple types of objects in the scene, the text meta-semantic interpretation of the demand text of the complex large scene image is performed to obtain a list of phrases whose information content corresponds to the complex scene in descending order;

[0048] S3, adaptively processing the complex large scene image with wide field of view and high resolution according to the co-occurrence relationship between the phrase list and the objects to obtain the local image to be understood;

[0049] S4, embeds the text features extracted based on the phrase list and the image features extracted based on the local image to be understood into the cross-modal feature space for cross-modal meta-semantic fusion and mapping, and obtains the image understanding result of the corresponding text.

[0050] It is understandable that, inspired by the human cognitive mechanism of understanding wide-field-of-view and high-resolution images and videos, which is continuously refined from global to local, the present invention proposes a cross-modal meta-semantic understanding method for complex large scenes, which continuously deepens the understanding of complex scenes through continuous iteration.

[0051] The embodiment of the present invention performs multi-target detection on a complex large scene to obtain detection results, and performs statistical analysis on the detection results to obtain the position distribution and scale distribution of multiple categories of objects in the scene; a complex relationship recognition model is trained based on the position distribution and scale distribution of multiple categories of objects in the scene, and a cross-scale co-occurrence relationship between multiple objects is established, wherein the co-occurrence relationship includes the spatial correlation distance between objects.

[0052] Specifically, it is necessary to interpret image meta-semantics for complex large scene images, including multi-target detection to obtain the position distribution and scale distribution of multiple types of objects in the scene. Existing deep neural networks (such as YOLOX, etc.) can be used to perform statistical analysis on the detection results to obtain the position distribution and scale distribution of multiple types of objects in the scene. Self-supervised learning is used to establish cross-scale co-occurrence relationships between multiple types of objects.

[0053] In one embodiment of the present invention, it is necessary to segment the text expression to identify meaningful phrases. Existing deep neural networks (such as Stanford interpreter, large language model, etc.) can be used. Deep neural networks trained on other tasks can be used, or they can be retrained with relevant data according to actual application scenarios.

[0054] It is understandable that if Figure 2 For example, the text expression is "a woman wearing a green top and holding an umbrella standing under the middle lawn". The expression is just an object in the image. For example, "The person wearing a white hat, talking to someone next to the pool in the bottom left corner of the picture, with a blue shoulder bag". The text semantic interpretation is as follows: Figure 3 shown.

[0055] It is understandable that text segmentation, also known as word segmentation, is a basic step in natural language processing, which aims to segment continuous text into meaningful phrases or word units. This process is crucial for subsequent tasks such as text analysis, information retrieval, and machine translation.

[0056] Furthermore, the relationship between phrases is inferred, such as the person←wearing a white hat. In establishing the semantic hierarchy of sentences in expressions, the semantic structure of text is established.

[0057] It is understandable that generally establishing the semantic structure of a text can include word segmentation and part-of-speech tagging, syntactic analysis, semantic role labeling, entity recognition and linking, discourse analysis, building a knowledge graph, etc. Through these steps and tools, the semantic structure of the text can be gradually and deeply parsed and constructed, laying a solid foundation for applications such as text understanding, information retrieval, question-and-answer systems, and knowledge graph construction.

[0058] The embodiment of the present invention segments the demand text of the complex large scene image to obtain a phrase set, and establishes a semantic structure between the phrase sets through syntactic analysis, and based on the semantic structure and the position distribution and scale distribution of multiple types of objects in the scene, the phrases are arranged in descending order of information based on the complex large scene image with wide field of view and high resolution, to obtain a list of phrases to be understood.

[0059] Furthermore, in the phrase list, the objects in the phrase are sorted in descending order according to the amount of information in the scene image corresponding to the objects in the phrase. [Example] picture>pool>someone>people>hat; the amount of information is determined by the object scale, object density, etc. The object scale refers to the average pixel size of the bounding box of the object of this type in this scene.

[0060] Furthermore, we can assign weights to phrases to help focus the model processing goals during the iterative understanding process. In the first iteration, the weight of each phrase is 1; in subsequent iterations, the weight is updated to 1-conf, where conf is the confidence of the understood semantics, calculated by cross-modal semantic fusion and mapping, such as Figure 4 shown.

[0061] In one embodiment of the present invention, based on the above output, i.e., the sorted list of phrases, the ultra-high resolution image is content-adaptively downsampled and cropped. When the largest-scale object in the phrase is an image, no cropping is required, and the downsampling operation is performed according to the scale of the second largest object. The degree of downsampling is determined by the performance of the detector used in the system. When the detector accuracy meets the standard, a higher degree of downsampling is beneficial to improve the speed of understanding.

[0062] It is understandable that the ultra-high resolution image is taken by a wide-field high-resolution light field array camera. The image of the present invention is not only the macroscopic scene of the attached figure, but also can be used for ultra-high resolution cell microscopic scenes taken by a microscope and ultra-high resolution astronomical telescopic scenes taken by a telescope.

[0063] In one embodiment of the present invention, a complex large scene image with wide field of view and high resolution is adaptively processed according to the co-occurrence relationship between the phrase list and the object to obtain a local image to be understood. A threshold α is set. If the weight of the phrase is greater than α, it is considered that the semantics referred to by the phrase has not been understood; if the weight of the phrase is less than α, it is considered that the phrase has been understood and does not need to be understood in the iterative process. The downsampled image is divided into N×M small blocks using a sliding window, and a deep neural network based on the attention mechanism is designed to extract the features of the small blocks.

[0064] It is understandable that the present invention first performs semantic detection on the features of the extracted small block image to obtain the objects in the downsampled image, and then identifies the relationship between the objects, such as person-holding-umbrella, person-below-pool. Object detection is achieved and the connection between objects is established. Due to the complex content of the scene and the complex cross-temporal and spatial relationships between objects, it is necessary to combine the visual features of the objects. According to the co-occurrence relationship prior between objects in the scene, the object frames between the detected objects are expanded and fused, and scale-adaptive amplification is performed to support a deep understanding of the image semantics.

[0065] Exemplarily, image features are extracted based on a local image to be understood: the local image to be understood is divided into multiple small blocks, a flattening operation is performed on each small block, a fully connected layer is used to extract temporal features of the small blocks, positional encoding is used to establish positional relationships between the small blocks, and a visual attention mechanism is used to extract visual features between the small blocks to obtain visual features of the local image.

[0066] Exemplarily, text features are extracted based on a phrase list: the text to be understood is transformed into a word vector list, and the text features are extracted using an attention mechanism.

[0067] In one embodiment of the present invention, the above-mentioned output features, text features and image features are transformed and embedded into the same feature space. That is, the visual features and text features are transformed into a unified cross-modal space of visual and text using a visual fully connected layer and a text fully connected layer respectively for fusion and mapping, so as to calculate and obtain the semantic understanding result in the cross-modal space. Figure 5 shown.

[0068] Specifically, feature matching is performed on the features of the unified space, and the response relationship between the small block and the phrase to be understood is measured by the inner product. For targets that are not ultimately needed to be found, this stage only estimates the confidence of the small block, and does not regress the coordinates of the object. The confidence is used to update the weight of the phrase in the text. For the object to be understood in the end, its coordinates need to be regressed. Through repeated iterations, until all objects in the phrase sequence are understood, the understanding results of all objects are obtained, and the upper left and lower right coordinates of the target box and the confidence of conf are output.

[0069] The complex large-scene cross-modal meta-semantic understanding method according to an embodiment of the present invention adopts a strategy of continuously deepening understanding from global to local. Since the amount of information of the post-processed images is compressed, the accuracy and speed of semantic understanding are significantly improved. In addition, a hierarchical understanding framework from "coarse-grained" to "fine-grained" is used to significantly reduce the complexity and speed of understanding.

[0070] In order to implement the above embodiment, Figure 6As shown, this embodiment also provides a complex large-scene cross-modal meta-semantic understanding system 10, which includes an image meta-semantic interpretation module 100, a text meta-semantic interpretation module 200, a wide-field-of-view high-resolution image adaptive processing module 300 and a cross-modal meta-semantic fusion and mapping module 400.

[0071] The image element semantic interpretation module 100 is used to perform image element semantic interpretation on complex large scene images, obtain the position distribution and scale distribution of multiple types of objects in the scene, and establish the cross-scale co-occurrence relationship between multiple types of objects;

[0072] A text meta-semantic interpretation module 200 is used to perform text meta-semantic interpretation on the demand text of the complex large scene image based on the position distribution and scale distribution of multiple types of objects in the scene, and obtain a phrase list whose information content corresponds to the complex scene in descending order;

[0073] A wide-field high-resolution image adaptive processing module 300 is used to adaptively process a wide-field high-resolution complex large scene image according to the co-occurrence relationship between the phrase list and the objects to obtain local image information to be understood;

[0074] The cross-modal meta-semantic fusion and mapping module 400 is used to embed the text features extracted based on the phrase list and the image features extracted based on the local image to be understood into the cross-modal feature space for cross-modal meta-semantic fusion and mapping to obtain the image understanding result of the corresponding text.

[0075] Furthermore, the image element semantic interpretation module 100 is also used for:

[0076] Perform multi-target detection on complex large scenes to obtain detection results, and perform statistical analysis on the detection results to obtain the position distribution and scale distribution of multiple types of objects in the scene;

[0077] A complex relationship recognition model is trained based on the position distribution and scale distribution of multiple types of objects in the scene, and a cross-scale co-occurrence relationship between multiple objects is established, wherein the co-occurrence relationship includes the spatial correlation distance between the objects.

[0078] Furthermore, the text meta-semantic interpretation module 200 is also used for:

[0079] Segmenting the demand text of the complex large scene image to obtain a phrase set, and establishing a semantic structure between the phrase sets through syntactic analysis;

[0080] Based on the semantic structure and the position distribution and scale distribution of multiple types of objects in the scene, the phrases are arranged in descending order of information based on the complex large scene image with wide field of view and high resolution to obtain a list of phrases to be understood.

[0081] Furthermore, the wide-field high-resolution image adaptive processing module 300 is also used for:

[0082] Perform scale-adaptive downsampling operations on complex large scene images according to the scale of the object to be understood;

[0083] According to the co-occurrence relationship between multiple types of objects, the local information of the image to be understood after the downsampling operation is sampled, amplified and expanded.

[0084] Furthermore, the cross-modal meta-semantic fusion and mapping module 400 is also used for:

[0085] The local image to be understood is divided into multiple small blocks, each small block is flattened, the time features of the small blocks are extracted using a fully connected layer, the positional relationship between the small blocks is established using position encoding, and the visual attention mechanism is used to extract the visual features between the small blocks to obtain the visual features of the local image;

[0086] Transform the text to be understood into a list of word vectors and use the attention mechanism to extract text features;

[0087] The visual fully connected layer and the text fully connected layer are used respectively to transform the visual features and text features into a unified cross-modal space of vision and text for fusion and mapping, so as to calculate in the cross-modal space to obtain the semantic understanding results.

[0088] The complex large-scene cross-modal meta-semantic understanding system according to the embodiments of the present invention adopts a strategy of continuously deepening understanding from the global to the local. Since the amount of information of the post-processed images is compressed, the accuracy and speed of semantic understanding are significantly improved. In addition, a hierarchical understanding framework from "coarse-grained" to "fine-grained" is used to significantly reduce the complexity and speed of understanding.

[0089] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0090] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

Claims

1. A complex large-scale cross-modal meta-semantic understanding method, characterized by: include: Perform image meta-semantic interpretation on complex large scene images to obtain the location distribution and scale distribution of multiple types of objects in the scene, and establish cross-scale co-occurrence relationships between multiple types of objects; Based on the position distribution and scale distribution of multiple types of objects in the scene, the demand text of the complex large scene image is interpreted by text meta-semantics to obtain a phrase list whose information content corresponds to the complex scene in descending order; Adaptively processing a complex large scene image with a wide field of view and high resolution according to the co-occurrence relationship between the phrase list and the objects to obtain a local image to be understood; The text features extracted based on the phrase list and the image features extracted based on the local image to be understood are embedded into the cross-modal feature space for cross-modal meta-semantic fusion and mapping, and the image understanding results of the corresponding text are obtained; Adaptively processing the complex large scene image according to the co-occurrence relationship between the phrase list and the objects to obtain a local image to be understood, including: Perform scale-adaptive downsampling operations on complex large scene images according to the scale of the object to be understood; According to the co-occurrence relationship between multiple types of objects, the local image to be understood after the downsampling operation is sampled, amplified and expanded; The text features extracted based on the phrase list and the image features extracted based on the local image to be understood are embedded into the cross-modal feature space for cross-modal meta-semantic fusion and mapping, and the image understanding results of the corresponding text are obtained, including: The local image to be understood is divided into multiple small blocks, each small block is flattened, the time features of the small blocks are extracted using a fully connected layer, the positional relationship between the small blocks is established using position encoding, and the visual attention mechanism is used to extract the visual features between the small blocks to obtain the visual features of the local image; Transform the text to be understood into a list of word vectors and use the attention mechanism to extract text features; The visual fully connected layer and the text fully connected layer are used respectively to transform the visual features and text features into a unified cross-modal space of vision and text for fusion and mapping, so as to calculate in the cross-modal space to obtain the semantic understanding results.

2. The method according to claim 1, characterized in that Perform image meta-semantic interpretation on complex large scene images to obtain the location distribution and scale distribution of multiple types of objects in the scene, and establish cross-scale co-occurrence relationships between multiple types of objects, including: Perform multi-target detection on complex large scenes to obtain detection results, and perform statistical analysis on the detection results to obtain the position distribution and scale distribution of multiple types of objects in the scene; A complex relationship recognition model is trained based on the position distribution and scale distribution of multiple types of objects in the scene, and a cross-scale co-occurrence relationship between multiple objects is established, wherein the co-occurrence relationship includes the spatial correlation distance between the objects.

3. The method according to claim 1, characterized in that Based on the position distribution and scale distribution of multiple types of objects in the scene, the text meta-semantic interpretation of the demand text of the complex large scene image with wide field of view and high resolution is performed to obtain a list of phrases whose information content corresponds to the complex scene in descending order, including: Segmenting the demand text of the complex large scene image to obtain a phrase set, and establishing a semantic structure between the phrase sets through syntactic analysis; Based on the semantic structure and the position distribution and scale distribution of multiple types of objects in the scene, the phrases are arranged in descending order of information based on the complex large scene image with wide field of view and high resolution to obtain a list of phrases to be understood.

4. A complex large-scale cross-modal meta-semantic understanding system, characterized by: include: Image meta-semantic interpretation module, used to interpret image meta-semantics of complex large scene images, obtain the position distribution and scale distribution of multiple types of objects in the scene, and establish cross-scale co-occurrence relationships between multiple types of objects; A text meta-semantic interpretation module, used to perform text meta-semantic interpretation on the demand text of the complex large scene image based on the position distribution and scale distribution of multiple types of objects in the scene, and obtain a phrase list whose information content corresponds to the complex scene in descending order; A wide-field high-resolution image adaptive processing module, used for adaptively processing a wide-field high-resolution complex large-scene image according to the co-occurrence relationship between the phrase list and the object to obtain local image information to be understood; A cross-modal meta-semantic fusion and mapping module is used to embed text features extracted based on the phrase list and image features extracted based on the local image to be understood into the cross-modal feature space for cross-modal meta-semantic fusion and mapping, and obtain the image understanding result of the corresponding text; The wide-field high-resolution image adaptive processing module is also used for: Perform scale-adaptive downsampling operations on complex large scene images according to the scale of the object to be understood; According to the co-occurrence relationship between multiple types of objects, the local information of the image to be understood after the downsampling operation is sampled, amplified and expanded; The cross-modal meta-semantic fusion and mapping module is also used for: The local image to be understood is divided into multiple small blocks, each small block is flattened, the time features of the small blocks are extracted using a fully connected layer, the positional relationship between the small blocks is established using position encoding, and the visual attention mechanism is used to extract the visual features between the small blocks to obtain the visual features of the local image; Transform the text to be understood into a list of word vectors and use the attention mechanism to extract text features; The visual fully connected layer and the text fully connected layer are used respectively to transform the visual features and text features into a unified cross-modal space of vision and text for fusion and mapping, so as to calculate in the cross-modal space to obtain the semantic understanding results.

5. The system according to claim 4, characterized in that The image element semantic interpretation module is also used for: Perform multi-target detection on complex large scenes to obtain detection results, and perform statistical analysis on the detection results to obtain the position distribution and scale distribution of multiple types of objects in the scene; A complex relationship recognition model is trained based on the position distribution and scale distribution of multiple types of objects in the scene, and a cross-scale co-occurrence relationship between multiple objects is established, wherein the co-occurrence relationship includes the spatial correlation distance between the objects.

6. The system according to claim 5, characterized in that The text meta-semantic interpretation module is also used for: Segmenting the demand text of the complex large scene image to obtain a phrase set, and establishing a semantic structure between the phrase sets through syntactic analysis; Based on the semantic structure and the position distribution and scale distribution of multiple types of objects in the scene, the phrases are arranged in descending order of information based on the complex large scene image with wide field of view and high resolution to obtain a list of phrases to be understood.

Images