Panoramic scene graph generation method and device, equipment and storage medium
By combining the visual text feature interaction mechanism of large language model and panoramic image segmentation model in panoramic scene map generation, the problem of insufficient utilization of text information in the existing technology is solved, and a more accurate and unbiased panoramic scene map generation is achieved. Especially when dealing with uncommon relationships, the prediction performance is significantly improved.
Patent Information
- Application Number
- CN202410942372.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-15
- Publication Date
- 2024-11-08
AI Technical Summary
The existing panoramic scene graph generation scheme lacks the utilization of rich text information, resulting in the inability to fully capture the relationships between complex objects and the semantic complexity of scenes, especially when dealing with uncommon relationship categories, the effect is limited.
By analyzing the specified images, obtaining the relationship proposed text features and judging the text features, combining visual features for fusion and decoding, using large language models and panoramic image segmentation models to design the visual text feature interaction mechanism to generate an unbiased panoramic scene map.
It realizes more accurate relationship prediction with the assistance of rich text information, generates unbiased panoramic scene maps, and improves the ability to identify complex relationships, especially when dealing with uncommon relationships, which significantly improves prediction performance.
Smart Images

Figure CN118918203A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer vision technology, and in particular to a method, device, equipment and storage medium for generating a panoramic scene graph. Background Art
[0002] The rapid development of computer vision has enabled many breakthrough results in visual tasks such as image classification, semantic segmentation, and visual relationship detection in a short period of time. These results are driven by convolutional neural networks and fully convolutional networks.
[0003] On this basis, the research on image understanding has gradually developed from low-level feature extraction to high-level semantic learning. The next step is to infer the semantic relationship between multiple objects, thereby promoting the development of multimodal tasks such as visual question answering, image description, and visual common sense reasoning. Among them, the emergence of panoramic scene graphs provides a platform for inferring visual scenes for image understanding.
[0004] A significant technical flaw in existing panoramic scene graph generation schemes is the lack of utilization of rich text information. Existing panoramic scene graph generation schemes mainly rely on visual information processing, or in the best case, only use simple text labels and basic descriptive information to assist in the recognition and classification of object relationships. The limitation of this is that simple text information cannot fully capture the complex relationships between objects and the semantic complexity of the scene, especially when dealing with uncommon relationship categories. Summary of the invention
[0005] Embodiments of the present invention provide a method, device, equipment and storage medium for generating a panoramic scene graph.
[0006] In a first aspect, an embodiment of the present invention provides a method for generating a panoramic scene graph, the method comprising:
[0007] Analyze the specified image to obtain the relationship proposal text features and relationship judgment text features;
[0008] Process the specified image to obtain visual features, object categories and masks;
[0009] The visual features are fused with the relation proposal text features and the relation judgment text features respectively;
[0010] The features obtained by fusing the visual features with the relation proposal text features are decoded to obtain the relation proposal prediction results; the features obtained by fusing the visual features with the relation judgment text features are decoded to obtain the relation judgment prediction results;
[0011] The relationship proposal prediction result is integrated with the relationship judgment prediction result to obtain the relationship prediction result;
[0012] Combine the relationship prediction results, object categories and masks to generate a panoramic scene graph.
[0013] In some implementations of the first aspect, analyzing a specified image to obtain a relationship proposal text feature and a relationship judgment text feature includes:
[0014] Analyze the specified image to obtain relationship proposal prompt questions and relationship judgment prompt questions;
[0015] According to the relationship proposal prompt question and the relationship judgment prompt question, the relationship proposal text features and the relationship judgment text features are obtained.
[0016] In some implementations of the first aspect, acquiring the relationship proposal text features and the relationship judgment text features according to the relationship proposal prompt question and the relationship judgment prompt question includes:
[0017] Ask the large language model questions based on the relationship proposal prompt questions and the relationship judgment prompt questions to obtain the relationship proposal text and the relationship judgment text;
[0018] Use a text feature extractor to extract text features from the relationship proposal text and the relationship judgment text to obtain relationship proposal text features and relationship judgment text features; or,
[0019] According to the relationship proposal prompt question and the relationship judgment prompt question, the corresponding relationship proposal text features and relationship judgment text features are searched from the text feature database.
[0020] In some implementations of the first aspect, processing a specified image to obtain visual features, object categories, and masks includes:
[0021] Use the panoramic image segmentation model to process the specified image to obtain visual features, object categories, and masks.
[0022] In some implementations of the first aspect, decoding a feature obtained by fusing a visual feature with a relation proposal text feature to obtain a relation proposal prediction result includes:
[0023] The features obtained by fusing the visual features with the relationship proposal text features are input into the relationship proposal decoder. In the relationship proposal decoder, the visual features are processed by the self-attention layer to concentrate the relationship information in the visual features. The processed visual features and the relationship proposal text features are processed by the cross-attention layer to integrate the common sense knowledge of the potential relationship into the visual features. Based on this, the possible relationship between the subject and the object is predicted by the relationship prediction layer to obtain the relationship proposal prediction result.
[0024] In some implementations of the first aspect, decoding a feature obtained by fusing a visual feature with a relationship judgment text feature to obtain a relationship judgment prediction result includes:
[0025] The features obtained by fusing the visual features with the relationship judgment text features are input into the relationship judgment decoder. In the relationship judgment decoder, the visual features interact independently with the relationship judgment text features corresponding to each <subject-relationship-object> triplet, and all the relationship prediction results obtained by parallel prediction of the relationship prediction layer are merged to obtain the relationship judgment prediction result.
[0026] In some implementations of the first aspect, the relationship proposal prediction result and the relationship judgment prediction result are merged to obtain the relationship prediction result, including:
[0027] Input the relation proposal text features, relation judgment text features and relation proposal text features into the gating network to generate the relation proposal prediction results and the weights corresponding to the relation judgment prediction results;
[0028] According to the weights corresponding to the relationship proposal prediction results and the relationship judgment prediction results, the relationship proposal prediction results and the relationship judgment prediction results are weighted summed to obtain the relationship prediction results.
[0029] In a second aspect, an embodiment of the present invention provides a panoramic scene graph generation device, the device comprising:
[0030] An analysis module is used to analyze a specified image to obtain relationship proposal text features and relationship judgment text features;
[0031] A processing module is used to process the specified image to obtain visual features, object categories and masks;
[0032] A fusion module is used to fuse the visual features with the relation proposal text features and the relation judgment text features respectively;
[0033] A decoding module is used to decode the features obtained by fusing the visual features with the relationship proposal text features to obtain the relationship proposal prediction results; and to decode the features obtained by fusing the visual features with the relationship judgment text features to obtain the relationship judgment prediction results;
[0034] The fusion module is also used to fuse the relationship proposal prediction result with the relationship judgment prediction result to obtain the relationship prediction result;
[0035] The generation module is used to combine the relationship prediction results, object categories and masks to generate a panoramic scene graph.
[0036] In a third aspect, an embodiment of the present invention provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described above.
[0037] In a fourth aspect, an embodiment of the present invention provides a non-transitory computer-readable storage medium storing computer instructions, where the computer instructions are used to enable a computer to execute the method described above.
[0038] In the embodiment of the present invention, with the assistance of rich text information, more accurate relationship prediction can be achieved, thereby generating an unbiased panoramic scene map.
[0039] It should be understood that the contents described in the summary of the invention are not intended to limit the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] The above and other features, advantages and aspects of the embodiments of the present invention will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. The accompanying drawings are provided for a better understanding of the present invention and do not constitute a limitation of the present invention. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:
[0041] Figure 1 A flow chart of a method for generating a panoramic scene graph provided by an embodiment of the present invention is shown;
[0042] Figure 2 A schematic diagram of a panoramic scene graph generation result provided by an embodiment of the present invention is shown;
[0043] Figure 3 A structural diagram of a panoramic scene graph generating device provided by an embodiment of the present invention is shown;
[0044] Figure 4 A structural diagram of an exemplary electronic device capable of implementing an embodiment of the present invention is shown. DETAILED DESCRIPTION
[0045] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0046] In addition, the term "and / or" in the present invention is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in the present invention generally indicates that the associated objects before and after are in an "or" relationship.
[0047] In order to solve the technical problems arising from the background technology, the embodiments of the present invention provide a method, device, equipment and storage medium for generating a panoramic scene graph. Specifically, the specified image is analyzed to obtain relationship proposal text features and relationship judgment text features; the specified image is processed to obtain visual features, object categories and masks; the visual features are fused with relationship proposal text features and relationship judgment text features respectively; the features obtained after the fusion of visual features and relationship proposal text features are decoded to obtain relationship proposal prediction results; the features obtained after the fusion of visual features and relationship judgment text features are decoded to obtain relationship judgment prediction results; the two types of prediction results are fused to obtain relationship prediction results; and the relationship prediction results, object categories and masks are combined to generate a panoramic scene graph.
[0048] In this way, with the assistance of rich text information, more accurate relationship prediction can be achieved, thereby generating an unbiased panoramic scene map.
[0049] In conjunction with the accompanying drawings, a method, device, equipment and storage medium for generating a panoramic scene graph provided by an embodiment of the present invention are described in detail below through specific embodiments.
[0050] Figure 1 A flow chart of a method for generating a panoramic scene graph provided by an embodiment of the present invention is shown. Figure 1 As shown, the panoramic scene graph generation method 100 may include the following steps:
[0051] S110, analyzing the designated image to obtain relationship proposal text features and relationship judgment text features.
[0052] In some embodiments, a designated image may be analyzed to obtain relationship proposal prompt questions and relationship judgment prompt questions, and relationship proposal text features and relationship judgment text features may be obtained based on the relationship proposal prompt questions and relationship judgment prompt questions.
[0053] Optionally, questions can be asked to a large language model (such as GPT-4) based on relationship proposal prompt questions and relationship judgment prompt questions to obtain relationship proposal text and relationship judgment text, and a text feature extractor (such as Bert) can be used to extract text features from the relationship proposal text and relationship judgment text to obtain relationship proposal text features and relationship judgment text features.
[0054] Alternatively, in order to avoid the computational burden caused by repeatedly calling a large language model, corresponding relationship proposal text features and relationship judgment text features can be searched from a text feature database according to the relationship proposal prompt question and the relationship judgment prompt question. The relationship proposal text features and relationship judgment text features in the text feature database are obtained by pre-processing all possible object and relationship combinations in the above manner.
[0055] S120, processing the designated image to obtain visual features, object categories, and masks.
[0056] In some embodiments, a panoramic image segmentation model (e.g., Mask2Former) can be used to process a specified image to obtain visual features (including visual features of the <subject-object> corresponding area extracted from the full image features, and spatial features of the <subject-object> area relative to the full image), object categories, and masks.
[0057] S130, respectively fusing the visual features with the relationship proposal text features and the relationship judgment text features.
[0058] That is to say, the visual features are fused with the relation proposal text features, and the visual features are fused with the relation judgment text features.
[0059] S140, decoding the features obtained by fusing the visual features with the relationship proposal text features to obtain the relationship proposal prediction results; decoding the features obtained by fusing the visual features with the relationship judgment text features to obtain the relationship judgment prediction results.
[0060] In some embodiments, the features obtained by fusing visual features with relationship proposal text features can be input into a relationship proposal decoder. In the relationship proposal decoder, the visual features are processed by a self-attention layer to concentrate the relationship information in the visual features. The processed visual features and the relationship proposal text features are processed by a cross-attention layer to integrate common sense knowledge of potential relationships into the visual features. Based on this, the possible relationship between the subject and the object is predicted by a relationship prediction layer to obtain a relationship proposal prediction result.
[0061] The features obtained by fusing the visual features with the relationship judgment text features can be input into the relationship judgment decoder. In the relationship judgment decoder, the visual features interact independently with the relationship judgment text features corresponding to each <subject-relationship-object> triplet, and all the relationship prediction results obtained by parallel prediction of the relationship prediction layer are merged to obtain the relationship judgment prediction result.
[0062] S150, the relationship proposal prediction result is integrated with the relationship judgment prediction result to obtain a relationship prediction result.
[0063] In some embodiments, the relationship proposal text features, the relationship judgment text features, and the relationship proposal text features can be input into the gating network to generate the relationship proposal prediction results and the weights corresponding to the relationship judgment prediction results. Then, according to the weights corresponding to the relationship proposal prediction results and the relationship judgment prediction results, the relationship proposal prediction results and the relationship judgment prediction results are weighted and summed to obtain the relationship prediction results.
[0064] S160, combining the relationship prediction results, object categories, and masks to generate a panoramic scene graph.
[0065] For further understanding, the panoramic scene graph generation method 100 provided by the embodiment of the present invention is described in detail below in conjunction with a specific embodiment:
[0066] The panoramic scene graph generation method 100 provided in an embodiment of the present invention is committed to using the rich knowledge contained in a large language model (such as GPT-4) to solve the long-tail distribution problem of the relationship categories between objects, so as to improve the overall performance of the panoramic scene graph generation task. Here, a large language model is used to generate text information and visual information provided by a panoramic image segmentation model (such as Mask2Former). By designing a visual text feature interaction mechanism, unbiased relationship prediction is achieved, and then an unbiased panoramic scene graph is generated. The specific process is divided into two parts, namely, object relationship knowledge augmentation guided by a large language model, and a visual text feature interaction mechanism and unbiased relationship prediction, as shown below:
[0067] Object relationship knowledge augmentation guided by large language model:
[0068] This part aims to introduce additional text information to the panoramic scene graph generation task with the common sense knowledge contained in the large language model, and then lay the foundation for unbiased relationship prediction. First, with the help of the thinking chain and contextual learning mechanism of the large language model, two prompts are designed for the image to prompt the large language model to play two roles: one is the relationship proposal prompt (RP-Prompt) as a relationship proposer (RP), which makes suggestions and explanations for the possible relationships of a given <subject-object> pair. These suggestions reflect the common relationships in the real world; the other is the relationship judgment prompt (RJ-Prompt) as a relationship judge (RJ), which evaluates the relationship in a given <subject-relation-object> triple, and even uncommon relationships can be described in detail. For example, for a pair of <subject-object> "person" and "dog", the relationship proposal prompt asks the large language model "In the image, what is the possible relationship between the person and the dog and what is the reason?" The large language model answers "pulling", "playing", "holding", etc., and attaches corresponding reasons. For the relationship judgment prompt question, the large language model is asked "Is it possible for the 'wearing' relationship between the person and the dog in the image to exist? Please explain the reason", and the large language model will answer "It is impossible for a person to wear a dog because this does not exist in reality". Through these two prompt questions, rich prior information that conforms to the laws of the real world can be obtained, providing a solid information foundation for unbiased relationship prediction of panoramic scene graphs. The process can be expressed as:
[0069] Desc RP =LLM(Prom RP );Desc RJ =LLM(Prom RJ ),
[0070] Among them, LLM(·) represents the large language model, Prom RP and Prom RJ They represent the relation proposal prompt question and relation judgment prompt question input to the large language model respectively. RP and Desc RJ They represent the extracted relationship proposal text and relationship judgment text respectively.
[0071] After successfully obtaining the relation proposal text (RP-Description) used to describe all potential relations between the subject and the object and the relation judgment text (RJ-Description) for judging and reasoning about a specific <subject-relation-object> triple, a text feature extractor (such as Bert) is used here to convert the above text into corresponding text features, namely, relation proposal text features (RP-Feature) and relation judgment text features (RJ-Feature). The process can be expressed as:
[0072] F RP =TextEncoder(Desc RP );F RJ =TextEncoder(Desc RJ ),
[0073] Among them, TextEncoder(·) represents the text feature extractor, F RP and F RJ They represent relation proposal text features and relation judgment text features respectively.
[0074] In order to improve the efficiency of the model in the training and prediction stages and avoid the computational burden of repeatedly calling a large language model, all possible combinations of objects and relationships are pre-processed, text features are obtained and stored in a text feature database, thereby improving the efficiency of model operation. Next, the extracted text features are interactively fused with visual features for unbiased panoramic scene graph generation.
[0075] Visual-text feature interaction mechanism and unbiased relationship prediction:
[0076] This part is dedicated to building a visual text feature interaction mechanism. First, the relation proposal text features, relation judgment text features and visual features obtained by the panoramic image segmentation model (such as Mask2Former) obtained in the previous part are interactively fused separately. The visual features include the visual features of the <subject-object> corresponding area extracted from the full image features, and the spatial features of the <subject-object> area relative to the full image. Subsequently, the fused multimodal features are input into two Transformer multimodal decoders designed based on the visual text feature interaction mechanism, namely the relation proposal decoder (RP-Decoder) and the relation judgment decoder (RJ-Decoder), to respectively predict the relationship between the subject and the object in the image. Among them, the relation proposal decoder focuses on decoding the common relationship between the given <subject-object> pair, while the relation judgment decoder pays more attention to predicting uncommon relationships. In order to combine these two prediction capabilities and maximize the performance of relation prediction, a relation prediction result fusion module will be developed here to fuse the relation prediction outputs of the two decoders into the final prediction result. The following is a detailed introduction to the relation proposal decoder, relation judgment decoder, and relation prediction result fusion module.
[0077] The relation proposal decoder takes the fusion of the relation proposal text features and the subject-object pair visual features as input to predict the relationship between the corresponding subject-object pairs. To achieve this goal, the visual features are first processed by the self-attention layer to concentrate the relationship information in the visual features. Next, the processed visual features will be processed with the relation proposal text features through the cross-attention mechanism to integrate the common sense knowledge of the potential relationship into the visual features. Finally, through the relationship prediction layer (such as a multi-layer perceptron network), the prediction of the possible relationship between the subject and the object is realized, that is, the relationship proposal prediction result. This process can be expressed by the following formula:
[0078]
[0079] Among them, F V represents visual features, SA(·) represents the self-attention layer, and F RP represents the relation proposal text feature, CA(·,·) represents the cross attention layer, RelPred(·) represents the relation prediction layer, R RP Represents the relationship proposal prediction result.
[0080] The relation judgment decoder takes the fusion of relation judgment text features and subject-object visual features as input. Given that the relation judgment text features represent the independent features of each <subject-relation-object> triple, this decoder is designed to allow the visual features to interact independently with the relation judgment text corresponding to each <subject-relation-object> triple. This is especially important for the prediction of uncommon relations. Finally, all the relation prediction results predicted in parallel by the relation prediction layer are merged to form the relation prediction result between the subject and object pairs, that is, the relation judgment prediction result. The results of each relation in this process are obtained in the same way as the relation proposal prediction results. The final merging process can be expressed by the following formula:
[0081]
[0082] Among them, Concat(·) represents the relation merging operation. represents the prediction result of each relationship obtained, K is the number of relationships, R RJ Represents the relationship judgment prediction result.
[0083] The relationship prediction result fusion module effectively integrates the relationship proposal prediction results and the relationship judgment prediction results, combining the advantages of both to enhance the accuracy and unbiasedness of relationship prediction. To achieve this goal, a gating network is designed here, which takes the subject-object pair visual features and two types of text features as input to generate two sets of weights, and then weights the relationship proposal prediction results and the relationship judgment prediction results to obtain the final relationship prediction results. This process can be expressed by the following formula:
[0084] W RP , W RJ =GateNet(F V , F RP , F FJ ),
[0085] R final =W RP ×R RP +W RJ ×R RJ ,
[0086] Among them, W RP and W RJ They represent the weights of the relationship proposal prediction results and the relationship judgment prediction results, GateNet(·) represents the gated network, and R final Represents the relationship prediction result.
[0087] Finally, the object categories and masks predicted by the panoramic image segmentation model are combined to form an unbiased panoramic scene graph. Figure 2As shown, the rare relations marked by the yellow dotted box can achieve more accurate relation prediction with the assistance of text information.
[0088] In summary, the embodiments of the present invention have at least the following innovative features:
[0089] 1. Introduction and application of large language models:
[0090] Use large language models (such as GPT-4) to generate rich text descriptions of possible relationships between objects, including but not limited to common and uncommon relationships. This application can better capture and understand the complex interactions between objects in the scene, especially in solving long-tail distribution problems.
[0091] 2. Object relationship knowledge augmentation strategy:
[0092] Two types of prompt questions are designed to prompt the large language model to provide information from two perspectives: one is to provide possible relationship suggestions and explanations as a relationship proposer; the other is to evaluate the rationality of a given relationship as a relationship judge. This strategy enhances the model's understanding of object relationships by providing rich common sense and contextual information.
[0093] 3. Visual-text feature interaction mechanism:
[0094] Develop a visual-text feature interaction mechanism that can fuse text features with visual features provided by panoramic image segmentation models (such as Mask2Former), thereby making full use of multimodal data to enhance the accuracy and unbiasedness of relationship prediction. It includes: a) Dual decoder design: Implement two Transformer-based multimodal decoders, namely the relationship proposal decoder and the relationship judgment decoder, to handle the prediction of common and uncommon relationships. This dual decoder architecture enables the model to handle various types of relationships more flexibly and optimizes the performance of relationship prediction. b) Relationship prediction result fusion module: The outputs of the relationship proposal decoder and the relationship judgment decoder are effectively fused through a gating network to produce a final relationship prediction result that combines the advantages of both. This fusion process strengthens the model's ability to predict rare relationships while maintaining high accuracy for common relationships.
[0095] 4. Optimization of efficiency and practicality:
[0096] In order to improve the efficiency of the model in the training and prediction stages and avoid the computational burden of repeatedly calling the large language model, all possible object and relationship combinations are pre-processed and the text features are stored in the text feature database. This pre-processing strategy significantly improves the operating efficiency and practicality of the entire model.
[0097] Furthermore, according to the embodiments of the present invention, at least the following technical effects are achieved:
[0098] 1. Optimization and improvement of panoramic scene graph generation tasks:
[0099] The present invention significantly enhances the accuracy and unbiasedness of panoramic scene graph generation by introducing a large language model and visual text feature interaction technology. Specifically, it can comprehensively utilize text and visual information to accurately identify and predict complex relationships between objects, especially when dealing with uncommon relationships, showing higher prediction capabilities than existing solutions. This not only improves the robustness of the model, but also makes scene understanding more comprehensive and detailed, thereby providing richer and more accurate information support for decision-making in complex open scenes.
[0100] 2. Significant contributions to application areas that rely on detailed scene understanding:
[0101] In the fields of unmanned driving, surgical robots, and embodied intelligence, this invention significantly improves the ability of environmental perception. By accurately analyzing the panoramic scene graph, the intelligent system can identify key objects and their relationships and make more accurate decisions, thereby greatly improving the safety and reliability of task execution. For example, in the field of unmanned driving, the enhanced scene understanding ability can effectively predict and analyze various elements and potential risks in complex traffic environments to ensure driving safety.
[0102] 3. Optimization of image / video retrieval and recommendation systems:
[0103] The present invention can be used to more deeply mine and understand high-level image / video semantic information, thereby accurately matching the user's search and recommendation needs. This capability not only improves the accuracy and efficiency of retrieval, but also recommends more in-demand content based on the user's interests and preferences, thereby improving user satisfaction and the overall performance of the recommendation system.
[0104] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described order of actions, because according to the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present invention.
[0105] The above is an introduction to a method embodiment. The following is a further explanation of the solution of the present invention through an apparatus embodiment.
[0106] Figure 3 FIG. 4 shows a structural diagram of a panoramic scene graph generating device provided by an embodiment of the present invention. Figure 3 As shown, the panoramic scene graph generating device 300 may include:
[0107] The analysis module 310 is used to analyze the specified image to obtain relationship proposal text features and relationship judgment text features.
[0108] The processing module 320 is used to process the specified image to obtain visual features, object categories and masks.
[0109] The fusion module 330 is used to fuse the visual features with the relationship proposal text features and the relationship judgment text features respectively.
[0110] The decoding module 340 is used to decode the features obtained by fusing the visual features with the relationship proposal text features to obtain the relationship proposal prediction results; and to decode the features obtained by fusing the visual features with the relationship judgment text features to obtain the relationship judgment prediction results.
[0111] The fusion module 330 is further used to fuse the relationship proposal prediction result with the relationship judgment prediction result to obtain a relationship prediction result.
[0112] The generation module 350 is used to generate a panoramic scene graph by combining the relationship prediction results, object categories and masks.
[0113] Understandably, Figure 3 Each module / unit in the panoramic scene graph generating device 300 shown in FIG. Figure 1 The functions of the various steps in the panoramic scene graph generation method 100 shown can achieve their corresponding technical effects, and for the sake of brevity, they will not be repeated here.
[0114] Figure 4 A block diagram of an exemplary electronic device capable of implementing an embodiment of the present invention is shown. Electronic device 400 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic device 400 may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown in the present invention, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required in the present invention.
[0115] like Figure 4As shown, the electronic device 400 may include a computing unit 401, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 402 or a computer program loaded from a storage unit 408 into a random access memory (RAM) 403. In the RAM 403, various programs and data required for the operation of the electronic device 400 can also be stored. The computing unit 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.
[0116] Multiple components in the electronic device 400 are connected to the I / O interface 405, including: an input unit 406, such as a keyboard, a mouse, etc.; an output unit 407, such as various types of displays, speakers, etc.; a storage unit 408, such as a disk, an optical disk, etc.; and a communication unit 409, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 409 allows the electronic device 400 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.
[0117] The computing unit 401 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 401 performs the various methods and processes described above, such as method 100. For example, in some embodiments, the method 100 may be implemented as a computer program product, including a computer program, which is tangibly contained in a computer-readable medium, such as a storage unit 408. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 400 via ROM 402 and / or communication unit 409. When the computer program is loaded into RAM 403 and executed by the computing unit 401, one or more steps of the method 100 described above may be performed. Alternatively, in other embodiments, the computing unit 401 may be configured to perform the method 100 in any other appropriate manner (e.g., by means of firmware).
[0118] The various embodiments described above in the present invention can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: being implemented in one or more computer programs, which may be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general programmable processor, which may receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0119] The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.
[0120] In the context of the present invention, a computer-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a computer-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0121] It should be noted that the present invention also provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute method 100 and achieve the corresponding technical effect achieved by executing the method in an embodiment of the present invention. For the sake of concise description, they will not be repeated here.
[0122] In addition, the present invention also provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the method 100 is implemented.
[0123] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution disclosed in the present invention can be achieved, and the present invention is not limited here.
[0124] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims
1. A method for generating a panoramic scene graph, characterized in that: The method comprises: Analyze the specified image to obtain the relationship proposal text features and relationship judgment text features; Process the specified image to obtain visual features, object categories and masks; fusing the visual feature with the relationship proposal text feature and the relationship judgment text feature respectively; Decoding the feature obtained by fusing the visual feature with the relationship proposal text feature to obtain a relationship proposal prediction result; decoding the feature obtained by fusing the visual feature with the relationship judgment text feature to obtain a relationship judgment prediction result; The relationship proposal prediction result is integrated with the relationship judgment prediction result to obtain a relationship prediction result; The relationship prediction result, the object category and the mask are combined to generate a panoramic scene graph.
2. The method according to claim 1, characterized in that The analyzing of the specified image to obtain the relationship proposal text features and the relationship judgment text features includes: Analyze the specified image to obtain relationship proposal prompt questions and relationship judgment prompt questions; According to the relationship proposal prompt question and the relationship judgment prompt question, relationship proposal text features and relationship judgment text features are obtained.
3. The method according to claim 2, characterized in that The acquiring, according to the relationship proposal prompt question and the relationship judgment prompt question, relationship proposal text features and relationship judgment text features includes: Asking questions to the large language model based on the relationship proposal prompt question and the relationship judgment prompt question to obtain a relationship proposal text and a relationship judgment text; Use a text feature extractor to extract text features from the relationship proposal text and the relationship judgment text to obtain relationship proposal text features and relationship judgment text features; or, According to the relationship proposal prompt question and the relationship judgment prompt question, corresponding relationship proposal text features and relationship judgment text features are searched from a text feature database.
4. The method according to claim 1, characterized in that: The processing of the specified image to obtain visual features, object categories and masks includes: Use the panoramic image segmentation model to process the specified image to obtain visual features, object categories, and masks.
5. The method according to claim 1, characterized in that The decoding of the feature obtained by fusing the visual feature with the relation proposal text feature to obtain a relation proposal prediction result includes: The features obtained by fusing the visual features with the relationship proposal text features are input into a relationship proposal decoder. In the relationship proposal decoder, the visual features are processed by a self-attention layer to concentrate the relationship information in the visual features. The processed visual features and the relationship proposal text features are processed by a cross-attention layer to integrate the common sense knowledge of the potential relationship into the visual features. Based on this, the possible relationship between the subject and the object is predicted by a relationship prediction layer to obtain a relationship proposal prediction result.
6. The method according to claim 1, characterized in that The decoding of the feature obtained by fusing the visual feature with the relationship judgment text feature to obtain the relationship judgment prediction result includes: The feature obtained by fusing the visual feature with the relationship judgment text feature is input into the relationship judgment decoder. In the relationship judgment decoder, the visual feature interacts independently with the relationship judgment text feature corresponding to each <subject-relationship-object> triple, and all relationship prediction results obtained by parallel prediction of the relationship prediction layer are merged to obtain a relationship judgment prediction result.
7. The method according to claim 1, characterized in that The step of fusing the relationship proposal prediction result with the relationship judgment prediction result to obtain a relationship prediction result includes: Inputting the relation proposal text feature, the relation judgment text feature and the relation proposal text feature into a gating network to generate the relation proposal prediction result and the weight corresponding to the relation judgment prediction result; According to the weights corresponding to the relationship proposal prediction result and the relationship judgment prediction result, the relationship proposal prediction result and the relationship judgment prediction result are weightedly summed to obtain a relationship prediction result.
8. A panoramic scene graph generating device, characterized in that: The device comprises: An analysis module is used to analyze a specified image to obtain relationship proposal text features and relationship judgment text features; A processing module is used to process the specified image to obtain visual features, object categories and masks; A fusion module, used to fuse the visual feature with the relationship proposal text feature and the relationship judgment text feature respectively; A decoding module, configured to decode the feature obtained by fusing the visual feature with the relationship proposal text feature to obtain a relationship proposal prediction result; and decode the feature obtained by fusing the visual feature with the relationship judgment text feature to obtain a relationship judgment prediction result; The fusion module is further used to fuse the relationship proposal prediction result with the relationship judgment prediction result to obtain a relationship prediction result; A generation module is used to generate a panoramic scene graph by combining the relationship prediction result, the object category and the mask.
9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.
10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to make a computer execute the method according to any one of claims 1 to 7.
