A method and device for visual relationship detection guided by contextual knowledge

Abstract

The present invention discloses a contextual knowledge-guided visual relationship detection method and device, comprising: acquiring a scene image, determining the semantic vector and context vector of an object pair in the scene image, extracting a local image containing the object pair from the scene image and determining a position mask of the object pair; utilizing a contextual knowledge generation module to extract the contextual knowledge of the object pair based on the context vector of the object pair; utilizing a visual feature extraction module to extract the visual features of the object pair based on the local image and the position mask of the object pair; utilizing a semantic feature extraction module to extract the semantic features of the object pair based on the semantic vector of the object pair; utilizing a visual relationship detection module to calculate a visual prediction result of the object pair based on the visual features and semantic features of the object pair; utilizing a comprehensive judgment module to comprehensively judge the visual prediction result and the contextual knowledge of the object pair to obtain a contextual knowledge-guided visual relationship of the object pair. The method and device improve the accuracy of visual relationship detection through contextual knowledge guidance.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and robotics technology, and particularly relates to a contextual knowledge-guided visual relationship detection method and device. Background Art

[0002] During the operation of the robot, it is necessary to analyze the scene, detect the visual relationships between people, people and objects, and objects and objects, and construct scene graphs through visual relationships to better complete tasks such as visual language navigation, visual language calibration, and visual question answering.

[0003] However, current visual relationship detection methods often directly extract relationship features from images for prediction, such as the method disclosed in the document "Hengyue Liu, Ning Yan, Masood Mortazavi, and Bir Bhanu. Fully convolutional scene graph generation. In Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition, CVPR, 2021.", or integrate externally constructed knowledge graphs, such as the method disclosed in the document "Zareian A, Karaman S, Chang S F. Bridging knowledge graphs to generate scene graphs. In Proceedings of the European Conference on Computer Vision, ECCV, 2020.". However, these visual relationship detection methods ignore the important position of contextual knowledge in relationship detection. Summary of the Invention

[0004] In view of the above, an object of the present invention is to provide a contextual knowledge-guided visual relationship detection method and device, so as to improve the accuracy of visual relationship detection through contextual knowledge guidance.

[0005] To achieve the above-mentioned object of the invention, a first aspect of the embodiment provides a contextual knowledge-guided visual relationship detection method, comprising the following steps:

[0006] Acquire a scene image, determine a semantic vector and a context vector of an object pair in the scene image, extract a local image containing the object pair from the scene image, and determine a position mask of the object pair;

[0007] Visual relationship detection is performed using a visual relationship detection model that includes a contextual knowledge generation module, a visual feature extraction module, a semantic feature extraction module, a visual relationship detection module, and a comprehensive judgment module, including: using the contextual knowledge generation module to generate contextual knowledge of the object pair based on the context vector of the object pair, using the visual feature extraction module to extract the visual features of the object pair based on the local image and the position mask of the object pair, using the semantic feature extraction module to extract the semantic features of the object pair based on the semantic vector of the object pair, using the visual relationship detection module to calculate the visual prediction results of the object pair based on the visual features and semantic features of the object pair, and using the comprehensive judgment module to comprehensively judge the visual prediction results and the contextual knowledge of the object pair to obtain the visual relationship of the object pair guided by the contextual knowledge.

[0008] In one embodiment, determining the semantic vector and the context vector of the object pair in the scene image includes:

[0009] Detect the position and category of objects in the scene image, form a position vector with the coordinates of the corner points and the center point of the bounding box representing the position, and convert the category into a word vector as a semantic vector;

[0010] The semantic vectors of the two objects are concatenated to form the semantic vector of the object pair;

[0011] The position vector and semantic vector of the object are spliced together to form a context vector, and then the context vectors of the two objects are spliced together to form the context vector of the object pair, or the position vectors of the two objects are spliced together to form the position vector of the object pair, and then the position vector and semantic vector of the object pair are spliced together to form the context vector of the object pair.

[0012] In one embodiment, extracting a local image containing an object pair from a scene image and determining a position mask of the object pair includes:

[0013] Detect the position of objects in the scene image and intercept the local image containing the object pair based on the bounding box of the two object positions;

[0014] In the local image, the area within the bounding box of each object is set to 1 and the other areas are set to 0 to obtain the position mask of each object. The position masks of two objects constitute the position mask of the object pair.

[0015] In one embodiment, the extracting contextual knowledge of the object pair based on the context vector of the object pair by using the contextual knowledge generation module includes:

[0016] The context vector of the object pair is input into the context knowledge generation module. After calculation, the context probabilities of all relationship categories are output. According to the threshold, high context probabilities are screened and set to 1, and all remaining context probabilities are set to 0. The relationship categories with a context probability of 1 and the relationship categories with a context probability of 1 constitute the context knowledge of the object pair.

[0017] In one embodiment, the visual relationship detection module is used to perform visual prediction of the object pair based on the visual features and semantic features of the object pair, including:

[0018] The visual features and semantic features of the object pair are input into the visual relationship detection module, and the visual probability of the relationship category is calculated and output as the visual prediction result;

[0019] The comprehensive judgment module obtains the visual relationship of the object pair by comprehensive judgment based on the visual prediction result and the contextual knowledge of the object pair, including:

[0020] The visual prediction results are multiplied with the contextual knowledge of the object pairs according to the relationship category to obtain the prediction results of the relationship category guided by the contextual knowledge, and the visual relationship of the object pairs is determined based on the prediction results.

[0021] In one embodiment, the contextual knowledge generation module adopts a first fully connected network, the visual feature extraction module adopts a convolutional network, the semantic feature extraction module adopts a second fully connected network, and the visual relationship detection module adopts a third fully connected network.

[0022] In one embodiment, the visual relationship detection model needs to undergo parameter optimization before being applied. The total loss function used in the parameter optimization includes the activation loss of the marked relationship category in the label determined based on the contextual knowledge of the object pair, and the activation loss of the incorrect relationship category in the label determined based on the contextual knowledge of the object pair and the visual prediction results.

[0023] In one embodiment, the method also includes: increasing the position diversity of object pairs by random position perturbation, thereby obtaining diverse context vectors, local images and position masks, and using the diverse context vectors, local images and position masks to optimize the parameters of the visual relationship detection model.

[0024] To achieve the above-mentioned object of the invention, a second aspect of the embodiment provides a contextual knowledge guided visual relationship detection device, comprising a preprocessing unit and a visual relationship detection unit.

[0025] The preprocessing unit is used to obtain a scene image, determine a semantic vector and a context vector of an object pair in the scene image, extract a local image containing the object pair from the scene image, and determine a position mask of the object pair;

[0026] The visual relationship detection unit is used to perform visual relationship detection using a visual relationship detection model that includes a contextual knowledge generation module, a visual feature extraction module, a semantic feature extraction module, a visual relationship detection module, and a comprehensive judgment module, including: using the contextual knowledge generation module to extract the contextual knowledge of the object pair based on the context vector of the object pair, using the visual feature extraction module to extract the visual features of the object pair based on the local image and the position mask of the object pair, using the semantic feature extraction module to extract the semantic features of the object pair based on the semantic vector of the object pair, using the visual relationship detection module to calculate the visual prediction result of the object pair based on the visual features and semantic features of the object pair, and using the comprehensive judgment module to obtain the visual relationship of the object pair based on the visual prediction result and the contextual knowledge of the object pair.

[0027] To achieve the above-mentioned purpose of the invention, the third aspect of the embodiment provides an electronic device, including one or more processors and one or more program memories. When the one or more programs are executed by the one or more processors, the one or more processors implement the steps of the above-mentioned contextual knowledge-guided visual relationship detection method.

[0028] To achieve the above-mentioned purpose of the invention, a fourth aspect of the embodiment provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-mentioned contextual knowledge-guided visual relationship detection method.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] Based on obtaining the scene image and extracting the semantic vector, context vector and position code of the object pair, context knowledge is generated according to the context vector, and then the visual prediction result is calculated based on the semantic vector, position code and local image. The context knowledge is used as guiding information to guide the visual prediction result to comprehensively judge the visual relationship of the object pair, thereby improving the accuracy of visual relationship detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0032] Figure 1 is a flowchart of a contextual knowledge-guided visual relationship detection method provided in an embodiment;

[0033] Figure 2 is a flow chart for constructing a context vector for an object pair provided in an embodiment;

[0034] Figure 3 is a flow chart of constructing a position mask of an object pair provided by an embodiment;

[0035] Figure 4 This is a flowchart of performing visual relationship detection using a visual relationship detection model provided by an embodiment;

[0036] Figure 5 is a flow chart of random perturbation of object position provided by an embodiment;

[0037] Figure 6 3 is a schematic diagram of the structure of a contextual knowledge-guided visual relationship detection device provided in an embodiment. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0039] The study found that even without seeing the image, useful knowledge can be extracted from the context description to guide visual relationship detection. For humans, given the categories and relative positions of objects, the relationship between the objects can be roughly guessed. For example, humans know the categories and relative positions of two objects, a person and a bicycle. Humans can guess through existing knowledge that the relationship between a person and a bicycle can be "riding", "on...", "next to...", etc., but cannot be "wearing", "embedded in", etc. When humans actually see two objects, a person and a bicycle in the scene, they judge the relationship based on existing knowledge and visual observations. Based on this discovery, the embodiment provides a contextual knowledge-guided visual relationship detection method and device, which guides the judgment of visual relationships by learning contextual knowledge from object categories and relative positions, so as to solve the problems of low accuracy of visual relationship detection of existing objects and insufficient scene understanding ability of robots.

[0040] Figure 1 FIG. 1 is a flow chart of a method for detecting visual relationships guided by contextual knowledge provided in an embodiment. Figure 1 As shown, the visual relationship detection method provided in the embodiment includes the following steps:

[0041] S110 , acquiring a scene image, determining a semantic vector and a context vector of an object pair in the scene image, extracting a local image containing the object pair from the scene image, and determining a position mask of the object pair.

[0042] In the embodiment, the obtained scene image contains multiple objects. The semantic vectors and context vectors of the object pairs in the scene image are determined by analyzing and calculating the scene image, such as Figure 2 As shown, the following steps are included:

[0043] Detect the position and category of objects in the scene image, combine the coordinates of the corner points of the bounding box representing the position and the coordinates of the center point of the bounding box to form a position vector, and convert the category into a word vector as a semantic vector; concatenate the semantic vectors of the two objects to form the semantic vector of the object pair; concatenate the position vector and the semantic vector of the object to form a context vector, and then concatenate the context vectors of the two objects to form the context vector of the object pair. Of course, the position vectors of the two objects can also be concatenated to form the position vector of the object pair, and then the position vector and semantic vector of the object pair can be concatenated to form the context vector of the object pair.

[0044] The position of the object is represented by a bounding box (x1, y1, x2, y2). Based on the bounding box (x1, y1, x2, y2), the center point of the bounding box (x c =(x1+x2) / 2,y c =(y1+y2) / 2), concatenate the bounding box (x1, y1, x2, y2) and the center point x of the bounding box c That is, the position vector of the object can be obtained.

[0045] In an embodiment, it is also possible to extract a local image containing an object pair from a scene image and determine a position mask of the object pair, such as Figure 3 As shown, the following steps are included: intercepting a local image containing the object pair based on the bounding boxes of the two object positions, specifically, for the position bounding box of object 1 and the position bounding box of object 2 Crop from the scene image as a local image containing an object pair.

[0046] According to the position bounding boxes of object 1 and object 2, in the local image, the area within the bounding box of each object is set to 1 and the other areas are set to 0 to obtain the position mask of each object. The position masks of object 1 and object 2 constitute the position mask of the object pair.

[0047] The local images obtained by analyzing the scene images, the semantic vectors of object pairs, the context vectors and the position masks are all used as data sources for visual relationship detection.

[0048] S120 , performing visual relationship detection using a visual relationship detection model including a contextual knowledge generation module, a visual feature extraction module, a semantic feature extraction module, a visual relationship detection module, and a comprehensive judgment module.

[0049] In this embodiment, the contextual knowledge generation module, as the contextual knowledge generation stage, is used to generate contextual knowledge for the object pair based on the context vector of the object pair. Specifically, the context vector of the object pair is input into the contextual knowledge generation module, which calculates and outputs the contextual probabilities of all relationship categories. Based on a threshold, high context probabilities are filtered and set to 1, and all remaining context probabilities are set to 0. Relationship categories with a context probability of 1 and relationship categories with a context probability of 1 constitute the contextual knowledge of the object pair. In this embodiment, the contextual knowledge generation module can utilize a first fully connected network.

[0050] like Figure 4 As shown in , the context knowledge generation module calculates and outputs the context probabilities of all relationship categories such as has, wears, with, using, on, and flying in based on the context vector of the object pair. Then, based on the set threshold, for example, the threshold is 0.7, the context probabilities above the threshold are filtered and reset to 1, and the remaining context probabilities are set to 0, as shown in Figure 4 In the example, the context probabilities for the four relationship categories (possess, wear, and, and use) are 1, while the context probabilities for the remaining relationship categories (such as "above" and "floating") are 0. This means that the relationships "has," "wears," "with," and "using" are likely to exist, with high probability, while "on" and "flying in" are unlikely, with low probability. These relationship categories with a context probability of 1 and 0 together form the contextual knowledge for the object pair. This contextual knowledge serves as a knowledge mask for the relationship categories and guides visual relationship detection.

[0051] In the embodiment, the visual feature extraction module, the semantic feature extraction module, the visual relationship detection module and the comprehensive judgment module serve as the visual relationship detection stage and are used for visual relationship detection calculation. Figure 4 As shown, specifically including:

[0052] A visual feature extraction module is used to extract visual features of the object pair based on the local image and the position mask of the object pair. In one embodiment, the visual feature extraction module can utilize a convolutional neural network. The local image and the position mask of the object pair are concatenated and then input into the convolutional neural network, which calculates and outputs the visual features of the object pair.

[0053] The semantic feature extraction module extracts the semantic features of the object pair based on the semantic vectors of the object pair. In one embodiment, the semantic feature extraction module may utilize a second fully connected network. The semantic vectors of the object pair are input to the second fully connected network, which calculates and outputs the semantic features of the object pair.

[0054] The visual relationship detection module calculates a visual prediction result for the object pair based on the visual features and semantic features of the object pair. In one embodiment, the visual relationship detection module can utilize a third fully connected network. The visual features and semantic features of the object pair are input to the third fully connected network, which calculates and outputs the visual probability of the relationship category as the visual prediction result.

[0055] The comprehensive judgment module is used to make a comprehensive judgment based on the visual prediction results and the contextual knowledge of the object pairs to obtain the visual relationship of the object pairs guided by the contextual knowledge. Specifically, the visual prediction results are multiplied by the contextual knowledge of the object pairs according to the relationship category to obtain the prediction results of the relationship category guided by the contextual knowledge, and the visual relationship of the object pairs is determined based on the prediction results.

[0056] In this embodiment, contextual knowledge consists of relationship categories with a contextual probability of 1 and relationship categories with a contextual probability of 0. That is, each relationship category has a contextual probability of 1 or 0. The visual prediction result is a visual probability, and each relationship category has a visual probability in the range of 0-1. Multiplying the contextual probability and visual probability of each relationship category yields the final prediction result for each relationship category. The relationship category with the highest probability value is then extracted from the prediction results for all relationship categories as the final visual relationship for the object pair.

[0057] like Figure 4 As shown, the visual probabilities of the floating relationship types of has, wears, with, using, on, and flying in are 0.63, 0.89, 0.83, 0.50, 0.92, and 0.05. After multiplying these visual probabilities with the corresponding context probabilities, the probability values obtained are 0.63, 0.89, 0.83, 0.50, 0, and 0, respectively.

[0058] In the contextual knowledge of the object pair, the relationship category of on is considered to be a low-probability relationship with a value of 0, while in the visual prediction of the object pair, the visual probability of on is the highest; by multiplying the visual probability of the object pair with the contextual knowledge of the object pair, the relationship of on is eliminated. Finally, the relationship of the object pair is both a high-probability relationship with a value of 1 in the contextual knowledge and the relationship category with the highest probability in the visual prediction of the object pair, namely wears.

[0059] In the embodiment, the visual relationship detection model needs to be parameter optimized before being applied. The total loss function used in the parameter optimization includes the activation loss L of the relationship category in the label determined based on the contextual knowledge of the object pair. pos , the activation loss L for incorrect relation categories in labels determined based on the contextual knowledge of object pairs and visual prediction results neg , respectively expressed as:

[0060]

[0061]

[0062] Among them, ReLu(·) represents the ReLu activation function, R know Represents the contextual knowledge of the object pair, R vision Represents the visual prediction result of the object pair, r = r t The relationship category is the relationship category marked in the label, r≠r t The representative relationship category is the relationship category that is not marked in the label. During specific training, according to L pos +L neg Optimize parameters of visual relationship detection models.

[0063] In order to improve the prediction performance of the visual relationship detection model, a position random perturbation method is used to generate diverse context vectors during training, such as Figure 5 As shown, it includes: for the object position and category detected in the scene image, for the position bounding box (x1, y1, x2, y2), a random perturbation method is used to add a random position offset to the position bounding box to obtain a diversified position bounding box (x1+ε, y1+ε, x2+ε, y2+ε), ε is a small random number; using the diversified position bounding box to generate a perturbed position vector, using the perturbed position vector and the semantic vector, generating a diversified context vector, and at the same time generating a diversified local image and position mask based on the diversified position bounding box, and then using the diversified context vector, local image and position mask to optimize the parameters of the visual relationship detection model.

[0064] In view of the difficulty in obtaining training images for traditional visual relationship detection, the present invention can easily expand the context description for training through simple position perturbations at the context level, and then mine the contextual knowledge in the object's context description to guide the visual relationship detection process and obtain better visual relationship detection results.

[0065] Existing visual relationship detection methods often directly extract the visual features of objects from scene images, while ignoring the information of context description. In the embodiment of the present invention, the semantic vector and the position vector are spliced together to form a context vector as a context description. Without relying on vision, the context probability of each relationship category is predicted only by the context description. From the context probabilities, high-probability relationship categories are selected as context knowledge. When judging the relationship between object pairs through vision, the context knowledge is used as a guide, so that only the high-probability relationships considered by the context knowledge are paid attention to, and the low-probability relationships considered by the context knowledge are ignored, thereby improving the accuracy of judging the visual relationship between object pairs. The embodiment of the present invention is suitable for improving the service robot's ability to analyze and judge scenes during operation.

[0066] Based on the same inventive concept, the present invention also provides a contextual knowledge guided visual relationship detection device, such as Figure 6 As shown, the apparatus 600 includes a pre-processing unit 610 and a visual relationship detection unit 620.

[0067] The pre-processing unit 610 is used to obtain a scene image, determine the semantic vector and the context vector of the object pair in the scene image, extract the local image containing the object pair from the scene image and determine the position mask of the object pair;

[0068] The visual relationship detection unit 620 is used to perform visual relationship detection using a visual relationship detection model including a contextual knowledge generation module, a visual feature extraction module, a semantic feature extraction module, a visual relationship detection module, and a comprehensive judgment module.

[0069] Specifically, in the visual relationship detection unit 620, the visual relationship detection includes: utilizing the contextual knowledge generation module to extract the contextual knowledge of the object pair based on the contextual vector of the object pair, utilizing the visual feature extraction module to extract the visual features of the object pair based on the local image and the position mask of the object pair, utilizing the semantic feature extraction module to extract the semantic features of the object pair based on the semantic vector of the object pair, utilizing the visual relationship detection module to calculate the visual prediction result of the object pair based on the visual features and semantic features of the object pair, and utilizing the comprehensive judgment module to obtain the visual relationship of the object pair based on the visual prediction result and the contextual knowledge of the object pair.

[0070] It should be noted that the contextual knowledge-guided visual relationship detection device provided in the above embodiment should be illustrated by the division of the above-mentioned functional units when performing visual relationship detection. The above-mentioned functions can be assigned to different functional units as needed, that is, the internal structure of the terminal or server is divided into different functional units to complete all or part of the functions described above. In addition, the contextual knowledge-guided visual relationship detection device provided in the above embodiment and the contextual knowledge-guided visual relationship detection method embodiment belong to the same concept. The specific implementation process is detailed in the contextual knowledge-guided visual relationship detection method embodiment, which will not be repeated here.

[0071] Based on the same inventive concept, an embodiment also provides an electronic device, including one or more processors and one or more program memories. When the one or more programs are executed by the one or more processors, the one or more processors implement S110 and S120 of the above-mentioned contextual knowledge-guided visual relationship detection method.

[0072] Based on the same inventive concept, an embodiment further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements S110 and S120 of the above-mentioned contextual knowledge-guided visual relationship detection method.

[0073] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0074] In the above embodiments of the present invention, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0075] In the embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0076] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0077] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0078] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk, etc. Various media that can store program codes.

[0079] The specific implementation methods described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A contextual knowledge-guided visual relationship detection method, characterized in that: The following steps are involved: Acquiring a scene image, and determining semantic vectors and context vectors of object pairs in the scene image, including: detecting positions and categories of objects in the scene image, forming a position vector from the coordinates of the corner points of a bounding box representing the position and the coordinates of the center point of the bounding box, and converting the category into a word vector as a semantic vector; concatenating the semantic vectors of two objects to form a semantic vector of the object pair; concatenating the position vectors and semantic vectors of the objects to form a context vector, and then concatenating the context vectors of the two objects to form a context vector of the object pair, or concatenating the position vectors of the two objects to form a position vector of the object pair, and then concatenating the position vector and semantic vector of the object pair to form a context vector of the object pair; Extracting a local image containing an object pair from a scene image and determining a position mask of the object pair; Visual relationship detection is performed using a visual relationship detection model that includes a contextual knowledge generation module, a visual feature extraction module, a semantic feature extraction module, a visual relationship detection module, and a comprehensive judgment module, including: using the contextual knowledge generation module to generate contextual knowledge of the object pair based on the context vector of the object pair, using the visual feature extraction module to extract the visual features of the object pair based on the local image and the position mask of the object pair, using the semantic feature extraction module to extract the semantic features of the object pair based on the semantic vector of the object pair, using the visual relationship detection module to calculate the visual prediction results of the object pair based on the visual features and semantic features of the object pair, and using the comprehensive judgment module to comprehensively judge the visual prediction results and the contextual knowledge of the object pair to obtain the visual relationship of the object pair guided by the contextual knowledge.

2. The context-based knowledge-guided visual relationship detection method according to claim 1, characterized in that: The extracting of a local image containing an object pair from a scene image and determining a position mask of the object pair includes: Detect the position of objects in the scene image and intercept the local image containing the object pair based on the bounding box of the two object positions; In the local image, the area within the bounding box of each object is set to 1 and the other areas are set to 0 to obtain the position mask of each object. The position masks of two objects constitute the position mask of the object pair.

3. The context-based knowledge-guided visual relationship detection method according to claim 1, characterized in that: The method of utilizing the context knowledge generation module to extract the context knowledge of the object pair according to the context vector of the object pair includes: The context vector of the object pair is input into the context knowledge generation module. After calculation, the context probabilities of all relationship categories are output. According to the threshold, high context probabilities are screened and set to 1, and all remaining context probabilities are set to 0. The relationship categories with a context probability of 1 and the relationship categories with a context probability of 1 constitute the context knowledge of the object pair.

4. The context-based knowledge-guided visual relationship detection method according to claim 1, characterized in that: The method of using the visual relationship detection module to perform visual prediction of the object pair based on the visual features and semantic features of the object pair includes: The visual features and semantic features of the object pair are input into the visual relationship detection module, and the visual probability of the relationship category is calculated and output as the visual prediction result; The comprehensive judgment module obtains the visual relationship of the object pair by comprehensive judgment based on the visual prediction result and the contextual knowledge of the object pair, including: The visual prediction results are multiplied with the contextual knowledge of the object pairs according to the relationship category to obtain the prediction results of the relationship category guided by the contextual knowledge, and the visual relationship of the object pairs is determined based on the prediction results.

5. The context-based knowledge-guided visual relationship detection method according to claim 1, characterized in that: The contextual knowledge generation module adopts a first fully connected network, the visual feature extraction module adopts a convolutional network, the semantic feature extraction module adopts a second fully connected network, and the visual relationship detection module adopts a third fully connected network.

6. The context-based knowledge-guided visual relationship detection method according to claim 1, characterized in that: The visual relationship detection model needs to undergo parameter optimization before being applied. The total loss function used in parameter optimization includes the activation loss of the marked relationship category in the label determined based on the contextual knowledge of the object pair, and the activation loss of the incorrect relationship category in the label determined based on the contextual knowledge of the object pair and the visual prediction results.

7. The context-based knowledge-guided visual relationship detection method according to claim 6, characterized in that: Also includes: By randomly perturbing the positions, the position diversity of object pairs is increased, and then diverse context vectors, local images and position masks are obtained. The diverse context vectors, local images and position masks are used to optimize the parameters of the visual relationship detection model.

8. A contextual knowledge-guided visual relationship detection device, characterized in that: Including pre-processing unit and visual relationship detection unit, The preprocessing unit is used to obtain a scene image and determine the semantic vector and context vector of an object pair in the scene image, including: detecting the position and category of the object in the scene image, forming a position vector with the coordinates of the corner points of the bounding box representing the position and the coordinates of the center point of the bounding box, and converting the category into a word vector as a semantic vector; splicing the semantic vectors of two objects to form a semantic vector of the object pair; splicing the position vector and the semantic vector of the object to form a context vector, and then splicing the context vectors of the two objects to form a context vector of the object pair, or splicing the position vectors of the two objects to form the position vector of the object pair, and then splicing the position vector and the semantic vector of the object pair to form the context vector of the object pair; It is also used to extract a local image containing an object pair from a scene image and determine a position mask of the object pair; The visual relationship detection unit is used to perform visual relationship detection using a visual relationship detection model that includes a contextual knowledge generation module, a visual feature extraction module, a semantic feature extraction module, a visual relationship detection module, and a comprehensive judgment module, including: using the contextual knowledge generation module to extract the contextual knowledge of the object pair based on the context vector of the object pair, using the visual feature extraction module to extract the visual features of the object pair based on the local image and the position mask of the object pair, using the semantic feature extraction module to extract the semantic features of the object pair based on the semantic vector of the object pair, using the visual relationship detection module to calculate the visual prediction result of the object pair based on the visual features and semantic features of the object pair, and using the comprehensive judgment module to obtain the visual relationship of the object pair based on the visual prediction result and the contextual knowledge of the object pair.

9. An electronic device comprising one or more processors and one or more program memories, characterized in that: When the one or more programs are executed by the one or more processors, the one or more processors implement the steps of the contextual knowledge-guided visual relationship detection method described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the contextual knowledge-guided visual relationship detection method as described in any one of claims 1 to 7 are implemented.

Images