Image processing method, device, electronic device and storage medium

Abstract

The present disclosure relates to an image processing method, device, electronic device and storage medium, the method comprising: inputting an image to be processed into an image detection model for object detection, obtaining object detection information corresponding to at least two objects in the image to be processed, inputting the object detection information into a visual relationship detection model for visual relationship detection, obtaining a visual relationship between two objects, the visual relationship being obtained after the visual relationship detection model adjusts the amount of semantic information corresponding to the visual relationship, inputting the visual relationship into a scene graph generation model for scene graph generation, obtaining a target scene graph corresponding to the image to be processed. The method detects the visual relationship between two objects based on the visual relationship detection model, and can improve the accuracy of visual relationship detection.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to an image processing method, device, electronic device and storage medium. Background Art

[0002] Through visual relationship detection, a scene graph annotated with visual relationships can be generated. As a structured representation of image content, the scene graph annotated with visual relationships is a bridge between computer vision and natural language. After generating a scene graph annotated with visual relationships, the visual relationship triples consisting of the subject, predicate, and object in the image can be detected based on the scene image.

[0003] In the related art, when performing visual relationship detection on the image to be processed, the detected visual relationship is easily confused, thereby reducing the accuracy of visual relationship detection and the effectiveness of the scene graph annotated with the visual relationship. Summary of the invention

[0004] The present disclosure provides an image processing method, device, electronic device and storage medium to at least solve the problem of low accuracy of visual relationship detection and low effectiveness of scene images annotated with visual relationships in the related art. The technical solution of the present disclosure is as follows:

[0005] According to a first aspect of an embodiment of the present disclosure, there is provided an image processing method, the method comprising:

[0006] Inputting the image to be processed into the image detection model to perform object detection, and obtaining object detection information corresponding to at least two objects in the image to be processed;

[0007] Inputting the object detection information into a visual relationship detection model to perform visual relationship detection to obtain a visual relationship between two objects, wherein the visual relationship represents an interactive relationship between two objects in the image to be processed;

[0008] The visual relationship and the object detection information corresponding to the visual relationship are input into a scene graph generation model to generate a scene graph, and a target scene graph corresponding to the image to be processed is obtained, wherein the target scene graph is structural information annotated with the visual relationship between the two objects.

[0009] As an optional embodiment, the visual relationship detection model includes a predicate recognition network, and the step of inputting the object detection information into the visual relationship detection model to perform visual relationship detection to obtain the visual relationship between two objects includes:

[0010] Inputting the object detection information into the predicate recognition network to perform predicate recognition corresponding to the predicate relationship between two objects, and obtaining a target predicate, wherein the target predicate represents the semantically adjusted predicate;

[0011] The visual relationship is obtained according to the target predicate and the object corresponding to the target predicate.

[0012] As an optional embodiment, the predicate recognition network includes an initial relevance calculation layer and a semantic adjustment layer, and the object detection information is input into the predicate recognition network to perform predicate recognition between two objects, and the target predicate is obtained, including:

[0013] Inputting the object detection information and the preset predicates into the initial relevance calculation layer, performing relevance calculation on the predicates corresponding to the pairwise object detection information and each preset predicate, and obtaining initial relevance distribution information, wherein the initial relevance distribution information represents the relevance between the predicates corresponding to the pairwise object detection information and each preset predicate before semantic adjustment;

[0014] Inputting the initial relevance distribution information into a semantic adjustment layer, performing predicate semantic adjustment on the initial relevance distribution information based on the preset matrix to obtain the target relevance distribution information, wherein the target relevance distribution information represents the relevance between the predicate corresponding to the pairwise object detection information and each preset predicate after the semantic adjustment;

[0015] The target predicate is determined according to the target relevance distribution information.

[0016] As an optional embodiment, the inputting the initial relevance distribution information into the semantic adjustment layer, performing predicate semantic adjustment on the initial relevance distribution information based on the preset matrix, and obtaining the target relevance distribution information includes:

[0017] Determining an initial predicate according to the initial relevance distribution information;

[0018] In the case where the initial predicate is a universal predicate, based on the semantic adjustment matrix in the preset matrix, the initial relevance distribution information is subjected to predicate semantic adjustment, wherein the universal predicate represents a predicate in the preset predicate whose use probability is greater than a preset threshold;

[0019] In the case where the initial predicate is a non-universal predicate, the initial relevance distribution information is determined as the target relevance distribution information based on a semantic preservation matrix in the preset matrix, and the non-universal predicate represents a predicate in the preset predicate whose usage probability is less than a preset threshold.

[0020] As an optional embodiment, the method further includes:

[0021] Inputting the annotated image into the image detection model to perform object detection, and obtaining training object detection information corresponding to each object in the annotated image, wherein the annotated image is annotated with reference visual relationships between the two objects;

[0022] Inputting the training object detection information into a first model to be trained to perform visual relationship detection, thereby obtaining a first training visual relationship between the two objects, wherein the first training visual relationship represents an interactive relationship between the two objects in the annotated image obtained by the first model to be trained;

[0023] Inputting the first training visual relationship and the training object detection information corresponding to the first training visual relationship into the second to-be-trained model to generate a scene graph, thereby obtaining a first training scene graph corresponding to the annotated image, wherein the first training scene graph is structural information annotated with the first training visual relationship between the two objects;

[0024] The first model to be trained and the second model to be trained are trained according to the first training visual relationship and the reference visual relationship to obtain a first visual relationship detection model and an initial scene graph generation model.

[0025] As an optional embodiment, after training the first model to be trained and the second model to be trained according to the training visual relationship and the reference visual relationship to obtain the first visual relationship detection model and the initial scene graph generation model, the method further includes:

[0026] detecting word frequency information corresponding to each reference predicate in the reference visual relationship;

[0027] Classifying the reference predicates according to preset word frequency segmentation information and the word frequency information to obtain a reference predicate type corresponding to each annotated image;

[0028] Combining the first visual relationship detection model and a preset matrix to obtain a second visual relationship detection model;

[0029] Inputting the training object detection information into the second visual relationship detection model to perform visual relationship detection to obtain a second training visual relationship between the two objects, wherein the second training visual relationship represents an interactive relationship between the two objects in the annotated image in the presence of a preset matrix;

[0030] Inputting the training object detection information corresponding to the second training visual relationship and the first training visual relationship into the initial scene graph generation model to generate a scene graph, thereby obtaining a second training scene graph corresponding to the annotated image, wherein the second training scene graph is annotated with structural information of the second training visual relationship between the two objects;

[0031] Based on the reference predicate type corresponding to each annotated image, the second training visual relationship and the reference visual relationship, the second visual relationship detection model and the initial scene graph generation model are adjusted to obtain the visual relationship detection model and the scene graph generation model.

[0032] As an optional embodiment, the method includes:

[0033] Inputting the training object detection information into the first visual relationship detection model to perform visual relationship detection to obtain an initial visual relationship between the two objects, wherein the initial visual relationship represents an interactive relationship between the two objects in the annotated image obtained by the first visual relationship detection model;

[0034] Inputting the initial visual relationship and the training object detection information corresponding to the first training visual relationship into the initial scene graph generation model to generate a scene graph, thereby obtaining an initial scene graph corresponding to the annotated image, wherein the initial scene graph is annotated with structural information of the initial visual relationship between the two objects;

[0035] determining an initial matrix according to the predicate in the initial visual relationship and the reference predicate in the reference visual relationship;

[0036] A preset matrix is ​​obtained according to a normalized matrix corresponding to the initial matrix and a unit matrix corresponding to the initial matrix.

[0037] According to a second aspect of an embodiment of the present disclosure, there is provided an image processing apparatus, the apparatus comprising:

[0038] An object detection module is configured to input the image to be processed into an image detection model to perform object detection, and obtain object detection information corresponding to at least two objects in the image to be processed;

[0039] A visual relationship detection module is configured to input the object detection information into a visual relationship detection model to perform visual relationship detection to obtain a visual relationship between two objects, wherein the visual relationship represents an interactive relationship between two objects in the image to be processed;

[0040] The scene graph generation module is configured to execute scene graph generation by inputting the visual relationship and the object detection information corresponding to the visual relationship into a scene graph generation model to obtain a target scene graph corresponding to the image to be processed, wherein the target scene graph is structural information annotated with the visual relationship between the two objects.

[0041] As an optional embodiment, the visual relationship detection model includes a predicate recognition network and a visual relationship determination network, and the visual relationship detection module includes:

[0042] A predicate recognition unit is configured to input the object detection information into the predicate recognition network to perform predicate recognition corresponding to the predicate relationship between two objects, and obtain a target predicate, wherein the target predicate represents a semantically adjusted predicate;

[0043] The visual relationship determining unit is configured to obtain the visual relationship according to the target predicate and the object corresponding to the target predicate.

[0044] As an optional embodiment, the predicate recognition network includes an initial relevance calculation layer and a semantic adjustment layer, and the predicate recognition unit includes:

[0045] an initial relevance calculation unit, configured to input the object detection information and the preset predicate into the initial relevance calculation layer, perform relevance calculation on the predicates corresponding to the pairwise object detection information and each preset predicate, and obtain initial relevance distribution information, wherein the initial relevance distribution information represents the relevance between the predicates corresponding to the pairwise object detection information and each preset predicate before semantic adjustment;

[0046] A semantic adjustment unit is configured to input the initial relevance distribution information into a semantic adjustment layer, perform predicate semantic adjustment on the initial relevance distribution information based on the preset matrix, and obtain the target relevance distribution information, wherein the target relevance distribution information represents the relevance between the predicate corresponding to the pairwise object detection information and each preset predicate after the semantic adjustment;

[0047] The target predicate determination unit is configured to determine the target predicate according to the target relevance distribution information.

[0048] As an optional embodiment, the semantic adjustment unit includes:

[0049] an initial predicate determination unit, configured to determine an initial predicate according to the initial relevance distribution information;

[0050] A first semantic adjustment unit is configured to perform predicate semantic adjustment on the initial relevance distribution information based on a semantic adjustment matrix in the preset matrix when the initial predicate is a universal predicate, wherein the universal predicate represents a predicate in the preset predicate whose use probability is greater than a preset threshold;

[0051] The second semantic adjustment unit is configured to determine the initial relevance distribution information as the target relevance distribution information based on a semantic preservation matrix in the preset matrix when the initial predicate is a non-universal predicate, wherein the non-universal predicate represents a predicate in the preset predicate whose usage probability is less than a preset threshold.

[0052] As an optional embodiment, the device further includes:

[0053] A first training feature extraction module is configured to perform feature extraction by inputting the annotated image into the image detection model to obtain training object detection information corresponding to each object in the annotated image, wherein the annotated image is annotated with reference visual relationships between the two objects;

[0054] A first training visual relationship detection module is configured to input the training object detection information into a first model to be trained to perform visual relationship detection to obtain a first training visual relationship between the two objects, wherein the first training visual relationship represents an interactive relationship between the two objects in the annotated image obtained by the first model to be trained;

[0055] A first training scene graph generation module is configured to execute the step of inputting the first training visual relationship and the training object detection information corresponding to the first training visual relationship into a second to-be-trained model to generate a scene graph, thereby obtaining a first training scene graph corresponding to the annotated image, wherein the first training scene graph is structural information annotated with the first training visual relationship between the two objects;

[0056] The model training module is configured to train the first model to be trained and the second model to be trained according to the training visual relationship and the reference visual relationship to obtain a first visual relationship detection model and an initial scene graph generation model.

[0057] As an optional embodiment, the device further includes:

[0058] A word frequency information detection module is configured to detect word frequency information corresponding to each reference predicate in the reference visual relationship;

[0059] A second visual relationship detection model acquisition module is configured to combine the first visual relationship detection model with a preset matrix to obtain a second visual relationship detection model;

[0060] A second training visual relationship acquisition module is configured to input the training object detection information into the second visual relationship detection model to perform visual relationship detection to obtain a second training visual relationship between the two objects, wherein the second training visual relationship represents an interactive relationship between the two objects in the annotated image obtained by the second visual relationship detection model;

[0061] A second training scene graph acquisition module is configured to execute inputting the training object detection information corresponding to the second training visual relationship and the first training visual relationship into the initial scene graph generation model to generate a scene graph, so as to obtain a second training scene graph corresponding to the annotated image, wherein the second training scene graph is structural information annotated with the second training visual relationship between the two objects;

[0062] The model adjustment module is configured to adjust the second visual relationship detection model and the initial scene graph generation model based on the reference predicate type corresponding to each annotated image, the second training visual relationship and the reference visual relationship to obtain the visual relationship detection model and the scene graph generation model.

[0063] As an optional embodiment, the device further includes:

[0064] an initial visual relationship detection module, configured to input the object detection information into the first visual relationship detection model to perform visual relationship detection, and obtain an initial visual relationship between the two objects, wherein the initial visual relationship represents an interactive relationship between the two objects in the annotated image obtained by the first visual relationship detection model;

[0065] a scene initial graph generation module, configured to execute inputting the training object detection information corresponding to the initial visual relationship and the first training visual relationship into the initial scene graph generation model to generate a scene graph, and obtain an initial scene graph corresponding to the annotated image, wherein the initial scene graph is annotated with structural information of the initial visual relationship between the two objects;

[0066] an initial matrix determination module, configured to determine an initial matrix according to the predicate in the initial visual relationship and the reference predicate in the reference visual relationship;

[0067] The preset matrix determination module is configured to execute a normalized matrix corresponding to the initial matrix and a unit matrix corresponding to the initial matrix to obtain a preset matrix.

[0068] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, the electronic device comprising:

[0069] processor;

[0070] a memory for storing instructions executable by the processor;

[0071] Wherein, the processor is configured to execute the instructions to implement the above-mentioned image processing method.

[0072] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the image processing method as described above.

[0073] According to a fifth aspect of an embodiment of the present disclosure, there is provided a computer program product, comprising computer instructions, which implement the above-mentioned image processing method when executed by a processor.

[0074] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0075] The image to be processed is input into the image detection model for object detection, and object detection information corresponding to at least two objects in the image to be processed is obtained. The object detection information is input into the visual relationship detection model for visual relationship detection, and the visual relationship between two objects is obtained after the visual relationship detection model adjusts the amount of semantic information corresponding to the visual relationship. The visual relationship is input into the scene graph generation model for scene graph generation, and the target scene graph corresponding to the image to be processed is obtained. Based on the visual relationship detection model, the method detects the visual relationship between two objects, which can improve the accuracy of visual relationship detection.

[0076] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0078] Figure 1 The figure is a schematic diagram of an application scenario of an image processing method according to an exemplary embodiment.

[0079] Figure 2 The figure is a flowchart of an image processing method according to an exemplary embodiment.

[0080] Figure 3 The present invention is a flowchart of performing predicate recognition in an image processing method according to an exemplary embodiment.

[0081] Figure 4 The present invention is a flowchart of performing predicate semantic adjustment in an image processing method according to an exemplary embodiment.

[0082] Figure 5 The present invention is a flowchart of performing model training in an image processing method according to an exemplary embodiment.

[0083] Figure 6 The present invention is a flowchart of adjusting a trained model in an image processing method according to an exemplary embodiment.

[0084] Figure 7 It is a schematic diagram of performing transfer learning in an image processing method according to an exemplary embodiment.

[0085] Figure 8 It is a schematic diagram of fixing parameters during model training in an image processing method according to an exemplary embodiment.

[0086] Fig. 9 It is a schematic diagram of a process of inputting a picture to be processed and generating a target scene graph in an image processing method according to an exemplary embodiment.

[0087] Fig.10 is a block diagram of an image processing apparatus according to an exemplary embodiment.

[0088] Fig.11 It is a block diagram of an electronic device on a server side according to an exemplary embodiment. DETAILED DESCRIPTION

[0089] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0090] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0091] Figure 1 is a flowchart of an image processing method according to an exemplary embodiment. Figure 1As shown, the application scenario includes a client 110 and a server 120. The client 110 collects the image to be processed, and the server 120 receives the image to be processed sent from the client 110. The server 120 inputs the image to be processed into the image detection model for object detection, and obtains object detection information corresponding to at least two objects in the image to be processed. The server 120 inputs the object detection information into the visual relationship detection model for visual relationship detection to obtain the visual relationship between the two objects, and inputs the visual relationship and the object detection information corresponding to the visual relationship into the scene graph generation model for image processing to obtain the target scene graph corresponding to the image to be processed. The server 120 sends the target scene graph to the client 110 for display.

[0092] In the disclosed embodiment, the client 110 includes physical devices such as smart phones, desktop computers, tablet computers, laptops, digital assistants, smart wearable devices, etc., and may also include software running in the physical devices, such as applications, etc. The operating system running on the physical device in the embodiment of the present application may include but is not limited to Android system, IOS system, Linux, Unix, Windows, etc. The client 110 includes a UI (User Interface) layer, and the client 110 provides the display of the target scene graph and the collection of the image to be processed to the outside through the UI layer. In addition, the image to be processed is sent to the server 120 based on the API (Application Programming Interface).

[0093] In the disclosed embodiment, the server 120 may include an independently operated server, or a distributed server, or a server cluster composed of multiple servers. The server 120 may include a network communication unit, a processor, a memory, etc. Specifically, the server 120 may be used to publish the published content through a publishing channel, and mark the channel tracking code of the publishing channel through a tracking code application interface, identification parameters or a fixed tracking code method, and receive log data fed back by a user terminal.

[0094] In the disclosed embodiment, the server 120 may perform visual relationship detection on the object detection information based on the technology of visual relationship detection. Visual relationship detection combines images with semantics. It not only needs to identify the objects in the image and the positions of the objects, but also the relationships between the objects. Visual relationship is defined as a pair of objects connected by a predicate, usually expressed in the form of subject, predicate and object, which can be used to describe the interactive relationship between two objects. Visual relationship detection is the basis of image understanding and can be applied to object detection, image description, visual question answering, image retrieval, etc.

[0095] Figure 2is a flowchart of an image processing method according to an exemplary embodiment. Figure 2 As shown, the method is used in a server and includes the following steps.

[0096] S210. Inputting the image to be processed into the image detection model to perform object detection, and obtaining object detection information corresponding to at least two objects in the image to be processed;

[0097] As an optional embodiment, in the image detection model, each object in the image to be processed is detected according to a preset annotation box, and the detection area of ​​each object in the image to be processed is extracted. The feature information corresponding to each object is extracted in the detection area corresponding to the object, and the object in the image to be processed is determined according to the feature information corresponding to each object, thereby obtaining object detection information. The image detection model can be a different image detection model such as the Faster R-CNN model, the Fast R-CNN model, the R-CNN model, etc. When the object detection information is input into the visual relationship detection model for visual relationship detection, the visual relationship corresponding to the object detection information can be detected.

[0098] As an optional embodiment, combined detection information can also be obtained based on the image detection model. The image detection model can detect two objects in the image to be processed, extract the detection area of ​​the two objects in the image to be processed, and extract the joint feature information corresponding to the two objects from the detection area corresponding to the two objects. According to the joint feature information, the two objects in the image to be processed are determined, thereby obtaining the combined detection information. The combined detection information includes detection information of two objects with an interactive relationship in the image to be processed. When the combined detection information is input into the visual relationship detection model for visual relationship detection, the visual relationship corresponding to the combined detection information can be detected.

[0099] The combined detection information can be used to exclude the combination of two objects that do not have an interactive relationship, thereby reducing the amount of data to be detected and improving the efficiency of visual relationship detection in subsequent steps.

[0100] S220. Inputting the object detection information into the visual relationship detection model to perform visual relationship detection, and obtaining the visual relationship between two objects, wherein the visual relationship represents the interaction relationship between two objects in the image to be processed;

[0101] As an optional embodiment, the interaction relationship between two objects may include an action relationship, a spatial relationship, a prepositional relationship and a comparative relationship. An action relationship can be expressed as an object performing a certain action on another object, such as a person riding a bicycle, and a spatial relationship can be expressed as the relative position between two objects, such as a cup on the left side of a book. A prepositional relationship can be expressed as the association between two objects in information such as subordination, state, and direction, such as a car has tires. A comparative relationship can represent the difference between two objects, such as the first apple is larger than the second apple. The visual relationship detection model can detect the visual relationship between objects corresponding to the two-object detection information and perform semantic adjustment on the visual relationship. The visual relationship between two objects can correspond to a triple consisting of two objects as subjects and objects, and predicates between the subjects and objects. When the visual relationship detection model performs semantic adjustment on the visual relationship, it can adjust the information content of the predicate between the subject and the object to obtain a predicate with rich meaning.

[0102] As an optional embodiment, the visual relationship detection model includes a predicate recognition network, and the object detection information is input into the visual relationship detection model to perform visual relationship detection, and the visual relationship between two objects is obtained, including:

[0103] The object detection information is input into the predicate recognition network to perform predicate recognition corresponding to the predicate relationship between two objects, and the target predicate is obtained;

[0104] According to the target predicate and the object corresponding to the target predicate, a visual relationship is obtained.

[0105] As an optional embodiment, the visual relationship detection model includes a predicate recognition network, in which the predicates corresponding to the predicate relationship between two objects are recognized, and the recognized target predicates are semantically adjusted predicates, and the two objects and the corresponding target predicates are formed into a visual relationship with a subject, a predicate and an object.

[0106] There is a predicate relationship between the objects corresponding to the pairwise object detection information. The predicates corresponding to the predicate relationship are identified in the predicate recognition network, and semantic adjustments can be made to obtain the target predicate. There are two objects corresponding to the target predicate, one can be used as the subject and the other can be used as the object, so the visual relationship with the subject, predicate and object can be determined. For example, when the object "person" is the subject, "have" is the target predicate and can be used as the predicate, and the object "hand" is the object, the obtained visual relationship can be (person, have, hand).

[0107] The target predicate between two objects is determined through the predicate recognition network. Since the predicate recognition network includes a semantic adjustment layer, the accuracy of predicate recognition can be improved.

[0108] As an alternative embodiment, see Figure 3 The predicate recognition network includes an initial relevance calculation layer and a semantic adjustment layer. The object detection information is input into the predicate recognition network to perform predicate recognition between two objects. The target predicates include:

[0109] S310. Input the object detection information and the preset predicates into the initial relevance calculation layer, perform relevance calculation on the predicates corresponding to the pairwise object detection information and each preset predicate, and obtain initial relevance distribution information;

[0110] S320. Inputting the initial relevance distribution information into the semantic adjustment layer, performing predicate semantic adjustment on the initial relevance distribution information based on a preset matrix, and obtaining target relevance distribution information;

[0111] S330. Determine the target predicate according to the target relevance distribution information.

[0112] As an optional embodiment, the predicate recognition network includes an initial relevance calculation layer and a semantic adjustment layer, the initial relevance calculation layer can be used to calculate initial relevance distribution information, and the semantic adjustment layer can be used to semantically adjust the initial relevance distribution information to obtain target relevance distribution information. The preset predicate includes multiple predicates, and the initial relevance distribution information is the probability distribution of the predicate corresponding to the pairwise object detection information before the semantic adjustment, which can be determined as a certain preset predicate, and represents the relevance between the predicate corresponding to the pairwise object detection information before the semantic adjustment and each preset predicate.

[0113] When the predicate semantic adjustment is performed on the initial relevance distribution information based on the preset matrix to obtain the target relevance distribution information, the predicate corresponding to the initial relevance distribution information before the semantic adjustment and the predicate corresponding to the target relevance distribution information after the semantic adjustment have semantic relevance. For example, the predicate corresponding to the initial relevance distribution information is "above", and the predicate corresponding to the target relevance distribution information is "ride", where "ride" also means "above". If there is semantic relevance between the two predicates, it can be determined that this semantic adjustment is correct. If the predicate corresponding to the initial relevance distribution information is "above", the predicate corresponding to the target relevance distribution information cannot be adjusted to "below", because "above" and "below" have completely opposite semantics, and there is no semantic relevance between the two predicates.

[0114] The target relevance distribution information represents the relevance between the predicates corresponding to the pairwise object detection information after semantic adjustment and each preset predicate. According to the magnitude of each relevance in the target relevance distribution information, the maximum relevance in the target relevance distribution information can be determined, and the preset predicate corresponding to the maximum relevance is determined as the target predicate.

[0115] As an optional embodiment, the calculation formula for identifying the predicate corresponding to the predicate relationship between the two object detection information can be expressed as:

[0116]

[0117] in, The probability distribution of the predicate before semantic adjustment can be expressed by the formula, and the initial relevance distribution information can be calculated. R represents the preset predicate, K represents the number of preset predicates, and the preset predicates include different types of predicates. i Represents a predicate. The predicate with a g superscript represents the output before semantic adjustment, and the predicate with an s superscript represents the output after semantic adjustment. j ,o k ) represents an object detection information pair, and θ represents a model parameter in a visual relationship detection model. The correlation between the predicate corresponding to the object detection information pair and each preset predicate is determined by the probability distribution of the predicate before semantic adjustment.

[0118] The probability distribution of the predicate after semantic adjustment can be expressed. The target relevance distribution information can be calculated through the formula. The probability distribution of the predicate before the semantic adjustment is adjusted based on the semantic adjustment matrix to obtain the probability distribution of the predicate after the semantic adjustment. The correlation between the predicate corresponding to the object detection information pair after the semantic adjustment and each preset predicate is determined, and the preset predicate corresponding to the maximum value in the correlation is determined as the predicate corresponding to the object detection information pair.

[0119] It represents the probability distribution of semantic adjustment, which can measure the confidence of converting a predicate without rich semantic information into a predicate with rich semantic information. The superscript s represents the output after semantic adjustment, and the superscript g represents the output before semantic adjustment.

[0120] A preset matrix may be used instead, and the preset matrix can perform semantic adjustment on the initial relevance distribution information in the semantic adjustment layer.

[0121] By performing predicate semantic adjustment on the initial relevance distribution information through the semantic adjustment layer, a target predicate with rich semantic information can be obtained, thereby improving the accuracy of predicate recognition and the effectiveness of the target scene graph generated in subsequent steps.

[0122] As an alternative embodiment, see Figure 4 , input the initial relevance distribution information into the semantic adjustment layer, perform predicate semantic adjustment on the initial relevance distribution information based on the preset matrix, and obtain the target relevance distribution information including:

[0123] S410. Determine an initial predicate according to the initial relevance distribution information;

[0124] S420. In the case where the initial predicate is a universal predicate, the initial relevance distribution information is semantically adjusted based on the semantic adjustment matrix in the preset matrix, and the universal predicate represents the predicate in the preset predicate whose usage probability is greater than a preset threshold;

[0125] S430. In the case where the initial predicate is a non-universal predicate, based on a semantic preservation matrix in a preset matrix, the initial relevance distribution information is determined as the target relevance distribution information, and the non-universal predicate represents a predicate in the preset predicate whose usage probability is less than a preset threshold.

[0126] As an optional embodiment, based on Shannon's semantic information theory, the semantic information contained in a predicate can be determined by the probability of occurrence of a predicate, and a predicate with a small probability of occurrence contains more semantic information. The semantic information can be used to determine whether a predicate is a general predicate or a non-general predicate. If the predicate is a general predicate, the semantic information contained in the general predicate is low, and the probability of using the general predicate in the preset predicate is greater than the preset threshold. If the predicate is a non-general predicate, the semantic information contained in the non-general predicate is high, and the probability of using the non-general predicate in the preset predicate is less than the preset threshold. For example, a person is on a bicycle, where "on top" can describe the relationship between the person and the bicycle, but the "riding" in "a person riding a bicycle" indicates the action of the person on the bicycle, so "riding" has more semantic information than "on top". "On top of" only indicates the relative relationship between the positions of two objects, while "ride" indicates an action. "On top of" can be applied wherever "ride" can be applied, such as "a person riding a horse" and "a person on a horse". However, "ride" may not be applied when applying "on top of" such as "the book is on the table". Therefore, the probability of "ride" appearing is significantly lower than that of "on top of" and so predicates with lower probability of appearance have more semantic information.

[0127] When the initial relevance distribution information is input into the semantic adjustment matrix for predicate semantic adjustment, if the initial predicate corresponding to the initial relevance distribution information is a general predicate, it means that the initial predicate contains less semantic information. Through the semantic adjustment matrix, the relevance distribution in the initial relevance distribution information can be adjusted to obtain the target relevance distribution information, thereby associating the predicates corresponding to the pairwise object detection information with the preset predicates containing more semantic information to obtain the target predicate.

[0128] When the initial predicate corresponding to the initial relevance distribution information is a non-universal predicate, it means that the initial predicate contains a large amount of semantic information. Through the semantic preservation matrix, the relevance distribution in the initial relevance distribution information can be kept unchanged, and the initial predicate corresponding to the initial relevance distribution information can be kept, so as to associate the predicate corresponding to the pairwise object detection information with the preset predicate containing a large amount of semantic information to obtain the target predicate.

[0129] Based on the semantic adjustment matrix in the preset matrix, when it is recognized that the predicate corresponding to the predicate relationship between the pairwise object detection information is a universal predicate, the initial relevance distribution information can be semantically adjusted to convert the recognition result with less semantic information into a recognition result with rich semantic information. Based on the semantic preservation matrix in the preset matrix, when it is recognized that the predicate corresponding to the predicate relationship between the pairwise object detection information is a non-universal predicate, the semantic adjustment of the initial relevance distribution information can be reduced to maintain the recognition result with rich semantic information.

[0130] When performing semantic adjustment, the recognition results that do not have rich semantic information are adjusted, and the recognition results that have rich semantic information are maintained, so as to avoid erroneous adjustment of predicates that have rich semantic information, thereby improving the effectiveness of semantic adjustment.

[0131] S230. Input the visual relationship and the object detection information corresponding to the visual relationship into the scene graph generation model to generate the scene graph, and obtain the target scene graph corresponding to the image to be processed. The target scene graph is structural information annotated with visual relationships between two objects.

[0132] As an optional embodiment, the visual relationship and the object detection information corresponding to the visual relationship are input into the scene graph generation model. According to the visual relationship between the objects corresponding to the object detection information, the target scene graph can be obtained. The target scene graph is structural information composed of points and edges. The points in the target scene graph can represent objects, and the edges can represent the visual relationship between the two objects. The visual relationship with subject, predicate and object can be displayed in the target scene graph. For example, the visual relationship (person, have, hand) is displayed in the target scene graph, where "person" is the subject, "have" is the predicate, and "hand" is the object.

[0133] As an optional embodiment, the method also includes a model training method, see Figure 5 , model training methods include:

[0134] S510. Input the annotated image into the image detection model to perform object detection, and obtain the training object detection information corresponding to each object in the annotated image;

[0135] S520. Input the training object detection information into the first model to be trained to perform visual relationship detection to obtain a first training visual relationship between two objects;

[0136] S530. Inputting the first training visual relationship and the training object detection information corresponding to the first training visual relationship into the second to-be-trained model to generate a scene graph, and obtaining a first training scene graph corresponding to the annotated image;

[0137] S540. According to the first training visual relationship and the reference visual relationship, the first model to be trained and the second model to be trained are trained to obtain a first visual relationship detection model and an initial scene graph generation model.

[0138] As an optional embodiment, during model training, the image detection model is a pre-trained model, so the training object detection information required for training can be extracted through the image detection model, and the training object detection information is the object detection information corresponding to the object in the annotated image. A fully supervised training method is adopted to obtain an annotated image, in which the reference visual relationship between two objects is annotated. The annotated image is input into the image detection model, and each object in the annotated image is detected according to a preset annotation box, and the detection area of ​​each object in the annotated image is extracted. The feature information corresponding to each object is extracted in the detection area corresponding to the object, and the object in the annotated image is determined according to the feature information corresponding to each object, so as to obtain the training object detection information. At this time, the first model to be trained does not include a preset matrix, that is, the first model to be trained does not have the function of semantic adjustment. After the preset matrix to be trained is trained based on the training visual relationship and the reference visual relationship, the trained preset matrix is ​​added to the first visual relationship detection model.

[0139] The training object detection information is input into the first model to be trained for visual relationship detection. Based on the first network to be trained in the first model to be trained, the predicate corresponding to the predicate relationship between the training object detection information can be identified to obtain a first training target predicate. Based on the second network to be trained in the first model to be trained, the training target predicate and the object corresponding to the first training target predicate can be combined to obtain a first training visual relationship with a subject, a predicate and an object. The first training visual relationship represents the interactive relationship between the two objects in the annotated image obtained by the first model to be trained.

[0140] The first training visual relationship and the training object detection information corresponding to the first training visual relationship are input into the second model to be trained to generate a scene graph. In the second model to be trained, the first training visual relationship between objects in the annotated image is annotated to obtain a first training scene graph corresponding to the annotated image. The first training scene graph is structural information annotated with the first training visual relationship between each pair of objects.

[0141] Calculate the first loss data between the first training visual relationship and the reference visual relationship. The first loss data can be a loss function between the first training visual relationship and the reference visual relationship. According to the first loss data, the first model to be trained and the second model to be trained are trained to obtain a first visual relationship detection model and an initial scene graph generation model. The first visual relationship detection model is a visual relationship detection model without a preset matrix.

[0142] When training the model, you only need to train the visual relationship detection model and the scene graph generation model, and complete most of the training steps based on the source domain. Subsequently, only fine-tuning is required on the target domain, which reduces the training cost.

[0143] As an alternative embodiment, see Figure 6 After training the first model to be trained and the second model to be trained according to the training visual relationship and the reference visual relationship to obtain the first visual relationship detection model and the initial scene graph generation model, the method further includes:

[0144] S610. Detecting the word frequency information corresponding to each reference predicate in the reference visual relationship;

[0145] S620. Classify the reference predicates according to the preset word frequency segmentation information and word frequency information to obtain the reference predicate type corresponding to each annotated image;

[0146] S630. Combining the first visual relationship detection model and the preset matrix to obtain a second visual relationship detection model;

[0147] S640. Input the training object detection information into the second visual relationship detection model to perform visual relationship detection to obtain a second training visual relationship between two objects;

[0148] S650. Inputting the second training visual relationship and the training object detection information corresponding to the second training visual relationship into the initial scene graph generation model to generate a scene graph, and obtaining a second training scene graph corresponding to the annotated image;

[0149] S660. Based on the reference predicate type corresponding to each annotated image, the second training visual relationship and the reference visual relationship, the second visual relationship detection model and the initial scene graph generation model are adjusted to obtain the visual relationship detection model and the scene graph generation model.

[0150] As an alternative embodiment, see Figure 7 ,like Figure 7The figure shows a schematic diagram of transfer learning. Since predicates with a small probability of occurrence contain more semantic information, the word frequency information corresponding to each reference predicate in the reference visual relationship is detected. The word frequency information represents the probability of each reference predicate appearing in all reference predicates. The semantic information contained in each reference predicate is calculated based on the word frequency information. The formula for estimating the semantic information contained in the reference predicate is as follows:

[0151] I(y i )=-log b [Pr(y i )]

[0152] Among them, y i represents the predicate, Pr(y i ) represents the word frequency information, that is, the probability of the predicate appearing, I(y i ) represents the amount of semantic information in the predicate. The smaller the word frequency information, the greater the amount of semantic information contained in the predicate, and the greater the word frequency information, the smaller the amount of semantic information contained in the predicate. According to the amount of semantic information, the reference predicates are sorted from small to large to obtain a reference predicate sequence. The preset number of reference predicates starting from the first reference predicate are taken as universal predicates, and the reference predicates other than the preset number of reference predicates are taken as non-universal predicates, and the reference predicates are divided into two categories. Universal predicates are predicates with a probability of occurrence greater than the preset probability, and non-universal predicates are predicates with a probability of occurrence less than the preset probability, and the preset probability corresponds to the probability of occurrence of the last reference predicate in the preset number of reference predicates. For example, the preset number can be 15, that is, the first fifteen reference predicates in the reference predicate sequence are taken as universal predicates, and the reference predicates after the first fifteen reference predicates are taken as non-universal predicates.

[0153] The annotated image is used as the source domain, and the annotated image of the general predicate is downsampled to obtain the target domain. The first visual relationship detection model, the preset matrix, and the initial scene graph generation model trained on the source domain are migrated to the target domain. The first visual relationship detection model and the preset matrix are combined to obtain a second visual relationship detection model. The last neural network layer in the neural network layers arranged in sequence in the second visual relationship detection model and the initial scene graph generation model is adjusted to obtain a visual relationship detection model and a scene graph generation model, and the last neural network layer is a classification layer. When adjusting the second visual relationship detection model and the initial scene graph generation model on the target domain, sample images can be obtained from annotated images with reference predicate types for adjustment, without using all annotated images.

[0154] When adjusting the first visual relationship detection model and the initial scene graph generation model, a preset matrix can be obtained, which is a matrix trained based on the first visual relationship detection model and the initial scene graph generation model. The preset matrix and the first visual relationship detection model are combined to obtain a second visual relationship detection model, which is a visual relationship detection model with a preset matrix.

[0155] After inputting the annotated image into the image detection model for object detection and obtaining the training object detection information, the training object detection information is input into the second visual relationship detection model for visual relationship detection to obtain the second training visual relationship between the two objects, and the second training visual relationship represents the interaction relationship between the two objects in the annotated image obtained by the second visual relationship detection model. The preset matrix can semantically adjust the initial relevance distribution information corresponding to the training object detection information to obtain the target relevance distribution information, and the second target training predicate can be determined according to the target relevance distribution information to obtain the second training visual relationship.

[0156] The second training visual relationship and the training object detection information corresponding to the second training visual relationship are input into the initial scene graph generation model to generate the scene graph, so as to obtain the second training scene graph corresponding to the annotated image. The second training scene graph is structural information annotated with the second training visual relationship between two objects.

[0157] The second training visual relationship is a detection result obtained based on the amount of information corresponding to the reference predicate type. The second loss data between the reference predicate type and the reference visual relationship corresponding to each annotated image and the second training visual relationship is calculated. The second loss data can be a loss function between the reference predicate type and the reference visual relationship and the second training visual relationship. According to the second loss data, the second visual relationship detection model and the initial scene graph generation model are adjusted to obtain a visual relationship detection model and a scene graph generation model.

[0158] According to the amount of semantic information contained in the predicates in the annotated images, the annotated images are classified, and then the model is adjusted based on the reference predicate type, training visual relationship and reference visual relationship, so that the model can have the ability to identify common predicates and non-common predicates, improving the accuracy of predicate recognition. In addition, the second visual relationship detection model and the initial scene graph generation model are adjusted instead of retraining to avoid overfitting problems.

[0159] As an alternative embodiment, see Figure 8 , according to the training visual relationship and the reference visual relationship, the first model to be trained and the second model to be trained are trained to obtain the first visual relationship detection model and the initial scene graph generation model, including:

[0160] S810. Input the training object detection information into the first visual relationship detection model to perform visual relationship detection to obtain the initial visual relationship between the two objects;

[0161] S820. Inputting the initial visual relationship and the training object detection information corresponding to the initial visual relationship into the initial scene graph generation model to generate a scene graph, and obtaining an initial scene graph corresponding to the annotated image, wherein the initial scene graph is structural information annotated with initial visual relationships between two objects;

[0162] S830. Determine an initial matrix according to the predicate in the initial visual relationship and the reference predicate in the reference visual relationship;

[0163] S840. Obtain a preset matrix according to the normalized matrix corresponding to the initial matrix and the unit matrix corresponding to the initial matrix.

[0164] As an optional embodiment, the training object detection information is input into the first visual relationship detection model for visual relationship detection, and initial correlation distribution information can be obtained. The initial predicate can be determined based on the initial correlation distribution information. The initial visual relationship between each pair of objects can be obtained through the initial predicate. The initial visual relationship represents the interactive relationship between each pair of objects in the annotated image obtained by the first visual relationship detection model.

[0165] Then, the initial visual relationship and the training object detection information corresponding to the initial visual relationship are input into the initial scene graph generation model to generate the scene graph, and the initial scene graph corresponding to the annotated image is obtained. The initial scene graph is the structural information annotated with the initial visual relationship between two objects. By comparing the predicates in the initial visual relationship with the reference predicates in the reference visual relationship, the correctly classified predicates and the incorrectly classified predicates can be determined.

[0166] As an optional embodiment, the preset matrix can be expressed as:

[0167] C * ∈R K×K

[0168] Among them, the matrix C * represents the preset matrix, R represents the preset predicate, K represents the number of preset predicates, and the preset predicates are different types of predicates. In the process of obtaining the preset matrix, the confusion matrix used to identify the predicate is first initialized to obtain the initial matrix, which can be expressed as:

[0169] C∈R K×K

[0170] Each element in the initial matrix is ​​represented by C j,k, this element represents the number of predicates that are marked as the jth category but predicted as the kth category, where j can be equal to k. When j is equal to k, it means that the labeling result of the predicate is consistent with the recognition result of the predicate.

[0171] The elements in the semantic adjustment matrix of the preset matrix are represented as the number of predicates marked as the jth category predicates but predicted as the kth category predicates. Therefore, the elements in the semantic adjustment matrix can be determined according to the number of correctly classified predicates and the number of incorrectly classified predicates. For example, there are 100 A predicates in the reference predicates, and the A predicates correspond to category number 3. However, in the initial visual relationship, there are only 50 A predicates among the predicates corresponding to the 100 A predicates, and there are also 30 B predicates and 20 C predicates. The B predicates correspond to category number 4, and the C category corresponds to predicate number 5. Then the number of correctly classified predicates is 50, and the number of incorrectly classified predicates is 30 and 20, which can be represented as C in the matrix. 3,3 , C 3,4 and C 3,5 .

[0172] According to the normalized matrix corresponding to the initial matrix and the unit matrix corresponding to the initial matrix, the preset matrix can be obtained. The normalized matrix corresponding to the initial matrix is ​​the semantic adjustment matrix, and the unit matrix corresponding to the initial matrix is ​​the semantic preservation matrix. The initial matrix is ​​normalized to obtain the normalized semantic adjustment matrix C′, which can be calculated by the following formula:

[0173]

[0174] The semantic adjustment matrix C′ represents the semantic relevance between predicates to a certain extent, but the diagonal elements of the semantic adjustment matrix C′ have smaller values ​​for predicates with rich semantic information. Directly multiplying the initial relevance distribution information by the semantic adjustment matrix C′ will reduce the probability of the identified predicates with rich semantic information. Therefore, adding a semantic preservation matrix to the semantic adjustment matrix C′ can retain the recognition results of predicates with rich semantic information based on the semantic preservation matrix, and the preset matrix C can be obtained. * The specific formula is as follows:

[0175] C * =(C′+I K )×0.5

[0176] Among them, I K ∈R K×K is an identity matrix. Multiplying the entire formula by 0.5 ensures that the sum of the elements in each row of the preset matrix is ​​1.

[0177] After obtaining the preset matrix, the preset matrix is ​​added to the initial scene graph generation model, so that the initial scene graph generation model has the function of semantic adjustment.

[0178] During the training process, a pre-trained preset matrix is ​​used, and the parameters in the preset matrix are fixed, thereby avoiding semantic drift and improving the accuracy of semantic adjustment.

[0179] As an alternative embodiment, see Fig. 9 ,like Fig. 9 The figure shows a schematic diagram of the process of inputting the image to be processed and generating the target scene graph. The image to be processed is input into the image detection model for object detection. The annotation box and feature information of each object are detected, and the positions of the four objects "racket", "hand", "person" and "short sleeves" can be determined, and the object detection information of the four objects "racket", "hand", "person" and "short sleeves" can be obtained. The object detection information is input into the visual relationship detection model to obtain the visual relationship between two objects. In the visual relationship detection model, the predicates corresponding to the two objects "racket" and "hand" are identified, and the target predicate "on top" can be obtained, so that the visual relationship composed of the subject, predicate and object (racket, on top, hand) can be determined. The predicates corresponding to the two objects "person" and "hand" are identified, and the target predicate "has" can be obtained, so that the visual relationship composed of the subject, predicate and object (person, has, hand) can be determined. By identifying the predicates corresponding to the two objects "short sleeves" and "person", the target predicate "on top" can be obtained, so that the visual relationship (short sleeves, on top, person) composed of the subject, predicate and object can be determined. The visual relationship (racket, on top, hand), the visual relationship (person, has, hand), the visual relationship (short sleeves, on top, person), and the object detection information of the four objects "racket", "hand", "person" and "short sleeves" are input into the scene graph generation model, and the visual relationship between the two objects is annotated in the object detection information corresponding to each object to obtain the target scene graph. The target scene graph can be applied to image retrieval, visual question answering and other aspects. For example, when searching for an image, the input search information is to search for the image of "person wearing short sleeves", then the target scene graph with the visual relationship (short sleeves, on top, person) in the target scene graph generated by the above-mentioned visual relationship detection model and scene graph generation model can be searched to obtain the search result. Or when the user inputs the question "What is the person wearing?", by identifying the visual relationship of (short-sleeved, on top, person) in the target scene graph generated by the above-mentioned visual relationship detection model and the scene graph generation model, the answer "short-sleeved" is obtained, thereby completing this visual question and answer.

[0180] The disclosed embodiment proposes an image processing method, which inputs an image to be processed into an image detection model to perform object detection, obtains object detection information, and inputs the object detection information into a visual relationship detection model to perform visual relationship detection. In the process of visual relationship detection, semantic adjustment is performed on predicates corresponding to predicate relationships between detected pairs of objects, so that target predicates contain rich semantic information, thereby improving the accuracy of predicate recognition. In subsequent steps, visual relationships between pairs of objects are generated through target predicates and objects corresponding to the target predicates, and the visual relationships are input into a scene graph generation model to generate a target scene graph, thereby improving the accuracy of visual relationships annotated in the target scene graph and thus improving the effectiveness of the target scene graph.

[0181] Fig.10 FIG. 1 is a block diagram of an image processing device according to an exemplary embodiment. Fig.10 , the device comprises:

[0182] The object detection module 1010 is configured to input the image to be processed into the image detection model to perform object detection, and obtain object detection information corresponding to at least two objects in the image to be processed;

[0183] The visual relationship detection module 1020 is configured to perform visual relationship detection by inputting the object detection information into the visual relationship detection model to obtain the visual relationship between the two objects, where the visual relationship represents the interaction relationship between the two objects in the image to be processed;

[0184] The scene graph generation module 1030 is configured to perform image processing by inputting visual relationships and object detection information corresponding to the visual relationships into a scene graph generation model to obtain a target scene graph corresponding to the image to be processed, wherein the target scene graph is structural information annotated with visual relationships between pairs of objects.

[0185] As an optional embodiment, the visual relationship detection model includes a predicate recognition network, and the visual relationship detection module 1020 includes:

[0186] A predicate recognition unit is configured to input the object detection information into the predicate recognition network to perform predicate recognition corresponding to the predicate relationship between two objects, and obtain a target predicate, wherein the target predicate represents the semantically adjusted predicate;

[0187] The visual relationship determining unit is configured to execute the target predicate and the object corresponding to the target predicate as the visual relationship.

[0188] As an optional embodiment, the predicate recognition network includes an initial relevance calculation layer and a semantic adjustment layer, and the predicate recognition unit includes:

[0189] an initial relevance calculation unit, configured to input the object detection information and the preset predicate into the initial relevance calculation layer, calculate the relevance between the predicates corresponding to the pairwise object detection information and each preset predicate, and obtain initial relevance distribution information, wherein the initial relevance distribution information represents the relevance between the predicates corresponding to the pairwise object detection information and each preset predicate before semantic adjustment;

[0190] A semantic adjustment unit is configured to input the initial relevance distribution information into the semantic adjustment layer, perform predicate semantic adjustment on the initial relevance distribution information based on a preset matrix, and obtain target relevance distribution information, wherein the target relevance distribution information represents the relevance between the predicate corresponding to the pairwise object detection information and each preset predicate after the semantic adjustment;

[0191] The target predicate determination unit is configured to determine the target predicate according to the target relevance distribution information.

[0192] As an optional embodiment, the semantic adjustment unit includes:

[0193] An initial predicate determination unit is configured to determine an initial predicate according to initial relevance distribution information;

[0194] A first semantic adjustment unit is configured to perform predicate semantic adjustment on the initial relevance distribution information based on a semantic adjustment matrix in a preset matrix when the initial predicate is a universal predicate, wherein the universal predicate represents a predicate in the preset predicate whose use probability is greater than a preset threshold;

[0195] The second semantic adjustment unit is configured to determine the initial relevance distribution information as the target relevance distribution information based on the semantic preservation matrix in the preset matrix when the initial predicate is a non-universal predicate, wherein the non-universal predicate represents a predicate in the preset predicate whose use probability is less than a preset threshold.

[0196] As an optional embodiment, the device further includes:

[0197] A first training feature extraction module is configured to perform feature extraction by inputting the annotated image into the image detection model to obtain training object detection information corresponding to each object in the annotated image;

[0198] A first training visual relationship detection module is configured to perform visual relationship detection by inputting the training object detection information into the first model to be trained to obtain a first training visual relationship between two objects, wherein the first training visual relationship represents an interactive relationship between two objects in the annotated image obtained by the first model to be trained;

[0199] A first training scene graph generation module is configured to execute inputting the first training visual relationship and the training object detection information corresponding to the first training visual relationship into the second to-be-trained model to generate a scene graph, and obtain a first training scene graph corresponding to the annotated image, wherein the first training scene graph is annotated with structural information of the first training visual relationship between two objects;

[0200] The model training module is configured to train the first model to be trained and the second model to be trained according to the training visual relationship and the reference visual relationship to obtain a first visual relationship detection model and an initial scene graph generation model.

[0201] As an optional embodiment, the device further includes:

[0202] A word frequency information detection module is configured to detect word frequency information corresponding to each reference predicate in the reference visual relationship;

[0203] A second visual relationship detection model acquisition module is configured to combine the first visual relationship detection model with a preset matrix to obtain a second visual relationship detection model;

[0204] A second training visual relationship acquisition module is configured to input the training object detection information into a second visual relationship detection model to perform visual relationship detection to obtain a second training visual relationship between two objects, wherein the second training visual relationship represents an interactive relationship between two objects in the annotated image obtained by the second visual relationship detection model;

[0205] A second training scene graph acquisition module is configured to execute inputting the second training visual relationship and the training object detection information corresponding to the second training visual relationship into the initial scene graph generation model to generate a scene graph, and obtain a second training scene graph corresponding to the annotated image, wherein the second training scene graph is structural information annotated with the second training visual relationship between two objects;

[0206] The model adjustment module is configured to adjust the second visual relationship detection model and the initial scene graph generation model based on the reference predicate type corresponding to each annotated image, the second training visual relationship and the reference visual relationship to obtain the visual relationship detection model and the scene graph generation model.

[0207] As an optional embodiment, the device further includes:

[0208] an initial visual relationship detection module, configured to input object detection information into a first visual relationship detection model to perform visual relationship detection, and obtain an initial visual relationship between two objects, wherein the initial visual relationship represents an interactive relationship between two objects in the annotated image obtained by the first visual relationship detection model;

[0209] The scene initial graph generation module is configured to execute the scene graph generation by inputting the initial visual relationship and the training object detection information corresponding to the initial visual relationship into the initial scene graph generation model to obtain the initial scene graph corresponding to the annotated image, wherein the initial scene graph is the structural information annotated with the initial visual relationship between two objects;

[0210] An initial matrix determination module is configured to determine an initial matrix according to a predicate in the initial visual relationship and a reference predicate in the reference visual relationship;

[0211] The preset matrix determination module is configured to execute a normalized matrix corresponding to the initial matrix and a unit matrix corresponding to the initial matrix to obtain a preset matrix.

[0212] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0213] Fig.11 is a block diagram of an electronic device for image processing according to an exemplary embodiment. The electronic device may be a server, and its internal structure diagram may be as shown in FIG. Fig.11 As shown. The electronic device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the electronic device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an image processing method is implemented.

[0214] Those skilled in the art will understand that Fig.11 The structure shown in the figure is merely a block diagram of a partial structure related to the scheme of the present disclosure, and does not constitute a limitation on the electronic device to which the scheme of the present disclosure is applied. The specific electronic device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0215] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 1104 including instructions, and the instructions can be executed by a processor 1120 of the electronic device 1100 to complete the above method. Alternatively, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0216] In an exemplary embodiment, a computer program product is also provided, comprising computer instructions, which implement the above-mentioned image processing method when executed by a processor.

[0217] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0218] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An image processing method, characterized in that: The method comprises: Inputting the image to be processed into the image detection model to perform object detection, and obtaining object detection information corresponding to at least two objects in the image to be processed; Inputting the object detection information into a visual relationship detection model to perform visual relationship detection to obtain a visual relationship between two objects, wherein the visual relationship represents an interactive relationship between two objects in the image to be processed; Inputting the visual relationship and the object detection information corresponding to the visual relationship into a scene graph generation model to generate a scene graph, thereby obtaining a target scene graph corresponding to the image to be processed, wherein the target scene graph is structural information annotated with the visual relationship between the two objects; The method for generating the visual relationship detection model and the scene graph generation model comprises: The annotated image is used as a source domain, and the annotated image of the general predicate is downsampled to obtain a target domain; the annotated image is annotated with reference visual relations between two objects; the general predicate is a preset number of reference predicates starting from the first reference predicate in a reference predicate sequence, and the reference predicate sequence is obtained by sorting the reference predicates from small to large according to the amount of semantic information; Migrating the first visual relationship detection model, the preset matrix, and the initial scene graph generation model trained on the source domain to the target domain, and combining the first visual relationship detection model and the preset matrix to obtain a second visual relationship detection model; Adjusting the last neural network layer in the neural network layers arranged in sequence in the second visual relationship detection model and the initial scene graph generation model to obtain a visual relationship detection model and a scene graph generation model; The method for determining the preset matrix is: Inputting the training object detection information into the first visual relationship detection model to perform visual relationship detection to obtain an initial visual relationship between the two objects, wherein the initial visual relationship represents an interactive relationship between the two objects in the annotated image obtained by the first visual relationship detection model; Inputting the initial visual relationship and the corresponding training object detection information into the initial scene graph generation model to generate a scene graph, thereby obtaining an initial scene graph corresponding to the annotated image, wherein the initial scene graph is structural information annotated with the initial visual relationship between the two objects; determining an initial matrix according to the predicate in the initial visual relationship and the reference predicate in the reference visual relationship; A preset matrix is ​​obtained according to a normalized matrix corresponding to the initial matrix and a unit matrix corresponding to the initial matrix.

2. The image processing method according to claim 1, characterized in that: The visual relationship detection model includes a predicate recognition network, and the object detection information is input into the visual relationship detection model to perform visual relationship detection to obtain the visual relationship between two objects. Inputting the object detection information output by the image detection model into the predicate recognition network to perform predicate recognition corresponding to the predicate relationship between two objects, and obtaining a target predicate, wherein the target predicate represents the semantically adjusted predicate; The visual relationship is obtained according to the target predicate and the object corresponding to the target predicate.

3. The image processing method according to claim 2, characterized in that: The predicate recognition network includes an initial relevance calculation layer and a semantic adjustment layer. The object detection information is input into the predicate recognition network to perform predicate recognition between two objects, and the target predicate is obtained, including: Inputting the object detection information and the preset predicates into the initial relevance calculation layer, performing relevance calculation on the predicates corresponding to the pairwise object detection information and each preset predicate, and obtaining initial relevance distribution information, wherein the initial relevance distribution information represents the relevance between the predicates corresponding to the pairwise object detection information and each preset predicate before semantic adjustment; Inputting the initial relevance distribution information into a semantic adjustment layer, performing predicate semantic adjustment on the initial relevance distribution information based on the preset matrix, and obtaining target relevance distribution information, wherein the target relevance distribution information represents the relevance between the predicate corresponding to the pairwise object detection information and each preset predicate after the semantic adjustment; The target predicate is determined according to the target relevance distribution information.

4. The image processing method according to claim 3, characterized in that: The step of inputting the initial relevance distribution information into the semantic adjustment layer, and performing predicate semantic adjustment on the initial relevance distribution information based on the preset matrix to obtain the target relevance distribution information comprises: Determining an initial predicate according to the initial relevance distribution information; In the case where the initial predicate is a universal predicate, based on the semantic adjustment matrix in the preset matrix, the initial relevance distribution information is subjected to predicate semantic adjustment, wherein the universal predicate represents a predicate in the preset predicate whose use probability is greater than a preset threshold; In the case where the initial predicate is a non-universal predicate, the initial relevance distribution information is determined as the target relevance distribution information based on a semantic preservation matrix in the preset matrix, and the non-universal predicate represents a predicate in the preset predicate whose usage probability is less than a preset threshold.

5. The image processing method according to claim 1, characterized in that: The method further comprises: Inputting the annotated image into the image detection model to perform object detection, and obtaining training object detection information corresponding to each object in the annotated image, wherein the annotated image is annotated with reference visual relationships between two objects; Inputting the training object detection information into a first model to be trained to perform visual relationship detection, thereby obtaining a first training visual relationship between two objects, wherein the first training visual relationship represents an interactive relationship between two objects in the annotated image obtained by the first model to be trained; Inputting the first training visual relationship and the training object detection information corresponding to the first training visual relationship into the second to-be-trained model to generate a scene graph, thereby obtaining a first training scene graph corresponding to the annotated image, wherein the first training scene graph is annotated with structural information of the first training visual relationship between the two objects; According to the first training visual relationship and the reference visual relationship, the first model to be trained and the second model to be trained are trained to obtain a first visual relationship detection model and an initial scene graph generation model, wherein the first visual relationship detection model is a visual relationship detection model without a preset matrix.

6. The image processing method according to claim 5, characterized in that: After training the first model to be trained and the second model to be trained according to the first training visual relationship and the reference visual relationship to obtain the first visual relationship detection model and the initial scene graph generation model, the method further includes: detecting word frequency information corresponding to each reference predicate in the reference visual relationship; Classifying the reference predicates according to preset word frequency segmentation information and the word frequency information to obtain a reference predicate type corresponding to each annotated image; Combining the first visual relationship detection model and a preset matrix to obtain a second visual relationship detection model; Inputting the training object detection information into the second visual relationship detection model to perform visual relationship detection to obtain a second training visual relationship between the two objects, wherein the second training visual relationship represents an interactive relationship between the two objects in the annotated image obtained by the second visual relationship detection model; Inputting the training object detection information corresponding to the second training visual relationship and the first training visual relationship into the initial scene graph generation model to generate a scene graph, thereby obtaining a second training scene graph corresponding to the annotated image, wherein the second training scene graph is annotated with structural information of the second training visual relationship between the two objects; Based on the reference predicate type corresponding to each annotated image, the second training visual relationship and the reference visual relationship, the second visual relationship detection model and the initial scene graph generation model are adjusted to obtain the visual relationship detection model and the scene graph generation model.

7. An image processing device, characterized in that: The device comprises: An object detection module is configured to input the image to be processed into an image detection model to perform object detection, and obtain object detection information corresponding to at least two objects in the image to be processed; A visual relationship detection module is configured to input the object detection information into a visual relationship detection model to perform visual relationship detection to obtain a visual relationship between two objects, wherein the visual relationship represents an interactive relationship between two objects in the image to be processed; A scene graph generation module is configured to execute the scene graph generation by inputting the visual relationship and the object detection information corresponding to the visual relationship into a scene graph generation model to obtain a target scene graph corresponding to the image to be processed, wherein the target scene graph is a structural information annotated with the visual relationship between the two objects; The device also includes a model generation module configured to execute: The annotated image is used as a source domain, and the annotated image of the general predicate is downsampled to obtain a target domain; the annotated image is annotated with reference visual relations between two objects; the general predicate is a preset number of reference predicates starting from the first reference predicate in a reference predicate sequence, and the reference predicate sequence is obtained by sorting the reference predicates from small to large according to the amount of semantic information; Migrating the first visual relationship detection model, the preset matrix, and the initial scene graph generation model trained on the source domain to the target domain, and combining the first visual relationship detection model and the preset matrix to obtain a second visual relationship detection model; Adjusting the last neural network layer in the neural network layers arranged in sequence in the second visual relationship detection model and the initial scene graph generation model to obtain a visual relationship detection model and a scene graph generation model; The device also includes: an initial visual relationship detection module, configured to input the training object detection information into the first visual relationship detection model to perform visual relationship detection, and obtain the initial visual relationship between the two objects, wherein the initial visual relationship represents the interaction relationship between the two objects in the annotated image obtained by the first visual relationship detection model; A scene initial graph generation module is configured to execute scene graph generation by inputting the initial visual relationship and the corresponding training object detection information into the initial scene graph generation model to obtain an initial scene graph corresponding to the annotated image, wherein the initial scene graph is annotated with structural information of the initial visual relationship between the two objects; an initial matrix determination module, configured to determine an initial matrix according to the predicate in the initial visual relationship and the reference predicate in the reference visual relationship; The preset matrix determination module is configured to execute a normalized matrix corresponding to the initial matrix and a unit matrix corresponding to the initial matrix to obtain a preset matrix.

8. The image processing device according to claim 7, characterized in that: The visual relationship detection model includes a predicate recognition network, and the visual relationship detection module includes: A predicate recognition unit is configured to input the object detection information output by the image detection model into the predicate recognition network to perform predicate recognition corresponding to the predicate relationship between two objects, and obtain a target predicate, wherein the target predicate represents the semantically adjusted predicate; The visual relationship determining unit is configured to obtain the visual relationship according to the target predicate and the object corresponding to the target predicate.

9. The image processing device according to claim 8, characterized in that: The predicate recognition network includes an initial relevance calculation layer and a semantic adjustment layer, and the predicate recognition unit includes: an initial relevance calculation unit, configured to input the object detection information and the preset predicate into the initial relevance calculation layer, perform relevance calculation on the predicates corresponding to the pairwise object detection information and each preset predicate, and obtain initial relevance distribution information, wherein the initial relevance distribution information represents the relevance between the predicates corresponding to the pairwise object detection information and each preset predicate before semantic adjustment; A semantic adjustment unit is configured to input the initial relevance distribution information into a semantic adjustment layer, perform predicate semantic adjustment on the initial relevance distribution information based on the preset matrix, and obtain target relevance distribution information, wherein the target relevance distribution information represents the relevance between the predicate corresponding to the pairwise object detection information and each preset predicate after the semantic adjustment; The target predicate determination unit is configured to determine the target predicate according to the target relevance distribution information.

10. The image processing device according to claim 9, characterized in that: The semantic adjustment unit comprises: an initial predicate determination unit, configured to determine an initial predicate according to the initial relevance distribution information; A first semantic adjustment unit is configured to perform predicate semantic adjustment on the initial relevance distribution information based on a semantic adjustment matrix in the preset matrix when the initial predicate is a universal predicate, wherein the universal predicate represents a predicate in the preset predicate whose use probability is greater than a preset threshold; The second semantic adjustment unit is configured to determine the initial relevance distribution information as the target relevance distribution information based on a semantic preservation matrix in the preset matrix when the initial predicate is a non-universal predicate, wherein the non-universal predicate represents a predicate in the preset predicate whose usage probability is less than a preset threshold.

11. The image processing device according to claim 10, characterized in that: The device also includes: A first training feature extraction module is configured to perform feature extraction by inputting the annotated image into the image detection model to obtain training object detection information corresponding to each object in the annotated image, wherein the annotated image is annotated with reference visual relationships between the two objects; A first training visual relationship detection module is configured to input the training object detection information into a first model to be trained to perform visual relationship detection to obtain a first training visual relationship between the two objects, wherein the first training visual relationship represents an interactive relationship between the two objects in the annotated image obtained by the first model to be trained; A first training scene graph generation module is configured to execute the step of inputting the first training visual relationship and the training object detection information corresponding to the first training visual relationship into a second to-be-trained model to generate a scene graph, thereby obtaining a first training scene graph corresponding to the annotated image, wherein the first training scene graph is structural information annotated with the first training visual relationship between the two objects; The model training module is configured to train the first model to be trained and the second model to be trained according to the first training visual relationship and the reference visual relationship to obtain a first visual relationship detection model and an initial scene graph generation model.

12. The image processing device according to claim 11, characterized in that: The device also includes: A word frequency information detection module is configured to detect word frequency information corresponding to each reference predicate in the reference visual relationship; A reference predicate classification module is configured to classify the reference predicates according to preset word frequency segmentation information and the word frequency information to obtain a reference predicate type corresponding to each annotated image; A second visual relationship detection model acquisition module is configured to combine the first visual relationship detection model with a preset matrix to obtain a second visual relationship detection model; A second training visual relationship acquisition module is configured to input the training object detection information into the second visual relationship detection model to perform visual relationship detection to obtain a second training visual relationship between the two objects, wherein the second training visual relationship represents an interactive relationship between the two objects in the annotated image obtained by the second visual relationship detection model; A second training scene graph acquisition module is configured to execute inputting the training object detection information corresponding to the second training visual relationship and the first training visual relationship into the initial scene graph generation model to generate a scene graph, so as to obtain a second training scene graph corresponding to the annotated image, wherein the second training scene graph is structural information annotated with the second training visual relationship between the two objects; The model adjustment module is configured to adjust the second visual relationship detection model and the initial scene graph generation model based on the reference predicate type corresponding to each annotated image, the second training visual relationship and the reference visual relationship to obtain the visual relationship detection model and the scene graph generation model.

13. An electronic device, characterized in that: The electronic device comprises: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the image processing method according to any one of claims 1 to 6.

14. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the image processing method according to any one of claims 1 to 6.

15. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by a processor, the image processing method according to any one of claims 1 to 6 is implemented.

Images