A visual relationship detection method, device, equipment and storage medium

Abstract

The present application relates to the technical field of visual relationship detection, and discloses a visual relationship detection method, device, equipment and storage medium, the method comprising: acquiring at least one image of a spatial environment region to be identified; according to the acquired image, a preset target feature extraction model is used to acquire the relationship features between the preset targets in the image, and then the visual relationship triplets of the scene graph in the image are obtained; the position relationship features between the preset targets are subjected to cluster analysis, and the preset target positions with overlapping relationship form a subgraph, one scene graph comprises at least one subgraph, and each subgraph comprises at least one visual relationship triplet; according to the acquired triplet information of the preset targets and each subgraph, a preset visual relationship prediction model is used to generate a scene graph composed of each subgraph; and according to the scene graph composed of each subgraph, the visual relationship of the image to be detected is obtained. The method of the present application can accurately determine the visual relationship between the targets when the targets are mutually occluded.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of visual relationship detection, and particularly relates to a visual relationship detection method, device, equipment and storage medium. BACKGROUND

[0002] In recent years, with the continuous acceleration of urbanization and the continuous increase of motor vehicles, many urban traffic congestion, parking conflicts and other problems have also arisen. Therefore, improving the intelligent level of urban traffic management is a social problem that needs to be solved at present. The intelligent level of urban traffic management can effectively improve the current traffic management situation.

[0003] The intelligent urban traffic management system is an intelligent management system that can collect, process and feedback traffic information in real time and accurately, relying on artificial intelligence algorithms, cloud service platforms, intelligent hardware devices and edge computing devices. At present, high-position video monitoring devices are mainly installed at urban road sides, important red and green light intersections and the like to collect monitoring scene video image data, and visual algorithms are used to process and analyze the data to realize analysis of the monitoring scene. For example, evidence capture and warning of traffic violations such as red light running and speeding, guidance and parking space recording of roadside parking, real-time updating, prediction and publishing of traffic congestion conditions, detection and early warning of abnormal behaviors and the like. Through intelligent traffic management, the urban road traffic conditions can be effectively improved, and more evidence about vehicle violations, non-standard driving and the like can be provided to the traffic management department, and more intelligent applications and management of the entire traffic flow can be realized, thereby having a positive promoting effect on urban traffic management, driving safety and other aspects.

[0004] In practice, monitoring devices are usually installed in relatively dense areas of people and vehicles, and the scene is complex. At present, the visual algorithms mainly used include two-dimensional target detection of vehicles, pedestrians, license plates and the like, pixel-by-pixel semantic segmentation of green plants, passable areas, parking areas and the like, to realize some single-function, more visually intuitive practical applications, and then calculate and compare the image positions of the recognized targets to further realize relationship prediction between different targets in the video monitoring image. However, in this process, only pixel positions are relied on to determine the relationship between different targets, and in the monitoring scene, there is a problem of inaccurate relationship determination due to mutual occlusion between targets. SUMMARY

[0005] Therefore, the present application provides a visual relationship detection method, device, equipment and storage medium to solve the problem of inaccurate relationship determination between different targets by relying only on pixels in the monitoring scene when there is mutual occlusion between targets.

[0006] In a first aspect, the present application provides a visual relationship detection method, the method comprising:

[0007] obtaining at least one image of a spatial environment region to be recognized;

[0008] According to the obtained image, a preset target feature extraction model is used to obtain the relationship features between the preset targets in the image, and then a visual relationship triple of a scene graph in the image is obtained;

[0009] The position relationship features between the preset targets are subjected to cluster analysis, and the preset target positions having an overlapping relationship form a subgraph, and a scene graph includes at least one subgraph, and each subgraph includes at least one visual relationship triple;

[0010] According to the obtained triple information of the preset targets and the subgraphs, a preset visual relationship prediction model is used to generate a scene graph composed of the subgraphs;

[0011] According to the scene graph composed of the subgraphs, the visual relationship of the image to be detected is obtained.

[0012] Through the method provided by the present application, different targets in the spatial environment region to be recognized are first identified by using a preset target feature extraction model to obtain the feature information such as the categories and positions of the different targets, and then a scene graph between the different targets is constructed to describe the visual relationship between the different targets, a preset visual relationship prediction model is used to generate a scene graph composed of the subgraphs, and according to the scene graph composed of the subgraphs, the visual relationship of the image to be detected is obtained. The method provided by the present application realizes the extraction of higher-level semantic features, can more comprehensively and meticulously analyze different targets in the image of the spatial environment region to be recognized, realizes more intelligent video image understanding, improves the intelligent analysis degree of the video image, and can accurately judge the visual relationship between the targets when the targets are occluded.

[0013] In an optional embodiment, the preset target feature extraction model comprises:

[0014] obtaining sample data, wherein each sample data includes a preset target and a scene graph in a sample image;

[0015] performing preset target recognition labeling on the preset targets in the sample image to obtain the categories of the preset targets and the positions of the preset targets;

[0016] According to the categories of the preset targets and the positions of the preset targets, the preset targets having a visual relationship in the sample image are subjected to visual relationship labeling by using a scene graph, and the scene graph includes: the preset targets, the attributes of the preset targets, and the relationships between different preset targets;

[0017] According to the visual relationship annotation, by modeling the preset target in the image, the attribute of the preset target and the relationship between the matching preset targets, the position relationship between the preset targets is obtained.

[0018] The preset target feature extraction model can identify the relationship features between the preset targets in the complex scene, the rectangular frame tool is used for target recognition annotation of the preset targets in the sample image, the scene graph is used for visual relationship annotation of the preset targets with visual relationship in the image, and the category of the preset target and the position of the preset target are obtained.

[0019] In an optional embodiment, the position relationship features between the preset targets are subjected to cluster analysis, including:

[0020] The multi-layer convolutional neural network is used for feature extraction of the image, and the feature vector of the preset target position is extracted;

[0021] The region feature extraction network is used for preset target extraction of the feature vector of the preset target position, and preset target information is generated, wherein the preset target information is represented by the following formula:

[0022] O i =[s i ,x i ,y i ,w i ,h i ]

[0023] O i is the original i-th target, s i is the confidence of the extracted preset target, x i , y i are the center point position coordinates of the preset target rectangular frame respectively, w i , h i are the length and width of the target respectively.

[0024] According to the position relationship between the preset target information, the positions of the preset targets with overlapping relationship form a subgraph, and a scene graph includes at least one subgraph, and each subgraph includes at least one visual relationship triple.

[0025] The cluster analysis is used, and the target positions with overlapping relationship are regarded as a whole to form a subgraph, and different targets in the image are analyzed more comprehensively and carefully through a plurality of subgraphs.

[0026] In an optional embodiment, the visual relationship triple is represented by the following formula:

[0027] R = <subject, relation, object>

[0028] Wherein, the subject is a subject, indicating a preset target; the object indicates another preset target having a visual relationship, and the relation is the relationship between the two preset targets.

[0029] The application identifies the action of each target in the image through the visual triplets.

[0030] In an optional embodiment, the preset visual relationship prediction model comprises:

[0031] The features of each preset target and each subgraph are iteratively updated through the preset loss function to generate a scene graph composed of each subgraph.

[0032] By iteratively updating the features of each preset target and each subgraph, the association relationship between the target and the subgraph is fully utilized, the features of each target and subgraph are iteratively updated, the features of the target and subgraph associated with the preset target are increased, and a scene graph composed of each subgraph is generated.

[0033] In an optional embodiment, the preset loss function is:

[0034] L = β1L class + β2L bbox + β3L sg_cls

[0035] Wherein, β1, β2, β3 are weight coefficients of the target classification loss function, the target detection frame regression loss function, and the target and subgraph relationship classification loss function, respectively, L class represents the classification loss function of the target, L bbox is the regression loss function of the target detection frame, and L sg_cls is the classification loss function of the target and the subgraph.

[0036] The features of each preset target and each subgraph are iteratively updated through the preset loss function.

[0037] In an optional embodiment, the features of the target after iteration are represented by the following formula:

[0038]

[0039] Wherein, is the feature of the i-th target after iteration, O i is the feature of the original i-th target, C (o,s)→o represents the target associated with the target o, and the subgraph passes the target O icharacteristics of the target, m represents m subgraphs associated with the target, n represents n targets associated with the target, characteristics of the mth subgraph associated with the target O i , characteristics of the mth subgraph associated with the target O characteristic weight coefficient passed to the target, characteristics of the nth target associated with the target O i , characteristics of the nth target associated with the target O characteristic weight coefficient passed to the target.

[0040] The present application further refines and optimizes the generation of the entire scene graph by iteratively extracting the characteristics of each target and subgraph, adding the characteristics of the associated targets and subgraphs, and generating a scene graph composed of each subgraph.

[0041] In a second aspect, the present application provides a visual relationship detection device, the device comprising:

[0042] An acquisition module is configured to acquire at least one image of a space environment region to be identified.

[0043] A feature extraction module is configured to acquire the relationship features between the preset targets in the image by using a preset target feature extraction model according to the acquired image, and further obtain the visual relationship triplets of the scene graph in the image.

[0044] A cluster analysis module is configured to perform cluster analysis on the position relationship features between the preset targets, and form a subgraph by the preset targets having an overlapping relationship, wherein one scene graph comprises at least one subgraph, and each subgraph comprises at least one visual relationship triplet.

[0045] A prediction module is configured to generate a scene graph composed of each subgraph by using a preset visual relationship prediction model according to the triplet information of the preset targets and each subgraph.

[0046] A detection module is configured to obtain the visual relationship of the image to be detected according to the scene graph composed of each subgraph.

[0047] In a third aspect, the present application provides a computer device, comprising a memory and a processor, the memory and the processor are communicatively connected, the memory stores computer instructions, and the processor executes the computer instructions to perform the visual relationship detection method of the first aspect or any one of the corresponding embodiments.

[0048] In a fourth aspect, the present application provides a computer readable storage medium, having stored thereon computer instructions for causing a computer to execute the visual relationship detection method according to the first aspect or any possible implementation thereof. BRIEF DESCRIPTION OF DRAWINGS

[0049] In order to more clearly illustrate the technical solutions of the present application or the prior art, the drawings needed in the description of the specific embodiments or the prior art will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0050] Figure 1 is a flowchart of a visual relationship detection method according to an embodiment of the present application;

[0051] Figure 2 is a structural block diagram of a visual relationship detection device according to an embodiment of the present application;

[0052] Figure 3 is a hardware structure diagram of a computer device according to an embodiment of the present application. DETAILED DESCRIPTION

[0053] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions of the embodiments of the present application will be described in detail below with reference to the drawings of the embodiments of the present application. Obviously, the described embodiments are some embodiments of the present application, but not all embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0054] The present application is mainly applied to an intelligent urban traffic management system, and performs visual relationship detection and recognition on video images. The visual relationship can include spatial position relationship (such as a target in front of a target), action (such as jumping), preposition or prepositional phrase (such as a target driving a car), etc., which can more comprehensively and meticulously analyze different targets in the monitored video images, realize more intelligent monitored video image understanding, and improve the intelligent analysis degree of the monitored video images.

[0055] According to an embodiment of the present application, a visual relationship detection method embodiment is provided. It should be noted that the steps shown in the flowchart of the drawings can be executed in a computer system such as a group of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0056] A visual relationship detection method is provided in the embodiment, which can be used in the intelligent urban traffic management system, Figure 1 is a flowchart of the visual relationship detection method according to the embodiment of the present application, as shown in the figure, the flow includes the following steps: Figure 1

[0057] In step S101, at least one image of a space environment region to be recognized is acquired.

[0058] In the embodiment of the present application, the image frames of the monitoring videos of different regions are collected by video devices such as cameras, for example, different cities, different streets, different monitoring scenes in different time periods and different seasons, etc. The device for collecting images is not limited here, and the corresponding collection device is selected according to the actual situation.

[0059] In step S102, according to the acquired image, a preset target feature extraction model is used to acquire the relationship features between the preset targets in the image, and then the visual relationship triplets of the scene graph in the image are obtained.

[0060] In the embodiment of the present application, according to the preset target feature extraction model, the relationship features between the preset targets in the image are acquired, and the preset targets include: vehicles, pedestrians, non-motor vehicles, different pets, roadblocks, trolleys, baby strollers, backpacks, helmets, luggage, etc., which are only examples and are not limited thereto. In actual application, the corresponding preset target is selected according to the actually collected image. The relationship between the targets includes action relationship, spatial relationship, and relationship described by preposition phrases and verbs, etc., which are not limited here, and the corresponding relationship is constructed according to the actual situation. Then the visual relationship triplets of the scene graph in the image are obtained to clearly describe the relationship features between the targets.

[0061] In step S103, the position relationship features between the preset targets are analyzed by clustering, the preset target positions with overlapping relationship form a subgraph, and a scene graph includes at least one subgraph, and each subgraph includes at least one visual relationship triplet.

[0062] In the embodiment of the present application, the clustering is performed by using the position relationship between the targets, and the targets with overlapping positions are regarded as a whole to form a subgraph, and the information of each subgraph is clearly described by the visual relationship triplets.

[0063] ​In a specific embodiment, assuming that there is a person wearing a hat, carrying a backpack, and holding a bottle of water in the image, through the target extraction module, the hat, the backpack, the bottle of water, and the pedestrian are obtained as four targets, and 4*(4-1) = 12 target visual relationships are generated, but many of the relationships are not actually meaningful, therefore, through clustering of the targets, the person is taken as a subject, and the hat, the backpack, and the bottle of water are taken as three objects, the relationships among the hat, the backpack, and the bottle of water are cancelled, and the relationships among the meaningless targets are reduced.

[0064] In step S104, according to the obtained preset target and the triple information of each subgraph, a preset visual relationship prediction model is used to generate a scene graph composed of each subgraph.

[0065] In the embodiment of the present application, the preset visual relationship prediction model is used to iteratively update the features of the target and the subgraph to generate a scene graph composed of each subgraph.

[0066] In step S105, according to the scene graph composed of each subgraph, the visual relationship of the to-be-detected image is obtained.

[0067] In the embodiment of the present application, the visual relationship of the target occlusion in the to-be-detected image is obtained through the scene graph composed of each subgraph.

[0068] The visual relationship detection method provided in the embodiment first uses a preset target feature extraction model to identify different targets in a to-be-recognized spatial environment region to obtain the class, position, and other feature information of the different targets, then constructs a scene graph between the different targets to describe the visual relationship between the different targets, uses a preset visual relationship prediction model to generate a scene graph composed of each subgraph, and obtains the visual relationship of the to-be-detected image according to the scene graph composed of each subgraph. The method provided in the present application realizes extraction of higher-level semantic features, can more comprehensively and meticulously analyze different targets in a to-be-recognized spatial environment region image, realizes more intelligent video image understanding, improves the intelligent analysis degree of the video image, and can accurately determine the visual relationship between the targets when the targets are occluded.

[0069] In some optional embodiments, the preset target feature extraction model in step S102 includes:

[0070] In step a1, sample data is obtained, wherein each sample data includes a preset target and a scene graph in a sample image.

[0071] In step a2, the preset target in the sample image is identified and labeled to obtain the class of the preset target and the position of the preset target.

[0072] In step a3, the preset target in the sample image having the visual relationship is annotated by using a scene graph according to the category of the preset target and the position of the preset target, and the scene graph comprises the preset target, the attribute of the preset target, and the relationship between different preset targets.

[0073] In step a4, the position relationship between the preset targets is obtained by modeling the preset targets, the attribute of the preset targets, and the relationship between the matching preset targets in the image according to the visual relationship annotation.

[0074] In the embodiment of the present application, the training sample data set about target recognition and visual relationship detection in the monitoring scene is constructed, and data annotation is performed. The target in the image is annotated by using a rectangular frame tool, and the annotation category of each target in the image and the rectangular frame position P=(X, Y, W, H) describing the target position are obtained, wherein (X, Y) represents the coordinate position of the target center point, and (W, H) represents the length and width of the target rectangular frame.

[0075] The target having the visual relationship in the image is annotated by using a scene graph. The scene graph is a structural representation of the visual relationship, the target, the attribute of the target, and the relationship between the matching targets are modeled, the position relationship between the preset targets is obtained, and the semantic details in the image are captured.

[0076] A specific scene graph comprises the target, the attribute of the target, and the relationship between different targets. The target is the target in the above target recognition annotation, which is the main component in the image and comprises the category and position information. The attribute of the target is the basic attribute of the target, such as the color of the target (red coat) and the state of the target (such as a person sitting), and not all targets have attributes, and some targets can have multiple attributes. The relationship between the targets can comprise the action relationship, the spatial relationship, and the relationship described by the preposition phrase and the verb.

[0077] In the embodiment of the present application, the preset target feature extraction model can recognize the relationship features between the preset targets in the complex scene, the target recognition annotation of the preset target in the sample image is performed by using the rectangular frame tool, the visual relationship annotation of the preset target having the visual relationship in the image is performed by using the scene graph, the category of the preset target and the position of the preset target are obtained, the multiple targets and the relationship therebetween can be contained, and the semantic details in the image are captured.

[0078] In some optional embodiments, the clustering analysis of the position relationship features between the preset targets in the above step S103 comprises:

[0079] In step b1, the multi-layer convolutional neural network is used for feature extraction of the image, and the feature vector of the preset target position is extracted.

[0080] Step b2, using the region feature extraction network to extract the feature vector of the preset target position to generate preset target information, wherein the preset target information is represented by the following formula:

[0081] O i = [s i ,x i ,y i ,w i ,h i ]

[0082] O i is the original i-th target, s i is the confidence of the extracted preset target, x i , y i are the center point position coordinates of the preset target rectangular frame respectively, w i , h i are the length and width of the target respectively.

[0083] Step b3, clustering according to the position relationship between the preset target information, the preset target with overlapping relationship forms a subgraph, and a scene graph includes at least one subgraph, and each subgraph includes at least one visual relationship triple.

[0084] In the embodiment of the application, the feature extraction of the image is performed by using a multi-layer convolutional neural network, the purpose of which is to extract the features of the whole image, and the multi-layer convolutional neural network uses a backbone network based on ResNet-50 to extract features, so as to ensure that more rich and effective features are extracted. In order to generate all the target candidate frames in the image, that is, to obtain the positions of all the to-be-recognized targets in the image, the input is the above-mentioned feature vector, and the region feature extraction network (Region Proposal Network, RPN) is used to extract the target to obtain the target information.

[0085] The clustering is used to reduce the meaningless relationship between the targets and improve the running efficiency of the model. Specifically, if n targets are obtained, any two targets can have a certain relationship, that is, there are n*(n-1) potential target relationships, but in practice, the relationship between many targets does not have actual meaning. Therefore, by clustering a group of targets into a combined target, which can also be defined as a subgraph, the unnecessary association relationship between the targets is reduced.

[0086] In a specific embodiment, assuming that there is a person wearing a hat, carrying a backpack and holding a bottle of water in the image, through the target extraction module, four targets of the hat, the backpack, the bottle of water and the pedestrian will be obtained, and 4*(4-1) = 12 target visual relationships will be generated, but many of the relationships do not have actual meaning, therefore, through clustering of the targets, as a whole, the person is the subject, the hat, the backpack and the bottle of water are three objects, the relationships among the hat, the backpack and the bottle of water are cancelled, and the relationships among the meaningless targets are reduced, and the definition is as follows:

[0087] S i =[s i ,x i ,y i ,w i ,h i ]

[0088] Wherein, S i is a subgraph associated with the target O i , s i represents a confidence degree, and is the product of confidence degrees of the multiple targets that are clustered.

[0089] The position relationships among the targets are utilized to perform clustering, and as a whole, the targets having overlapping positions form a subgraph.

[0090] The application forms a subgraph for the targets having overlapping positions as a whole through clustering analysis, and different targets in the image are analyzed more comprehensively and carefully through the subgraphs.

[0091] In some optional embodiments, the visual relationship triplets are represented by the following formula:

[0092] R = <subject, relation, object>

[0093] Wherein, the subject is a subject, represents a preset target; the object represents an object, represents another preset target having a visual relationship, and the relation is the relationship between the two preset targets.

[0094] In a specific embodiment, the subject represents a subject, the object represents an object, and the relation is the relationship between the two targets, for example: the person with red hair rides an electric vehicle, wherein the person with red hair is the subject, the riding is the relationship (action), and the electric vehicle is the object, the red hair is the attribute of the subject, and the object has no attribute; for example, the bus is in front of the car, the bus is the subject, the relationship in front of is the spatial relationship, and the car is the object.

[0095] The embodiment of the application can label the image by target and scene graph, and one scene graph can contain one or more subgraphs for a relatively complex scene, each subgraph can also be composed of the above plurality of triple relationships, and the action of each target in the image is identified by the visual triple.

[0096] In some optional embodiments, the preset visual relationship prediction model in the step S104 comprises:

[0097] The features of each preset target and each subgraph are iteratively updated by the preset loss function to generate the scene graph composed of each subgraph.

[0098] The embodiment of the application can fully utilize the association relationship between the target and the subgraph by iteratively updating the features of each preset target and each subgraph, and iteratively update the features of each target and subgraph, increase the features of the target and subgraph associated with the preset target, and generate the scene graph composed of each subgraph.

[0099] In some optional embodiments, the preset loss function is:

[0100] L = β1L class + β2L bbox + β3L sg_cls

[0101] Wherein, β1, β2, β3 are weight coefficients of the target classification loss function, the target detection frame regression loss function, and the target and subgraph relationship classification loss function, respectively, L class represents the classification loss function of the target, L bbox is the regression loss function of the target detection frame, and L sg_cls is the classification loss function of the target and the subgraph.

[0102] In the embodiment of the application, the loss function includes three aspects, which are the classification loss function of the target, the detection frame regression loss function of the target, and the relationship classification loss function between the target and the subgraph, and the total loss function is defined as:

[0103] L = β1L class + β2L bbox + β3L sg_cls

[0104] Specifically, β1, β2, β3 are weight coefficients of the target classification loss function, the target detection frame regression loss function, and the target and subgraph relationship classification loss function, respectively, and can be respectively set to 0.5, 0.5, and 1.0, and can be set according to actual conditions in actual application.

[0105] Wherein, L classThe classification loss function representing the target is defined as follows, considering that the target categories to be detected and identified in the visual relationship detection task are more, covering most targets in the monitoring scene. Among these targets, some categories are common categories, such as pedestrians and vehicles, some categories have a medium frequency, such as trolleys, backpacks and pets, and some categories have a low frequency, such as badminton rackets and displays. The number of targets of different categories is not balanced, and the identification of infrequently appearing target categories is prone to inaccuracy. Therefore, a weighted softmax classification loss function is used for supervised learning of target categories, which is defined as follows:

[0106]

[0107] where C represents the number of target categories, Y represents the label corresponding to the output feature map X, n is the total number of targets, and W c represents the weight of the category, which is used to balance the weights between different categories, and is specifically defined as follows:

[0108]

[0109] where N c is the total number of samples of the category c, and B is the total number of samples in each training iteration. In order to strengthen the learning of data categories with low frequency, the sampling probability of each category is defined to be inversely proportional to the reciprocal of the number of the category, that is, the probability of being sampled is higher for the category with less number, where 1 / N c represents the reciprocal of the total number of samples of the category c, and C represents the total number of categories.

[0110] Specifically, L bbox is the regression loss function of the target detection frame, which adopts a smoothL1 loss function to supervise the model learning of the position of the target detection frame, and is defined as follows:

[0111]

[0112] where x represents the difference between the predicted value and the label value.

[0113] Specifically, the classification loss function L sg_cls of the target and the subgraph adopts a standard softmax classification loss function.

[0114] The features of each preset target and each subgraph are iteratively updated by using the preset loss function.

[0115] In some optional embodiments, the features of the iteratively updated target are represented by the following formula:

[0116]

[0117] wherein, is the feature of the i-th target after iteration, O i is the feature of the original i-th target, C (o,s)→o denotes the target associated with the target o, and the subgraph passes the feature to the target O i , m denotes m subgraphs associated with the target, and n denotes n targets associated with the target, denotes the feature of the m-th subgraph associated with the target O i , is the feature of the m-th subgraph associated with the target O the feature weight coefficient passed to the target, denotes the feature of the n-th target associated with the target O i , is the feature of the n-th target associated with the target O the feature weight coefficient passed to the target.

[0118] In a specific embodiment, by clustering the targets, the set of targets and subgraphs is obtained, for each target O i there can be multiple subgraphs S i associated therewith, for each subgraph S k there can also be multiple targets O k associated therewith, the single feature of each target and subgraph obtained in step S102 is iterated by using the association relationship between the target and the subgraph, the features of the targets and subgraphs associated therewith are increased, a scene graph composed of various subgraphs is generated, and further feature refinement and optimization of the subsequent entire scene graph are performed.

[0119] Specifically, the feature of the target after iteration is represented by the following formula:

[0120]

[0121] wherein, is the feature of the i-th target after iteration, O i is the feature of the original i-th target, C (o,s)→o denotes the target and subgraph associated with the target o, and the subgraph passes the feature to the target O i , wherein m denotes m subgraphs associated with the target, and n denotes n targets associated with the target, denotes the feature of the m-th subgraph associated with the target O i , is the feature of the m-th subgraph associated with the target O the feature weight coefficient passed to the target; denotes the feature of the n-th target associated with the target O i , For The feature weight coefficient passed to the target is iterated and updated for each target and subgraph by the above process.

[0122] A visual relationship detection apparatus is also provided in the embodiment, which is used to implement the above-described embodiments and preferred embodiments, and will not be described again. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, implementation in hardware, or a combination of software and hardware, is also possible and contemplated.

[0123] The embodiment provides a visual relationship detection apparatus, which comprises: Figure 2 as shown, comprising:

[0124] The acquisition module 201 is configured to acquire at least one image of a spatial environment region to be recognized.

[0125] The feature extraction module 202 is configured to acquire relationship features between preset targets in the image by using a preset target feature extraction model according to the acquired image, and further obtain a visual relationship triple of a scene graph in the image.

[0126] The clustering analysis module 203 is configured to perform clustering analysis on the position relationship features between the preset targets, and form a subgraph by the preset targets having an overlapping relationship. A scene graph comprises at least one subgraph, and each subgraph comprises at least one visual relationship triple.

[0127] The prediction module 204 is configured to generate a scene graph composed of the subgraphs by using a preset visual relationship prediction model according to the triple information of the preset targets and the subgraphs.

[0128] The detection module 205 is configured to obtain a visual relationship of the image to be detected according to the scene graph composed of the subgraphs.

[0129] In some optional embodiments, the preset target feature extraction model in the feature extraction module 202 comprises:

[0130] The acquisition unit is configured to acquire sample data, wherein each sample data comprises preset targets and a scene graph in a sample image.

[0131] The identification unit is configured to perform preset target identification annotation on the preset targets in the sample image to obtain a category of the preset targets and a position of the preset targets.

[0132] The labeling unit is configured to label visual relationships of preset targets in the sample image by using a scene graph according to categories of the preset targets and positions of the preset targets, the scene graph including the preset targets, attributes of the preset targets, and relationships between different preset targets.

[0133] The generating unit is configured to obtain position relationships between the preset targets by modeling the preset targets, attributes of the preset targets, and relationships between matching preset targets in the image according to the visual relationship labeling.

[0134] In some optional embodiments, the clustering analysis module 203 performs clustering analysis on the position relationship features between the preset targets, including:

[0135] The feature extraction unit is configured to extract a feature vector of the position of the preset target by using a multi-layer convolutional neural network to perform feature extraction on the image.

[0136] The target extraction unit is configured to perform preset target extraction on the feature vector of the position of the preset target by using a region feature extraction network to generate preset target information, wherein the preset target information is represented by the following formula:

[0137] O i =[s i ,x i ,y i ,w i ,h i ]

[0138] O i is an original ith target, s i is a confidence degree of the extracted preset target, x i and y i are respectively center point position coordinates of the preset target rectangular frame, w i and h i are respectively a length and a width of the target.

[0139] The target clustering unit is configured to cluster according to position relationships between the preset target information, and preset targets having overlapping positions form a subgraph, and a scene graph includes at least one subgraph, and each subgraph includes at least one visual relationship triple.

[0140] Further function descriptions of the above various modules and units are the same as those of the above corresponding embodiments, and will not be described here again.

[0141] In this embodiment, the visual relationship detection device is presented in the form of a functional unit. Here, a unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0142] This invention also provides a computer device having the above-described features. Figure 2 The visual relationship detection device shown.

[0143] Please see Figure 3 , Figure 3 This is a schematic diagram of the structure of a computer device provided in an optional embodiment of the present invention, such as... Figure 3 As shown, the computer device includes one or more processors 10, memory 20, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components communicate with each other via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the computer device, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interfaces). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple computer devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 3 Take a processor 10 as an example.

[0144] Processor 10 may be a central processing unit, a network processor, or a combination thereof. Processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GDA), or any combination thereof.

[0145] The memory 20 stores instructions executable by at least one processor 10 to cause the at least one processor 10 to perform the method shown in the above embodiments.

[0146] The memory 20 can include a program storage area and a data storage area. The program storage area can store an operating system, application programs required for at least one function, and the like. The data storage area can store data created according to the use of the computer device, and the like. In addition, the memory 20 can include a high-speed random access memory, and can further include a non-transitory memory such as at least one of a magnetic disk storage device, a flash memory device, or other non-transitory solid state memory device. In some alternative embodiments, the memory 20 can optionally include a memory disposed remotely from the processor 10, which can be connected to the computer device through a network. Examples of the network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0147] The memory 20 can include a volatile memory such as a random access memory, and can further include a non-volatile memory such as a flash memory, a hard disk, or a solid state disk, and a combination thereof.

[0148] The computer device further includes a communication interface 30 for communication of the computer device with other devices or communication networks.

[0149] The embodiments of the present application also provide a computer readable storage medium. The method according to the embodiments of the present application can be implemented in hardware, firmware, or as software code to be recorded in a storage medium, or originally stored in a remote storage medium or a non-transitory machine readable storage medium to be downloaded through a network and stored in a local storage medium, so that the method described herein can be processed by such software using a general purpose computer, a special purpose processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, an optical disk, a read-only memory, a random access memory, a flash memory, a hard disk, or a solid state disk, and the like. Further, the storage medium can include a combination of the above-mentioned storage media. It can be understood that the computer, the processor, the microprocessor controller, or the programmable hardware includes a storage component that can store or receive software or computer code, which, when accessed and executed by the computer, the processor, or the hardware, implements the method shown in the above embodiments.

[0150] Although the embodiments of the present application have been described with reference to the accompanying drawings, various modifications and changes can be suggested to one skilled in the art, and it is intended that the present application encompass such modifications and changes as fall within the scope of the appended claims.

Claims

1. A method of visual relationship detection, the method comprising: The method comprises: acquiring at least one image of a spatial environment region to be recognized; according to the acquired image, using a preset target feature extraction model to acquire the relationship features between preset targets in the image, and then obtaining the visual relationship triplets of the scene graph in the image; performing clustering analysis on the position relationship features between the preset targets, and forming a subgraph with the preset target positions having an overlapping relationship; one scene graph comprises at least one subgraph, and each subgraph comprises at least one visual relationship triplet; according to the acquired triplet information of the preset targets and the subgraphs, using a preset visual relationship prediction model to generate a scene graph composed of the subgraphs; according to the scene graph composed of the subgraphs, obtaining the visual relationship of the image to be detected; the preset target feature extraction model comprises: acquiring sample data, wherein each sample data comprises preset targets and a scene graph in a sample image; performing preset target recognition labeling on the preset targets in the sample image to obtain the category of the preset targets and the positions of the preset targets; according to the category of the preset targets and the positions of the preset targets, performing visual relationship labeling on the preset targets having visual relationships in the sample image by using a scene graph, wherein the scene graph comprises: preset targets, attributes of the preset targets, and relationships between different preset targets; according to the visual relationship labeling, modeling the preset targets, the attributes of the preset targets, and the relationships between the matching preset targets in the image to obtain the position relationships between the preset targets; the preset visual relationship prediction model comprises: iteratively updating the features of each preset target and each subgraph by using a preset loss function to generate a scene graph composed of the subgraphs; The default loss function is: in, These are the weight coefficients of the target classification loss function, the target detection box regression loss function, and the target and subgraph relationship classification loss function, respectively. The classification loss function representing the target. Let be the regression loss function for the target detection bounding box. Let be the classification loss function for the target and the subgraph; the features of the target after iteration are represented by the following formula: in, For the iteration of the th i Characteristics of each target For the original first i Characteristics of each target Indicates the relationship with the target o The associated target, and the subgraph passed to that target Features This indicates that the following are associated with the target: m Sub-image, n This indicates that the following are associated with the target: n One goal, Representation and target The associated first m Features of subgraphs The feature weight coefficients passed to the target. Representation and target The associated first n Characteristics of each target for The feature weight coefficients passed to the target.

2. The method of claim 1, wherein, the clustering analysis on the position relationship features between the preset targets comprises: extracting feature vectors of the positions of the preset targets by using a multi-layer convolutional neural network to extract features of the image; A region feature extraction network is used to extract the feature vector of a preset target location to generate preset target information, which is represented by the following formula: For the original first i One goal, The confidence level for extracting the preset target, These are the coordinates of the center point of the preset target rectangle. These are the length and width of the target, respectively; performing clustering according to the position relationships between the preset target information, and forming a subgraph with the positions of the preset targets having an overlapping relationship; one scene graph comprises at least one subgraph, and each subgraph comprises at least one visual relationship triplet.

3. The method of claim 2, wherein, The visual relationship triple is expressed by the following formula: Wherein, the subject is a subject, indicating a preset target; the object indicates another preset target with a visual relationship, and the relation is the relationship between the two preset targets.

4. A visual relationship detection apparatus for implementing the method of claim 1, characterized by The device comprises: an acquisition module configured to acquire at least one image of a spatial environment region to be recognized; a feature extraction module configured to acquire, according to the acquired image, relationship features between preset targets in the image by using a preset target feature extraction model, and then obtain visual relationship triplets of a scene graph in the image; a clustering analysis module configured to perform clustering analysis on position relationship features between the preset targets, and form a subgraph with the preset target positions having an overlapping relationship; one scene graph comprises at least one subgraph, and each subgraph comprises at least one visual relationship triplet; a prediction module configured to generate a scene graph composed of the subgraphs by using a preset visual relationship prediction model according to the acquired triplet information of the preset targets and the subgraphs; a detection module configured to obtain a visual relationship of an image to be detected according to the scene graph composed of the subgraphs.

5. A computer device, comprising: comprise: A memory and a processor, which are connected in communication with each other, the memory storing computer instructions, and the processor executing the computer instructions to perform the visual relationship detection method of any one of claims 1 to 3.

6. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions for causing a computer to perform the visual relationship detection method of any one of claims 1 to 3.

Images