Method, device and storage medium for identifying positional relationships of objects in an image

Abstract

The present invention relates to the field of image recognition technology, and the present invention provides a method, device and storage medium for identifying the position relationship of objects in an image. The recognition method is to obtain an RGB image and a depth image containing multiple objects to be identified; the depth image contains depth information corresponding to each pixel in the RGB image; the RGB image is detected and feature extracted to obtain visual features and semantic features corresponding to multiple target images; the center depth of the target object of each target image is determined by clustering the depth of each pixel in the depth image, so as to determine the depth difference between any two target objects, and finally the visual features, semantic features corresponding to each target image and the depth difference between any two target objects are input into a trained position relationship classification model, so as to output the position results of any two target objects. The object position relationship recognition method provided by the present application has the characteristic of high accuracy in identifying position results.

Description

Technical Field

[0001] The present invention relates to the field of image recognition technology, and in particular to a method, device and storage medium for recognizing the positional relationship of objects in an image. Background Art

[0002] With the maturity of technologies like object detection and instance segmentation, computer vision has become relatively accurate in detecting objects in images. To further enhance its intelligence, a deeper understanding of the relationships between objects in an image is required, known as scene graph generation. A typical application scenario involves first detecting the objects contained in an image. Then, given two randomly selected objects, the relationship between them must be inferred. Objects are typically represented by nouns, forming a "subject-verb-object" structure, where the predicate embodies the relationship between the subject and the object. For example, in the sentence "a person sits on a chair," "sitting" represents the relationship between "person" and "chair."

[0003] The positional relationship between objects is a very important type of inter-object relationship. For example, in a football match, the fore-aft relationship between the ball and the goal line determines whether a goal is scored, while the fore-aft relationship between the attacker and defender can influence offside calls. Furthermore, due to viewing angles, it is sometimes difficult to accurately determine the fore-aft relationship between the ball and the goal line, or between the attacker and defender, from a single image. This means that images containing only RGB information are insufficient for understanding spatial positional relationships. Existing techniques based solely on RGB information cannot accurately identify the positional relationships of objects in an image. Summary of the Invention

[0004] The present invention aims to solve the technical problem in the prior art that the position relationship of objects in an image cannot be accurately identified.

[0005] To solve the above technical problems, the present application discloses, on one hand, a method for identifying the position relationship of objects in an image, which includes:

[0006] Obtain an RGB image and a depth image containing multiple objects to be identified; the depth image contains depth information corresponding to each pixel in the RGB image;

[0007] Performing detection processing on the RGB image to obtain a target image set; the target image set includes a plurality of target images; each of the plurality of target images includes a target object, which is the identified object to be identified;

[0008] For each target image in the target image set, a feature extraction operation is performed on the target image to obtain visual features and semantic features;

[0009] Clustering the depth information of multiple pixels in the target image to obtain a target cluster set;

[0010] Determining a center depth of a target object corresponding to the target cluster set based on the depths of a plurality of pixel points in the target cluster set;

[0011] The position result between any two target objects in the plurality of target objects is determined based on the visual features, semantic features and center depth of each target image in the target image set; the position result represents the relative position relationship between the two target objects.

[0012] Optionally, clustering is performed on depth information of multiple pixels in the target image to obtain a target cluster set, including:

[0013] Determine a target depth map from the depth map based on the pixel correspondence;

[0014] Clustering is performed on depth information of multiple pixels included in the target depth map to obtain the target cluster set.

[0015] Optionally, clustering the depth information of multiple pixels included in the target depth map to obtain the target cluster set includes:

[0016] Clustering the depths of multiple pixels included in the target depth map to obtain a cluster set; the cluster set includes multiple sub-cluster sets; each cluster set in the multiple sub-cluster sets corresponds to an object in the target image;

[0017] Obtaining the coordinates of each pixel in each sub-cluster;

[0018] Determine the placeholder value corresponding to each sub-cluster set according to the coordinates of each pixel point in each sub-cluster set;

[0019] The sub-cluster set with the largest occupancy value is determined as the target cluster set.

[0020] Optionally, determining the center depth of the target object corresponding to the target cluster set based on the depths of the plurality of pixels in the target cluster set includes:

[0021] Determine a first depth set and a second depth set from the target cluster set; a value of any depth in the first depth set is smaller than a value of any depth in the second depth set;

[0022] determining a first target depth based on a plurality of depths in the first depth set;

[0023] determining a second target depth based on a plurality of depths in the second depth set;

[0024] The center depth of the target object corresponding to the target cluster set is determined according to the first target depth and the second target depth.

[0025] Optionally, determining the first depth set and the second depth set from the target cluster set includes:

[0026] Sorting the depths of the multiple pixels in the target cluster set from small to large to obtain a sorted target cluster set; the sorted target cluster set includes the depth of each pixel in the multiple pixels and the corresponding arrangement number;

[0027] Determine a depth set formed by depths that meet a first preset condition in the sorted target cluster set as a first depth set;

[0028] A depth set formed by depths satisfying a second preset condition in the sorted target cluster set is determined as a second depth set.

[0029] Optionally, the first preset condition is: the arrangement sequence number corresponding to the depth is less than or equal to a first threshold;

[0030] The second preset condition is that the arrangement sequence number corresponding to the depth is greater than or equal to a second threshold.

[0031] Optionally, the determining of the position result between any two target objects among the plurality of target objects according to the visual features, semantic features and center depths of the target objects corresponding to the respective target images in the target image set includes:

[0032] Obtaining a depth difference set according to a difference in center depths of target objects corresponding to any two target images in the target image set;

[0033] The trained position relationship classification model is used to process the depth difference set, the visual features and the semantic features corresponding to each target image in the target image set to obtain the position result between any two target objects among the multiple target objects.

[0034] In another aspect, the present application further discloses a device for identifying positional relationships of objects, comprising:

[0035] An image acquisition module is used to acquire an RGB image and a depth image containing multiple objects to be identified; the depth image contains depth information corresponding to each pixel in the RGB image;

[0036] A target image set acquisition module is used to perform detection processing on the RGB image to obtain a target image set; the target image set includes multiple target images; each target image in the multiple target images contains a target object, and the target object is the identified object to be identified;

[0037] A feature extraction module is used to perform a feature extraction operation on each target image in the target image set to obtain visual features and semantic features;

[0038] A clustering processing module is used to perform clustering processing on the depth information of multiple pixels in the target image to obtain a target cluster set;

[0039] A center depth determination module is used to determine the center depth of the target object corresponding to the target cluster set based on the depths of multiple pixels in the target cluster set;

[0040] A position result determination module is used to determine the position result between any two target objects in a plurality of target objects based on the visual features, semantic features and center depth of each target image in the target image set; the position result represents the relative position relationship between the two target objects.

[0041] On the other hand, the present application also discloses an electronic device, which includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the identification method as described above.

[0042] On the other hand, the present application also discloses a computer storage medium, which stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by a processor to implement the above-mentioned identification method.

[0043] By adopting the above technical solution, the method for identifying the position relationship of objects in an image provided by this application has the following beneficial effects:

[0044] The recognition method is to obtain an RGB image and a depth image containing multiple objects to be identified; the depth image contains depth information corresponding to each pixel in the RGB image; and the RGB image is detected and feature extracted, so as to obtain semantic features corresponding to multiple target images; subsequently, by clustering the depth of each pixel in the depth image, the center depth of the target object of each target image can be determined to determine the depth difference between any two target objects, and finally each of the above target images and the corresponding semantic features and the depth difference between any two target objects are input into the trained position relationship classification model, and the position results of any two target objects can be output, such as "a person sitting on a chair". Since the depth information of the two target objects is integrated into the recognition process, the accuracy of the position result recognition is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0046] Figure 1 An optional application scenario diagram for this application;

[0047] Figure 2 This is a flowchart of the first optional identification method of this application;

[0048] Figure 3 This is a flow chart of the second optional identification method of this application

[0049] Figure 4 An optional position relationship diagram of an image and a coordinate system for this application;

[0050] Figure 5 This is a flowchart of the third optional identification method of this application;

[0051] Figure 6 A schematic diagram of an optional image for this application;

[0052] Figure 7 This is a schematic structural diagram of an optional device for identifying object position relationships in the present application.

[0053] The following is a supplementary description of the accompanying drawings:

[0054] 10-Target detection module; 20-Semantic segmentation model; 30-Position relationship recognition module. DETAILED DESCRIPTION

[0055] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making any creative work are within the scope of protection of this application.

[0056] It should be noted that the terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

[0057] like Figure 1 As shown, Figure 1 This is an optional application scenario diagram for this application. The scenario includes a target detection module 10, a semantic segmentation model 20, and a position relationship recognition module 30; the target detection module 10 is used to perform detection processing on the acquired RGB image to obtain an image containing multiple detection frames; each detection frame contains a target object, which is the object to be recognized after recognition; it is also used to extract the image corresponding to the detection frame in the image containing the detection frame to obtain a target image set; the target image set includes multiple target images; each target image in the multiple target images corresponds to a detection frame; the semantic segmentation model 30 is used to perform feature extraction operations on each target image in the target image set to obtain visual features. features and semantic features; the position relationship recognition module 40 is used to obtain a depth map, which contains depth information corresponding to each pixel in the RGB image; for each target image in the target image set, the depth information of multiple pixels in the target image is clustered to obtain a target cluster set, and the center depth of the target object corresponding to the target cluster set is determined based on the depth of the multiple pixels in the target cluster set, and then the position result between any two target objects in the multiple target objects is determined according to the visual features, semantic features and center depth of the target object corresponding to each target image in the target image set; the position result represents the relative position relationship between the two target objects.

[0058] Optionally, the position relationship identification module 40 includes a clustering processing module and a position relationship classification model; the clustering processing module is used to determine the center depth of the target object in each of the above target images based on the depth map; the position relationship classification model is used to determine the position result between any two target objects.

[0059] In the embodiment of the present application, the target detection module 10, the semantic segmentation model 20, and the position relationship identification module 30 can be set in the same device, such as a mobile terminal, a computer terminal, a server, or a similar computing device; optionally, the target detection module 10, the semantic segmentation model 20, and the position relationship identification module 40 can be set in multiple devices, and the multiple devices are in a system; optionally, the target detection module 10, the semantic segmentation model 20, and the position relationship identification module 30 can be set on a platform. Therefore, the execution subject of the embodiment of the present application can be a mobile terminal, a computer terminal, a server, or a similar computing device; it can be a system or a platform.

[0060] Optionally, the terminal may be a physical device such as a desktop computer, a laptop computer, a mobile phone, a tablet computer, a digital assistant, or a smart wearable device; wherein the smart wearable device may include a smart bracelet, a smart watch, smart glasses, a smart helmet, etc., but is not limited thereto.

[0061] The following describes a specific embodiment of an identification method of the present application. Figure 2 This is a flowchart of the first optional identification method of the present application. This specification provides method operation steps such as the embodiments or flowcharts, but may include more or fewer operation steps based on conventional or non-creative work. The order of steps listed in the embodiments is only one way of executing the steps among many orders, and does not represent the only order of execution. When the actual system or server product is executed, it can be executed in sequence or in parallel (for example, in a parallel processor or multi-threaded processing environment) according to the method shown in the embodiments or the drawings. Specifically, Figure 2 As shown, the method may include:

[0062] S201: Acquire an RGB image and a depth image containing a plurality of objects to be identified; the depth image contains depth information corresponding to each pixel in the RGB image.

[0063] In the embodiment of the present application, an RGB-D image can be directly acquired through a depth camera to obtain an RGB image and a depth map. The RGB image and the depth map have the same size, and each pixel in the RGB image has a one-to-one correspondence with each pixel in the depth map. The depth map contains the surface distance information (i.e., depth information) between the viewpoint and the scene object. Alternatively, an RGB image can be acquired through a binocular camera, and the depth map corresponding to each frame of the RGB image can be calculated.

[0064] S202: Detect and process the RGB image to obtain a target image set; the target image set includes multiple target images; each target image in the multiple target images includes a target object, and the target object is the identified object to be identified.

[0065] Optionally, step S202 can be specifically expressed as: performing detection processing on the acquired RGB image to obtain an image containing multiple detection frames; each detection frame contains a target object, which is the identified object to be identified; performing an extraction operation on the image corresponding to the detection frame in the image containing the detection frame to obtain a target image set; the target image set includes multiple target images; each target image in the multiple target images corresponds to a detection frame.

[0066] In this embodiment, the target objects in the RGB image can be annotated manually to determine an image containing multiple detection frames. To improve detection efficiency, the target detection model can be used to perform detection processing on the RGB image to obtain images containing detection frames and category results of objects within the detection frames. The images within the detection frames can then be extracted to obtain the target image set.

[0067] It should be noted that, in addition to the target object, the target image also contains non-target objects, where the target object can be a person, car, cat, dog, chair, etc. The non-target object can also be called a background object, which can be the sky, buildings, houses, roads, and zebra crossings, etc.; the non-target object also includes foreground objects, which will cause certain occlusion to the target object, and the foreground object can be the above-mentioned person, car, cat, dog, building, house, etc.

[0068] Using target detection models to detect target objects is a relatively mature technology, such as FasterRCNN, SSD and YOLO models.

[0069] In this embodiment, in order to further simplify the subsequent processing steps of the depth information corresponding to the target image, step S202 can also be specifically explained as: inputting the RGB image into the instance segmentation network model to obtain the target image set.

[0070] Optionally, the instance segmentation network model can provide different labels for separate instances of objects belonging to the same class, thereby directly obtaining more accurate segmentation results.

[0071] For example, the RGB image contains two cats. By inputting the RGB image into the instance segmentation network model, the classification result image containing different labels of cat 1 and cat 2 can be directly output.

[0072] Optionally, the instance segmentation network model can be an unsupervised semantic segmentation model.

[0073] S203: For each target image in the target image set, perform a feature extraction operation on the target image to obtain visual features and semantic features.

[0074] In this embodiment, step S203 can be specifically expressed as: inputting the target image set into the semantic segmentation model to obtain the visual features and semantic features corresponding to each target image in the target image set.

[0075] Specifically, the semantic segmentation model can obtain the corresponding visual features by performing feature extraction operations on the target image, and classify each pixel point, so as to obtain a semantic feature map after classifying each pixel point, thereby determining the classification result of the target object and obtaining the semantic features corresponding to the target image, such as the semantics of people, cars, cats, dogs, chairs, etc.

[0076] Optionally, the semantic segmentation model can be obtained by implementing feature extraction based on a residual network (ResNet) or a VGG network.

[0077] S204: performing clustering processing on depth information of multiple pixels in the target image to obtain a target cluster set.

[0078] Since the target image corresponding to the detection frame may contain not only the target object but also foreground and background objects, the depth information of the background and foreground objects will affect the accuracy of the depth information acquisition of the object. Figure 3 , Figure 3 This is a flow chart of the second optional identification method of this application. Step S204 can be specifically described as follows:

[0079] S2041: Determine a target depth map from the depth map based on the pixel correspondence relationship;

[0080] S2042: Clustering the depth information of multiple pixels included in the target depth map to obtain the target cluster set.

[0081] In a feasible embodiment, step S2042 can be specifically expressed as follows: clustering the depths of multiple pixels contained in the target depth map to obtain a cluster set; the cluster set contains multiple sub-clusters; each cluster set in the multiple sub-clusters corresponds to an object in the target image; obtaining the coordinates of each pixel in each sub-cluster set; determining the occupancy value corresponding to each sub-cluster set based on the coordinates of each pixel in each sub-cluster set; determining the sub-cluster set with the largest occupancy value as the target cluster set. In this way, the depth information of the target object corresponding to the detection frame can be determined.

[0082] Optionally, the DBSCAN density clustering analysis method can be used to analyze the depth information of the object and classify and aggregate the pixels with reachable depth density. This method is based on a set of neighborhood parameters (Eps, MinPts) to characterize whether the depth information is closely connected, where Eps represents the maximum radius of the neighborhood and MinPts represents the minimum number of samples in the Eps neighborhood. That is, for two pixel points r belonging to the depth map Depth i With r j , the neighborhood of Eps is N Eps (r i )={r j ∈Ddist(r i ,r j )≤Eps}, where dist(r i ,r j ) is r in the deep feature space i With r j The distance between the two points is the depth difference.

[0083] In this process, the parameters Eps and MinPts satisfy the following formula:

[0084]

[0085] Among them, W B ,H B They refer to the width and length of the detection box B respectively, α is the coefficient, α=10, MinPts=12.

[0086] Optionally, a MEAN SHIF (mean shift algorithm) may be used to perform clustering processing on the object depth information.

[0087] In this embodiment, the method for determining the placeholder value corresponding to each sub-cluster set may specifically include: for each sub-cluster set in the cluster set, determining a target pixel point coordinate set from the coordinate set corresponding to the sub-cluster set; the target pixel point coordinate set includes the coordinates of the first pixel point, the coordinates of the second pixel point, the coordinates of the third pixel point and the coordinates of the fourth pixel point; the area formed by connecting the vertices corresponding to the first pixel point, the second pixel point, the third pixel point and the fourth pixel point in sequence can contain all the pixels in the sub-cluster set; the coordinate set contains the coordinates of each pixel point in the corresponding sub-cluster set; and determining the placeholder value of the sub-cluster set based on the coordinates of the first pixel point, the coordinates of the second pixel point, the coordinates of the third pixel point and the coordinates of the fourth pixel point.

[0088] For example, see Figure 4 , Figure 4 This is an optional position relationship diagram of an image and a coordinate system for this application. Figure 4The gray area in is the area corresponding to the subcluster set. The coordinates of the first pixel point are P1(x1,y1), the coordinates of the second pixel point are P2(x2,y2), the coordinates of the third pixel point are P3(x3,y3), and the coordinates of the fourth pixel point are P4(x4,y4). Since the size of each pixel point is certain, assuming it is a square with a width of a, the coordinates of the vertex farthest from the center point of the subcluster set can be determined based on the coordinates of the above four pixel points. That is, the coordinates of the four vertices of the area corresponding to the subcluster set can be obtained; the four vertices are A(x1-a,y1-a), B(x2,y2-a), C(x3,y3), and D(x4-a,y4). Based on the coordinates of the above four fixed points, the occupancy value of the corresponding area can be determined.

[0089] Optionally, in order to improve the efficiency of calculating the placeholder value of the area corresponding to each sub-cluster set, continuing the above example, assume that the cluster set contains n sub-cluster sets, which can be expressed as cluster set R = {R1, R2, ..., Rn}, n is an integer greater than 1, Ri (0 < i ≦ n) represents the i-th sub-cluster set, assuming that the corresponding initial area is a rectangle, then the four vertices of the initial area are A1 (W min ,H min ), B1(W min ,H max ), C1(W max ,H max ), D1(W max ,H min ), where W min ,W max ,H min ,H max The minimum horizontal axis value, maximum horizontal axis value, minimum vertical axis value, and maximum vertical axis value of the pixels in the region are respectively calculated. The initial region and the cluster set are intersected, and the region corresponding to the intersection is determined as the target region. The placeholder value corresponding to the target region is used to determine the placeholder value of the region corresponding to the sub-cluster set. In this way, the placeholder value of the region corresponding to each sub-cluster set can be solved in sequence.

[0090] Optionally, since there is a one-to-one correspondence between each pixel in the depth map and each pixel in the GRB map, when the above-mentioned instance segmentation network model is used to perform target detection on the GRB map, a target image set can be directly obtained, in which each target image in the target image set only contains each pixel of the target object, but does not contain non-target objects; then step S204 can be specifically elaborated as follows: based on the pixel correspondence, determine the target depth map from the depth map; determine the set formed by the depth information of the pixels corresponding to the target depth map as the target cluster set. This simplifies the step of determining the target cluster set of the target object corresponding to each target image.

[0091] S205: Determine the center depth of the target object corresponding to the target cluster based on the depths of the multiple pixels in the target cluster.

[0092] In order to further solve the problem that the depth of the object is approximated based on the average value of all pixels in the target cluster, the depth of the object determined based on this method will be quite different from the center depth of the object due to reasons such as the object volume and the observation angle. It only considers the surface depth and ignores the influence of the length direction of the object on the center depth of the object. In a feasible embodiment, referring to FIG. Figure 5 , Figure 5 This is a flowchart of the third optional identification method of this application. Step S205 can be specifically described as follows:

[0093] S2051: Determine a first depth set and a second depth set from the target cluster set; a value of any depth in the first depth set is smaller than a value of any depth in the second depth set.

[0094] In a feasible embodiment, step S2051 can be specifically expressed as: sorting the depths of multiple pixel points in the target cluster set from small to large to obtain a sorted target cluster set; the sorted target cluster set includes the depth of each pixel point in the multiple pixel points and the corresponding arrangement number; the depth set formed by the depths in the sorted target cluster set that meet the first preset condition is determined as the first depth set; the depth set formed by the depths in the sorted target cluster set that meet the second preset condition is determined as the second depth set.

[0095] In a feasible embodiment, the first preset condition is that the arrangement number corresponding to the depth is less than or equal to a first threshold; the second preset condition is that the arrangement number corresponding to the depth is greater than or equal to a second threshold.

[0096] For example, the target cluster set of target object X is Where N is the number of pixels in the target cluster set, and N is an integer greater than or equal to 1. Arrange all depths in ascending order into a sequence d1', d1', ..., d' N , where 1, 2, ... N are the order numbers corresponding to the depths. The smaller the order number, the smaller the corresponding depth value. Optionally, the first threshold can be set to M, where M is an integer greater than 0 and less than N; the second threshold can be set to K, where K is an integer greater than M and less than N. Then the first depth set can be determined to be {d1', d1', ..., d' M}, the second depth set is {d' k , d' k+1 ,……,d' N}.

[0097] Optionally, the first threshold may also be the same as the second threshold.

[0098] Optionally, in order to improve the application flexibility of the method. The method for determining the first depth set and the second depth set includes: sorting the depths of multiple pixel points in the target cluster set from small to large to obtain a sorted target cluster set; obtaining a preset constant, which is greater than 0 and less than 0.5, and determining the target quantity based on the preset constant and the number of pixel points in the target cluster set; determining the depth set formed by the depth of the target quantity closest to the minimum depth as the first depth set; and determining the depth set formed by the depth of the target quantity closest to the maximum depth as the second depth set.

[0099] For example, continuing the above example, all depths are arranged in ascending order into a sequence d1', d1', ..., d' N , the number of targets M = [P × N], where [] represents an integer, then the first depth set is {d1', d1', ..., d' M}, the second depth set is {d' N-M+1 , d' N-M+2 ,……,d' N In fact, the number of the first depth set and the number of the second depth set may be different.

[0100] Optionally, the preset constant P may be 1%, 3%, 5%, etc. This can ensure the accuracy of the first target depth and the second target depth described below, thereby ensuring the accuracy of determining the depth of the target object.

[0101] S2052: Determine a first target depth based on multiple depths in the first depth set.

[0102] In this embodiment, continuing the above example, the first target depth d min It can be determined using the following formula:

[0103]

[0104] S2053: Determine a second target depth based on the multiple depths in the second depth set.

[0105] In this embodiment, continuing the above example, the second target depth d max It can be determined using the following formula:

[0106]

[0107] S2054: Determine the center depth of the target object corresponding to the target cluster set according to the first target depth and the second target depth.

[0108] In this embodiment, the center depth of the target object X may be determined based on the following formula:

[0109]

[0110] See Figure 6 , Figure 6 This is a schematic diagram of an optional image for this application. This application uses this solution to approximate the center depth of the object by using the maximum and minimum depth difference information of the same object. Compared with the prior art for objects with relatively regular shapes, such as cuboids, cylinders, cones, etc. and their combinations, the center position of the object is calculated by measuring the shape of the outer surface of the object. For objects with irregular shapes, it is usually necessary to model the shape of the object in detail and then use the method of multiple integral solution. This method has the advantage of simple calculation and can also avoid the depth feature extraction error caused by averaging the depth information of the object surface.

[0111] S206: Determine a position result between any two target objects in the plurality of target objects based on the visual features, semantic features and center depths of the target objects corresponding to each target image in the target image set; the position result represents a relative positional relationship between the two target objects.

[0112] In a feasible embodiment, step S206 can be specifically explained as: obtaining a depth difference set based on the difference in center depth of the target objects corresponding to any two target images in the target image set; using the trained position relationship classification model to process the depth difference set and the visual features and semantic features corresponding to each target image in the target image set to obtain the position result between any two target objects among the multiple target objects.

[0113] In the embodiment, the center depth of the target object Y is determined by using the above steps S2051-S2054. Thus, the difference d in the center depth of the target object X relative to the target object Y can be further determined. X→Y ,

[0114]

[0115] Optionally, the basic architecture of the positional relationship classification model can be a DRNet (Deep Relational Networks), and the core reasoning process of DRNet is a network based on conditional random fields.

[0116] Optionally, when there are more than n objects in the image, n×(n-1) pairs of objects can be formed. In order to improve the recognition efficiency of the image, the above-mentioned multiple pairs of objects can be screened to eliminate a pair of objects with little correlation and avoid analyzing the pair of objects. For example, a pair of objects whose relative depth difference is greater than a preset threshold can be eliminated. A pair of objects with a high correlation can be objects with a certain overlap in the images.

[0117] In this embodiment, the position result output by the position relationship classification model is a description of a pair of objects in the image. The output can be a triple (s, r, o), which respectively represents the s subject (object 1), r predicate (relationship) and o object (object 2) of the pair of objects. Figure 6 The image shown in FIG2 is a pictorial representation of an object, and the position result corresponding to the image is (object X, in front, object Y), which means that object X is in front of object Y.

[0118] Optionally, the position result output by the position relationship classification model may also be processed text, for example, object X is located in front of object Y.

[0119] Optionally, the position relationship classification model may be a trained position relationship classification model. Optionally, the position relationship classification model may be trained using the following steps:

[0120] Step 1: Obtain a training sample set; the training sample set includes multiple training visual features, semantic features corresponding to each training visual feature, depth differences between each pair of target objects, and annotated position relationship information of each pair of target objects; each training visual feature corresponds to a target image set; each target image set includes multiple target images; and each target image in the multiple target images corresponds to a target object;

[0121] Step 2: Construct a preset machine learning model and determine the preset machine learning model as the current machine learning model;

[0122] Step 3: Based on the current machine learning model, extract each training visual feature and the corresponding semantic feature to obtain the first feature atlas;

[0123] Step 4: Jointly estimate the depth difference between the first feature atlas and each pair of target objects to obtain the predicted position relationship information of each pair of target objects in each visual feature;

[0124] Step 5: determining a loss value based on the labeled position relationship information of each pair of target objects and the predicted position relationship information;

[0125] Step 6: When the loss value is greater than a preset threshold, back propagation is performed based on the loss value, the current machine learning model is updated to obtain an updated machine learning model, and the updated machine learning model is re-determined as the current machine learning model; repeating steps: based on the current machine learning model, a first feature atlas is obtained by performing feature extraction processing on each training visual feature and the corresponding semantic feature; the first feature atlas and the depth difference of each pair of target objects are jointly estimated to obtain predicted position relationship information of each pair of target objects in each visual feature;

[0126] When the loss value is less than or equal to the preset threshold, the current machine learning model is determined as a position relationship classification model.

[0127] In another optional embodiment, the positional relationship classification model can be an untrained positional relationship classification model. That is, a positional relationship classification model in the positional relationship classification model training process described above. By adding a depth information input branch based on the existing structure of DRNet, the feature extraction sub-model and the positional relationship joint estimation module in the positional relationship classification model can be guided to learn the positional relationship of each pair of target objects, which can constrain the positional relationship classification model as a whole and improve the accuracy of the positional relationship classification model.

[0128] Optionally, in order to better reflect the beneficial effects of the present application, the following is a specific example to illustrate the improved part of the above-mentioned position relationship classification model. The original DRNet input features include 256-dimensional visual features, 256-dimensional bounding box features (corresponding to the detection box), and 600-dimensional semantic features. By adding 2-dimensional depth features (corresponding to the depth difference), the 256-dimensional visual features, 256-dimensional bounding box features and 2-dimensional depth features are connected and transformed through a fully connected network so that the output is still 512 dimensions. (The input of the original DRNet is 512 dimensions, and the output is 512 dimensions). Subsequently, the 512 features and the above-mentioned 600-dimensional semantic features can be processed based on the relevant structure of the original DRNet, so that the corresponding position results of each pair of target objects can be obtained. The input of the depth feature can effectively improve the accuracy of the position relationship classification model in determining the position relationship between any two pairs of target objects.

[0129] We applied the method presented in this application to a specific spatial relationship detection task to verify its effectiveness, yielding the following results. The test scenario was the Rel 3D dataset. The performance metric was the average recognition accuracy of 30 spatial relationships.

[0130] This application DRNet Spatial Vip-CNN VTransE PPR-FCN 77.79％ 74.30％ 74.55％ 72.65％ 73.79％ 74.11％

[0131] It can be seen from the data in the above table that the detection of positional relationships in images based on the method provided by this application has the advantage of high detection accuracy.

[0132] On the other hand, see Figure 7 , Figure 7 This is a schematic diagram of the structure of an optional device for identifying the position relationship of objects in the present application. The present application also discloses a device for identifying the position relationship of objects, which includes:

[0133] The image acquisition module 701 is used to acquire an RGB image and a depth image containing multiple objects to be identified; the depth image contains depth information corresponding to each pixel in the RGB image;

[0134] The target image set acquisition module 702 is configured to perform detection processing on the RGB image to obtain a target image set; the target image set includes a plurality of target images; each of the plurality of target images includes a target object, which is the identified object to be identified;

[0135] A feature extraction module 703 is configured to perform a feature extraction operation on each target image in the target image set to obtain visual features and semantic features;

[0136] A clustering processing module 704 is used to perform clustering processing on the depth information of multiple pixels in the target image to obtain a target cluster set;

[0137] A center depth determination module 705 is configured to determine a center depth of a target object corresponding to the target cluster based on the depths of a plurality of pixels in the target cluster;

[0138] The position result determination module 706 is used to determine the position result between any two target objects in multiple target objects based on the visual features, semantic features and center depth of each target image in the target image set; the position result represents the relative position relationship between the two target objects.

[0139] In a feasible embodiment, the clustering processing module is used to determine a target depth map from the depth map based on pixel correspondences; and cluster the depth information of multiple pixels included in the target depth map to obtain the target cluster set.

[0140] In a feasible embodiment, a clustering processing module is used to cluster the depths of multiple pixels contained in the target depth map to obtain a cluster set; the cluster set contains multiple sub-cluster sets; each cluster set in the multiple sub-cluster sets corresponds to an object in the target image; the coordinates of each pixel point in each sub-cluster set are obtained; the occupancy value corresponding to each sub-cluster set is determined according to the coordinates of each pixel point in each sub-cluster set; and the sub-cluster set with the largest occupancy value is determined as the target cluster set.

[0141] In a feasible embodiment, a center depth determination module is used to determine a first depth set and a second depth set from the target cluster set; the value of any depth in the first depth set is less than the value of any depth in the second depth set; the first target depth is determined based on multiple depths in the first depth set; the second target depth is determined based on multiple depths in the second depth set; and the center depth of the target object corresponding to the target cluster set is determined according to the first target depth and the second target depth.

[0142] In a feasible embodiment, the central depth determination module is used to sort the depths of multiple pixel points in the target cluster set from small to large to obtain a sorted target cluster set; the sorted target cluster set includes the depth of each pixel point in the multiple pixel points and the corresponding arrangement number; the depth set formed by the depths that meet the first preset condition in the sorted target cluster set is determined as the first depth set; the depth set formed by the depths that meet the second preset condition in the sorted target cluster set is determined as the second depth set.

[0143] In a feasible embodiment, the first preset condition is: the arrangement sequence number corresponding to the depth is less than or equal to a first threshold;

[0144] The second preset condition is that the arrangement sequence number corresponding to the depth is greater than or equal to a second threshold.

[0145] In a feasible embodiment, the position result determination module is used to obtain a depth difference set based on the difference in center depth of the target objects corresponding to any two target images in the target image set; and use the trained position relationship classification model to process the depth difference set and the visual features and semantic features corresponding to each target image in the target image set to obtain the position result between any two target objects among the multiple target objects.

[0146] An embodiment of the present application also provides an electronic device, which includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the identification method as described above.

[0147] An embodiment of the present application also provides a computer storage medium, which can be set in a server to store at least one instruction, at least one program, code set or instruction set related to an identification method in the method embodiment. The at least one instruction, the at least one program, the code set or instruction set is loaded and executed by the processor to implement the above-mentioned identification method.

[0148] Optionally, in this embodiment, the storage medium may be located in at least one of a plurality of network servers in a computer network. Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing program code, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0149] It should be noted that the order of the embodiments of the present application described above is for descriptive purposes only and does not represent the superiority or inferiority of the embodiments. The above description is of specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0150] The various embodiments in this specification are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the device embodiments are generally similar to the method embodiments, so the description is relatively simple. For relevant parts, refer to the description of the method embodiments.

[0151] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or by a program to instruct the relevant hardware, and the program may be stored in a computer-readable storage medium, which may be a read-only memory, a disk, or an optical disk, etc.

[0152] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application.

Claims

1. A method for identifying the position relationship of objects in an image, characterized in that: include: Obtaining an RGB image and a depth image containing a plurality of objects to be identified; wherein the depth image contains depth information corresponding to each pixel in the RGB image; Performing detection processing on the RGB image to obtain a target image set; the target image set includes a plurality of target images; each of the plurality of target images includes a target object, and the target object is an identified object to be identified; For each target image in the target image set, performing a feature extraction operation on the target image to obtain visual features and semantic features; Performing clustering processing on depth information of multiple pixels in the target image to obtain a target cluster set; Determining a center depth of a target object corresponding to the target cluster set based on the depths of a plurality of pixels in the target cluster set; Based on the visual features and semantic features corresponding to each target image in the target image set, and the difference in the center depth of the target objects corresponding to any two target images in the target image set, the position result between any two target objects in the multiple target objects is determined; the position result represents the relative position relationship between the two target objects.

2. The identification method according to claim 1, characterized in that The clustering process is performed on the depth information of a plurality of pixels in the target image to obtain a target cluster set, including: Determining a target depth map from the depth map based on the pixel correspondence relationship; Clustering is performed on depth information of a plurality of pixels included in the target depth map to obtain the target cluster set.

3. The identification method according to claim 2, characterized in that The clustering process of the depth information of the plurality of pixels included in the target depth map to obtain the target cluster set includes: Clustering the depths of a plurality of pixels included in the target depth map to obtain a cluster set; the cluster set includes a plurality of sub-clusters; each of the plurality of sub-clusters corresponds to an object in the target image; Obtaining the coordinates of each pixel in each sub-cluster; Determining a placeholder value corresponding to each sub-cluster according to the coordinates of each pixel point in each sub-cluster; The sub-cluster set with the largest placeholder value is determined as the target cluster set.

4. The identification method according to claim 1, wherein: The determining the center depth of the target object corresponding to the target cluster set based on the depths of the plurality of pixels in the target cluster set includes: Determining a first depth set and a second depth set from the target cluster set; a value of any depth in the first depth set is smaller than a value of any depth in the second depth set; determining a first target depth based on a plurality of depths in the first depth set; determining a second target depth based on a plurality of depths in the second depth set; The center depth of the target object corresponding to the target cluster set is determined according to the first target depth and the second target depth.

5. The identification method according to claim 4, characterized in that: The determining of the first depth set and the second depth set from the target cluster set includes: Sorting the depths of the plurality of pixels in the target cluster set from small to large to obtain a sorted target cluster set; the sorted target cluster set includes the depth of each pixel in the plurality of pixels and a corresponding arrangement number; Determine a depth set formed by depths that meet a first preset condition in the sorted target cluster set as a first depth set; A depth set formed by depths satisfying a second preset condition in the sorted target cluster set is determined as a second depth set.

6. The identification method according to claim 5, characterized in that The first preset condition is that the arrangement sequence number corresponding to the depth is less than or equal to a first threshold; The second preset condition is that the arrangement sequence number corresponding to the depth is greater than or equal to a second threshold.

7. The identification method according to claim 1, characterized in that: The determining of the position result between any two target objects in the plurality of target objects according to the visual features, semantic features and center depths of the target objects corresponding to the respective target images in the target image set includes: Obtaining a depth difference set according to a difference in center depths of target objects corresponding to any two target images in the target image set; The trained position relationship classification model is used to process the depth difference set and the visual features and semantic features corresponding to each target image in the target image set to obtain the position result between any two target objects among the multiple target objects.

8. A device for identifying positional relationships of objects, characterized in that: include: An image acquisition module is configured to acquire an RGB image and a depth image containing a plurality of objects to be identified; the depth image contains depth information corresponding to each pixel in the RGB image; a target image set acquisition module, configured to perform detection processing on the RGB image to obtain a target image set; the target image set includes a plurality of target images; each of the plurality of target images includes a target object, and the target object is an identified object to be identified; A feature extraction module is used to perform a feature extraction operation on each target image in the target image set to obtain visual features and semantic features; A clustering processing module, configured to perform clustering processing on depth information of a plurality of pixels in the target image to obtain a target cluster set; A center depth determination module, configured to determine a center depth of a target object corresponding to the target cluster set based on the depths of a plurality of pixels in the target cluster set; A position result determination module is used to determine the position result between any two target objects in a plurality of target objects based on the visual features and semantic features corresponding to each target image in the target image set and the difference in the center depth of the target objects corresponding to any two target images in the target image set; the position result represents the relative position relationship between the two target objects.

9. An electronic device, comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the identification method according to any one of claims 1 to 7.

10. A computer storage medium, characterized in that The computer storage medium stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by the processor to implement the identification method according to any one of claims 1 to 7.

Images