Method and apparatus for detecting object affordances

Abstract

The present disclosure relates to the field of computers, and provides a method and apparatus for detecting object affordance. The method includes the steps of obtaining a reference image and an image to be detected, the reference image including a person and a first object to which the person performs an action, extracting a feature of the reference image and a feature of the image to be detected, extracting motion intention information of the person with respect to the first object in the reference image according to the feature of the reference image, and transferring the motion intention information to the image to be detected according to the motion intention information of the person with respect to the first object in the reference image and the feature of the image to be detected, and segmenting a second object capable of complementing the motion intention information from the image to be detected. The ability to detect the affordance of an object at first sight is improved.

Description

[Technical Field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application is based on and claims priority from Chinese Patent Application No. 202110269399.3, filed on March 12, 2021, the disclosure of which is hereby incorporated in its entirety into this application.

[0002] The present disclosure relates to the field of computers, and more particularly to a method and apparatus for detecting object affordances. [Background technology]

[0003] Affordance was proposed by psychologist Gibson in 1966. It explains how we directly perceive the intrinsic value or meaning of objects in our environment, and how this information correlates with the possibilities for action offered by the environment.

[0004] In practice, perceiving the affordances of each object in an unknown environment is very important and has important application value in aspects such as scene understanding, action recognition, and human-computer interaction. Summary of the Invention [Means for solving the problem]

[0005] Some embodiments of the present disclosure provide a method for detecting object affordances, the method comprising the steps of: acquiring a reference image and an image to be detected, wherein the reference image includes a person and a first object to which the person will perform an action; extracting features of the reference image and features of the image to be detected; extracting action intention information of the person with respect to the first object in the reference image according to the features of the reference image; and transferring the action intention information to the image to be detected according to the action intention information of the person with respect to the first object in the reference image and the features of the image to be detected, and segmenting a second object from the image to be detected that can complement the action intention information.

[0006] In some embodiments, the step of extracting action intention information of the person relative to the first object in the reference image comprises the steps of: performing a weighting operation on the feature amounts of the reference image according to a feature representation of the person in the reference image to obtain a first output; performing a weighting operation on the feature amounts of the reference image according to a feature representation of the first object in the reference image to obtain a second output; obtaining a third output describing a relative position of the action of the person relative to the first object according to the feature representation of the person and the feature representation of the first object in the reference image; and determining a action intention code of the person relative to the first object in the reference image according to the first output, the second output, and the third output.

[0007] In some embodiments, performing a weighting operation on the features of the reference image according to the feature representations of the people in the reference image to obtain a first output comprises performing a correlation operation between the pooled feature representations of the people in the reference image and each position of the features of the reference image; normalizing the result of the correlation operation to obtain a weight for each position; and multiplying the weight for each position and the features of the reference image to obtain the first output.

[0008] In some embodiments, performing a weighting operation on the features of the reference image according to the feature representations of the first objects in the reference image to obtain a second output comprises performing a correlation operation between the pooled feature representations of the first objects in the reference image and each position of the features of the reference image; normalizing the result of the correlation operation to obtain a weight for each position; and multiplying the weight for each position and the features of the reference image to obtain the second output.

[0009] In some embodiments, obtaining a third output describing a relative location of the movement of the person relative to the first object according to the feature representation of the person and the feature representation of the first object in the reference image comprises performing a correlation operation on the pooled feature representation of the first object and the feature representation of the person; and convolving a result of the correlation operation to obtain the third output describing the relative location of the movement of the person relative to the first object.

[0010] In some embodiments, the step of determining the person's action intention code for the first object in the reference image according to the first output, the second output, and the third output comprises the steps of multiplying and pooling the third output and the first output to obtain first action intention sub-information, multiplying and pooling the third output and the second output to obtain second action intention sub-information, and adding the first action intention sub-information and the second action intention sub-information to obtain the person's action intention code for the first object in the reference image.

[0011] In some embodiments, the feature representation of the person in the reference image is obtained by multiplying a bounding box of the person in the reference image and the feature of the reference image, and the feature representation of the first object in the reference image is obtained by multiplying a bounding box of the first object in the reference image and the feature of the reference image.

[0012] In some embodiments, the step of transferring the action intention information to the image to be detected according to the action intention information of the person for the first object in the reference image and the feature quantities of the image to be detected, and segmenting a second object from the image to be detected that can complement the action intention information, comprises the steps of: performing a correlation operation on the action intention information of the person for each position of the first object in the reference image and the feature quantities of the image to be detected, and normalizing to obtain a weight for each position; and multiplying the weight for each position and the feature quantities of the image to be detected, and adding the result of the multiplication and the feature quantities of the image to be detected to obtain the second object that has been segmented from the image to be detected and that can complement the action intention information.

[0013] In some embodiments, the method further comprises the steps of: performing a reconstruction on first features of the second object by using a set of bases, wherein the set of bases is capable of capturing common features between different objects having one same affordance; determining second features of the second object according to the first features of the second object and the reconstructed first features of the second object; and outputting an image of the second object according to the second features of the second object.

[0014] In some embodiments, the method for obtaining the set of bases comprises: randomly initializing a set of bases by using a preset optimization algorithm; iteratively updating the set of bases by reducing different information between training images and the training images after correlation operation using the set of bases; and regarding the updated set of bases as a set of bases that has been learned and can capture the common features between different objects with one same affordance, wherein the optimization algorithm comprises an expectation-maximization algorithm or a gradient descent algorithm.

[0015] Some embodiments of the present disclosure provide an apparatus for detecting object affordances, comprising: a memory; and a processor coupled to the memory, wherein the processor is configured to perform a method for detecting the object affordances based on instructions stored in the memory.

[0016] Some embodiments of the present disclosure provide an object affordance detection device, comprising: a feature extraction module configured to acquire a reference image and an image to be detected, where the reference image includes a person and a first object to which the person will perform an action, and extract features of the reference image and features of the image to be detected; an intention learning module configured to extract motion intention information of the person with respect to the first object in the reference image according to the features of the reference image; and an intention transfer module configured to transfer the motion intention information to the image to be detected according to the motion intention information of the person with respect to the first object in the reference image and the features of the image to be detected, and to segment a second object capable of complementing the motion intention information from the image to be detected.

[0017] In some embodiments, the apparatus further comprises: a synergistic enhancement module configured to reconstruct for first features of the second object by using a set of bases, the set of bases being capable of capturing common features between different objects having one same affordance; and determine second features of the second object according to the first features of the second object and the reconstructed first features of the second object; and a decoding module configured to output an image of the second object according to the second features of the second object.

[0018] Some embodiments of the present disclosure provide a non-transitory computer-readable storage medium having stored thereon a computer program that, when executed by a processor, implements the steps of the method for detecting object affordances.

[0019] The drawings necessary for describing the embodiments or related art will be briefly described below. The present disclosure can be more clearly understood in accordance with the following detailed description proceeding with reference to the accompanying drawings.

[0020] In the following description, the drawings are only some embodiments of the present disclosure, and it is obvious to those skilled in the art that other drawings can be obtained without any creative efforts. [Brief explanation of the drawings]

[0021] [Figure 1] 1 shows a schematic flow diagram of a method for detecting object affordances according to some embodiments of the present disclosure. [Figure 2] FIG. 10 shows a schematic diagram of a step of extracting action intention information of a person with respect to a first object in a reference image according to some embodiments of the present disclosure. [Figure 3] FIG. 1 shows a schematic diagram of an element-wise multiplication process according to some embodiments of the present disclosure. [Figure 4] FIG. 1 shows a schematic diagram of position-wise dot product processing according to some embodiments of the present disclosure. [Figure 5] FIG. 10 shows a schematic diagram of the steps of transferring action intention information to an image to be detected and segmenting a second object capable of complementing action intention information from the image to be detected, according to some embodiments of the present disclosure. [Figure 6] FIG. 1 shows a schematic diagram of the synergistic enhancement steps according to some embodiments of the present disclosure. [Figure 7] 1 shows a schematic diagram of an object affordance detection device according to some embodiments of the present disclosure; [Figure 8] 10 shows a schematic diagram of an apparatus for detecting object affordances according to another embodiment of the present disclosure;

[0022] (Detailed explanation) Hereinafter, the technical solutions in the embodiments of the present disclosure will be clearly and completely described with reference to the drawings in the embodiments of the present disclosure.

[0023] Unless otherwise specified, the terms "first", "second", etc. in this disclosure do not denote any sense of size or chronological order, but are used to distinguish between different objects.

[0024] According to an embodiment of the present disclosure, a person's action intention toward an object is captured based on a reference image comprising the person and the object, and the action intention is transferred to all images to be detected, from which all objects capable of completing the action intention are segmented, thereby improving the ability to detect the affordances of "unseen" objects. Furthermore, common features (i.e., essential relationships) between different objects with the same affordance are captured in a synergistic manner, and various objects with this affordance are detected based on the common features, thereby improving the robustness of object affordance detection.

[0025] "Unseen" objects are objects that have not been "seen" by the detection model of object affordances, i.e., objects that have not been learned by the detection model of object affordances.

[0026] FIG. 1 shows a schematic flow diagram of a method for detecting object affordances according to some embodiments of the present disclosure.

[0027] As shown in FIG. 1, the method for detecting object affordances of this embodiment includes steps 110 to 160, among which step 150 can be selectively performed as needed.

[0028] In the image acquisition step, step 110, a reference image (ie, support image) and an image to be detected (ie, query image) are acquired.

[0029] The reference image may comprise a person and a first object on which the person performs an action, and may be marked with a bounding box of the person and a bounding box of the first object. For example, a "person kicking a ball" reference image may comprise a "person" kicking the ball and a "ball" being kicked by the person, and may be marked with a rectangular box for the "person" and a rectangular box for the "ball."

[0030] There can be one or more images to be detected, and if there are multiple images to be detected, the same affordance detection operation is performed for each image to be detected as is performed for one image to be detected.

[0031] In step 120, which is a feature extraction step, the feature of the reference image and the feature of the image to be detected are extracted.

[0032] By using image feature extraction networks such as Resnet (Residual Network) and VGGnet, features of the reference image / image to be detected are extracted.

[0033] In step 130, which is an intention learning step, information on the person's intention to act with respect to the first object in the reference image is extracted according to the feature amount of the reference image.

[0034] In some embodiments, extracting the person's action intention information for the first object in the reference image comprises steps 130.1 to 130.4, as shown in FIG.

[0035] In step 130.1, a weighting operation is performed on the features of the reference image according to the feature representations of people in the reference image to obtain a first output.

[0036] To obtain a feature representation of the person in the reference image, the bounding box of the person in the reference image and the feature of the reference image are multiplied.

[0037] In some embodiments, performing a weighting operation on the features of the reference image according to the feature representations of the people in the reference image to obtain a first output comprises performing a correlation operation (such as element-wise multiplication) for each position of the pooled feature representations (such as Global Average Pooling) of the people in the reference image and the features of the reference image; normalizing the result of the correlation operation (such as by Softmax) to obtain a weight for each position; and performing a multiplication (such as element-wise multiplication) on the weight for each position and the features of the reference image to obtain the first output.

[0038] Each position of the feature of the image can be, for example, each pixel position of the image, and correspondingly, each position of the feature of the reference image can be each pixel position of the reference image, and each position of the feature of the image to be detected can be each pixel position of the image to be detected.

[0039] In step 130.2, a weighting operation on the features of the reference image is performed according to the feature representation of the first object in the reference image to obtain a second output.

[0040] To obtain a feature representation of the first object in the reference image, a bounding box of the first object in the reference image and the feature of the reference image are multiplied.

[0041] In some embodiments, performing a weighting operation on the features of the reference image according to the feature representation of the first object in the reference image to obtain a second output comprises performing a correlation operation (such as element-wise multiplication) for each position of the pooled feature representation (such as Global Average Pooling) of the first object in the reference image and the features of the reference image; normalizing (such as by Softmax) the result of the correlation operation to obtain a weight for each position; and performing a multiplication (such as element-wise multiplication) of the weight for each position and the features of the reference image to obtain the second output.

[0042] In step 130.3, a third output is obtained describing the relative position of the person's action relative to the first object according to the feature representation of the person and the feature representation of the first object in the reference image.

[0043] In some embodiments, obtaining a third output describing a relative position of the person's action relative to the first object according to the feature representation of the person and the feature representation of the first object in the reference image comprises: performing a correlation operation (e.g., element-wise multiplication) on the pooled feature representation of the first object (e.g., global average pooling) and the feature representation of the person, such that a relative position of the person's action relative to the first object in the reference image is tracked of interest; and performing a convolution (e.g., 1×1 or 3×3 convolution) of the result of the correlation operation to obtain a third output describing the relative position of the person's action relative to the first object. The third output is a 1×H×W feature map, where H is the height of the feature map of the reference image and W is the width of the feature map of the reference image.

[0044] In step 130.4, the action intention code of the person for the first object in the reference image is determined according to the first output, the second output, and the third output.

[0045] In some embodiments, the step of determining the person's action intention code for the first object in the reference image according to the first output, the second output, and the third output comprises the steps of multiplying and pooling (such as a dot product per position) the third output and the first output to obtain first action intention sub-information, multiplying and pooling (such as a dot product per position) the third output and the second output to obtain second action intention sub-information, and adding the first action intention sub-information and the second action intention sub-information to obtain the person's action intention code for the first object in the reference image.

[0046] As shown in Figure 3, the element-wise multiplication process is as follows: take two matrices A ([1,1,C]) and B ([W,H,C]) as input, and multiply by the third dimension (i.e., channel, C) of A and B, that is, multiply by the corresponding channel of A and B, which is also called the "channel correlation operation."

[0047] As shown in Figure 4, the positional dot product process is as follows: Two matrices A([W,H,1]) and B([W,H,C]) are input, and the first two dimensions of A and B, which represent the height (H) and width (W) of the feature map matrix, respectively, are multiplied, i.e., the corresponding positions of A and B are multiplied. This is also called the "position correlation operation."

[0048] In step 140, which is the intention transfer step, the action intention information is transferred to the image to be detected according to the person's action intention information for the first object in the reference image and the feature amount of the image to be detected, and a second object (first feature amount of the second object) that can complement the action intention information is segmented from the image to be detected.

[0049] In some embodiments, as shown in FIG. 5 , the step of transferring the action intention information to the image to be detected and segmenting a second object from the image to be detected that can complement the action intention information comprises the steps of: performing a correlation operation (e.g., element-wise multiplication) at each position of the feature of the image to be detected by using the action intention information of the person (i.e., action intention code) for the first object in the reference image to obtain the second object (first feature of the second object) that has been segmented from the image to be detected and that can complement the action intention information, i.e., to obtain features of relevant regions activated by the action intention information (step 140.1); performing normalization (e.g., Softmax method) to obtain weights for each position (step 140.2); performing multiplication (e.g., element-wise multiplication) between the weights for each position and the feature of the image to be detected (step 140.3); and performing addition between the multiplication result and the feature of the image to be detected (step 140.4).

[0050] The fact that different objects can complement the same action intention means that these different objects have one and the same affordance that corresponds to this action intention.

[0051] Step 150, which is a synergistic enhancement step, performs reconstruction on a first feature of a second object by using a set of bases, where the set of bases can capture common features between different objects having one same affordance, and determines a second feature of the second object according to the first feature of the second object and the reconstructed first feature of the second object.

[0052] In some embodiments, as shown in FIG. 6, the synergistic enhancement step particularly comprises a step of performing a correlation operation (such as a position-wise dot product) on the first feature of the second object and the set of bases to perform a reconstruction (step 150.1), and a step of convolution (conv) of the reconstructed first feature of the second object (step 150.2), followed by addition with the first feature of the second object to obtain the second feature of the second object.

[0053] The basis set can capture the essential relationships (embodied by common features) between different objects with the same affordance, and based on the basis set it is possible to suppress irrelevant background areas in the detection process, thereby achieving better detection results.

[0054] The method for obtaining the basis set includes the steps of randomly initializing a set of bases by using a preset optimization algorithm; iteratively updating the set of bases by successively reducing the difference information between the training images after a correlation operation (such as a positional dot product) using the training images and the set of bases; and regarding the updated set of bases as a set of learned bases that can capture common features between different objects with the same affordance. The optimization algorithm includes an expectation-maximum (EM) algorithm or a gradient descent algorithm. Some of the bases in the set are usually much smaller than the product of the height (H) and width (W) of the image. The number of bases in the set is, for example, several or a dozen.

[0055] For example, a set of bases is randomly initialized, each of which has the format [1,1,C], where C represents a channel. Correlation operations (e.g., position-wise dot products) are performed on each base and features of training images with dimensions [W,H,C], where W and H represent the width and height of the feature map of the training images, respectively. Correlation results corresponding to a number of bases are combined to obtain features of the training images after correlation using the set of bases. The features of the training images after correlation are convolved to convert them into features with dimensions [W,H,C]. The set of bases is iteratively updated using an expectation-maximization optimization method, so that the differences between the features of the training images and the features of the training images after correlation and convolution are continuously reduced until a preset number of iterations is reached or the differences become smaller than a preset value, and the iterations are stopped. The updated set of bases is a set of bases that can be learned and capture common features between different objects with the same affordance.

[0056] In step 160, which is a decoding and outputting step, the image of the second object is output by decoding.

[0057] After step 140, the image of the second object is output by decoding according to the first feature of the second object. After step 150, the image of the second object is output by decoding according to the second feature of the second object.

[0058] The so-called decoding means that the image features are restored to the corresponding image. Decoding can be implemented, for example, by deconvolution or by first upsampling and then convolution.

[0059] In the above embodiment, a person's action intention toward an object is captured based on a reference image comprising the person and the object, the action intention is transferred to all images to be detected, and then all objects that can complement the action intention are segmented, thereby improving the ability to detect affordances of unseen objects. Furthermore, in the above embodiment, common features (i.e., essential relationships) between different objects having one same affordance are captured in a synergistic manner, and various objects having this affordance are detected based on the common features, thereby improving the robustness of detecting object affordances.

[0060] In some embodiments, for example, a reference image of "a person kicking a ball" is given, and the action intention of "a person kicking a ball" is captured from the reference image. Based on the action intention of "a person kicking a ball" captured from the reference image, the action intention is transferred to all images to be detected, and then all spherical objects that match the action intention are segmented, thereby improving the ability to detect affordances of unseen objects. Also, common features between different objects that match the action intention, such as the appearance of a circular arc, can be captured in a synergistic manner, and various objects that match the action intention are detected based on the common features, thereby improving the robustness of detecting object affordances.

[0061] 7 shows a schematic diagram of an object affordance detection device according to some embodiments of the present disclosure. The object affordance detection device is also called an object affordance detection network.

[0062] As shown in FIG. 7, the object affordance detection device 700 of this embodiment includes modules 710 to 750, and module 740 is selectively configured or executed.

[0063] The feature extraction module 710 is configured to obtain a reference image and an image to be detected, where the reference image includes a person and a first object on which the person performs an action, and to extract features of the reference image and features of the image to be detected.

[0064] The intention learning module 720 is configured to extract the person's action intention information toward the first object in the reference image according to the feature amount of the reference image.

[0065] The intention transfer module 730 is configured to transfer the action intention information to the image to be detected according to the action intention information of the person for the first object in the reference image and the feature amount of the image to be detected, and to segment the second object (first feature amount of the second object) that can complement the action intention information from the image to be detected.

[0066] The decoding module 750 is configured to output an image of the second object according to the first feature of the second object.

[0067] In some embodiments, the object affordance detection apparatus 700 further comprises a synergistic enhancement module 740 configured to perform a reconstruction for a first feature of a second object by using a set of bases, where the set of bases can capture common features between different objects having one same affordance, and determine a second feature of the second object according to the first feature of the second object and the reconstructed first feature of the second object. Currently, a decoding module 750 is configured to output an image of the second object according to the second feature of the second object.

[0068] For the specific processing of the operations performed by the above modules, reference may be made to the foregoing embodiments, which will not be repeated here.

[0069] Although the object affordance detector 700 needs to be trained before it can be used, the object affordance detector 700 can be pre-trained and used directly.

[0070] The training process of the object affordance detection device 700 includes the steps of: acquiring a dataset; dividing the dataset into a training set and a test set, both of which include a reference image and an image to be detected, where one reference image can correspond to one or more images to be detected; inputting the reference image in the training set, which is marked with a bounding box of a person and a first object, and the image to be detected in the training set, which is pre-marked with a second object that can match the person's action intention for the first object in the reference image, into the object affordance detection device 700 for detection, where the detected object is set as a third object; determining a loss according to difference information between the detected third object and the pre-marked second object and a loss function (such as cross entropy); and optimizing network parameters (such as various parameters in the convolutions in the modules) in the detection device 700 by using an optimization function (such as adam and sgd (Stochastic Gradient Descent)), so that the loss can be reduced to a certain extent and training is completed. Then, the trained detection device 700 is tested by using a test set. A reference image in the test set is marked with a bounding box of a person and a first object, an image to be detected in the test set is pre-marked with a second object that can match the person's action intention for the first object in the reference image, the reference image in the test set and the image to be detected are input into the object affordance detection device 700 for detection, the detected object is set as a third object, and the detection accuracy is determined according to whether the detected third object belongs to the pre-marked second object, and if the detection accuracy is higher than a certain level, the detection device 700 passes the test and is deemed qualified.If the detection device 700 does not pass the test, the training of the detection device 700 can be continued by increasing the number of training samples or the number of training iterations. During training, for example, to improve the training effect, data containing one-third of the affordance categories can be selected as a test set, and the data of the remaining affordance categories can be regarded as a training set for training.

[0071] FIG. 8 shows a schematic diagram of an object affordance detection device according to another embodiment of the present disclosure.

[0072] As shown in FIG. 8, the object affordance detection device 800 of this embodiment includes a memory 810 and a processor 820 coupled to the memory 810, and the processor 820 is configured to perform the object affordance detection method of any of the above-described embodiments based on instructions stored in the memory 810.

[0073] Memory 810 may comprise, for example, system memory, fixed non-volatile storage media, etc. The system memory may store, for example, an operating system, application programs, boot loaders, other programs, etc.

[0074] The detection device 800 may further include an input / output interface 830, a network interface 840, a storage interface 850, etc. These interfaces 830, 840, 850 and the memory 810 may be connected to the processor 820 via, for example, a bus 860. The input / output interface 830 provides a connection interface for input / output devices such as a display, a mouse, a keyboard, and a touch screen. The network interface 840 provides a connection interface for various networking devices. The storage interface 850 provides a connection interface for external storage devices such as an SD card and a USB flash disk.

[0075] Some embodiments of the present disclosure provide a non-transitory computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method for detecting object affordances in any of the preceding embodiments.

[0076] It should be understood by those skilled in the art that embodiments of the present disclosure can be provided as a method, a system, or a computer program product. Thus, the present disclosure can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present disclosure can take the form of a computer program product embodied on one or more non-transitory computer-readable storage medium (including, but not limited to, disk memory, CD-ROM, optical memory, etc.) having computer program code embodied therein.

[0077] The present disclosure will be described with reference to flow diagrams and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the present disclosure. It should be understood that each flow and / or block in the flow diagrams and / or block diagrams, and combinations of flows and / or blocks in the flow diagrams and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or another programmable data processing device to produce a machine, such that the instructions, executed by a processor of a computer or other programmable data processing device, create means for implementing the specified function(s) in one or more flows and / or blocks in the flow diagrams and / or block diagrams.

[0078] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a particular manner to produce a product that includes instruction means that implement the specified functions in one or more flows of the flow diagrams and / or one or more blocks of the block diagrams, where the instructions stored in the computer-readable memory.

[0079] These computer program instructions may also be loaded onto a computer or other programmable data processing device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process in which the instructions, when executed on the computer or other programmable device, provide steps for implementing the specified functions in one or more flows of the flow diagrams and / or one or more blocks of the block diagrams.

[0080] The above description is only a preferred embodiment of the present disclosure and is not used to limit the present disclosure, and any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present disclosure should be included in the protection scope of the present disclosure. [Explanation of symbols]

[0081] 110 steps 120 steps 130 steps 140 steps 150 steps 160 steps 700 Detection Device 710 Feature Extraction Module 720 Intention Learning Module 730 Intention Transfer Module 740 Synergistic Enhancement Module 750 Decryption Module 800 Detector 810 memory 820 processor 830 Input / Output Interface 840 network interface 850 Storage Interface 850 Interface 860 Bus

Claims

1. A method for detecting an object affordance, comprising: acquiring a reference image and an image to be detected, the reference image comprising a person and a first object on which the person performs an action; extracting features of the reference image and features of the image to be detected; extracting action intention information of the person with respect to the first object in the reference image according to the feature amount of the reference image; transferring the action intention information to the image to be detected according to the action intention information of the person with respect to the first object in the reference image and the feature amount of the image to be detected, and segmenting a second object from the image to be detected that can complement the action intention information.

2. the step of extracting the person's action intention information with respect to the first object in the reference image comprises: performing a weighting operation on the features of the reference image according to a feature representation of the person in the reference image to obtain a first output; performing a weighting operation on the features of the reference image according to a feature representation of the first object in the reference image to obtain a second output; obtaining a third output describing a relative position of the movement of the person relative to the first object according to the feature representation of the person and the feature representation of the first object in the reference image; and determining an action intention code of the person with respect to the first object in the reference image according to the first output, the second output, and the third output.

3. performing a weighting operation on the feature amounts of the reference image according to a feature representation of the person in the reference image to obtain a first output, 3. The method of claim 2, comprising: performing a correlation operation between the pooled feature representation of the person in the reference image and each position of the feature of the reference image; normalizing a result of the correlation operation to obtain a weight for each position; and multiplying the weight for each position and the feature of the reference image to obtain the first output.

4. performing a weighting operation on the features of the reference image according to a feature representation of the first object in the reference image to obtain a second output, 3. The method of claim 2, comprising: performing a correlation operation on the pooled feature representation of the first object in the reference image and each position of the feature of the reference image; normalizing the result of the correlation operation to obtain a weight for each position; and multiplying the weight for each position and the feature of the reference image to obtain the second output.

5. obtaining a third output describing a relative position of the movement of the person relative to the first object according to the feature representation of the person and the feature representation of the first object in the reference image, 3. The method of claim 2, comprising: performing a correlation operation on the pooled feature representation of the first object and the feature representation of the person; and convolving a result of the correlation operation to obtain the third output describing the relative location of the movement of the person relative to the first object.

6. determining an action intention code of the person for the first object in the reference image according to the first output, the second output, and the third output, multiplying and pooling the third output and the first output to obtain first action intention sub-information; multiplying and pooling the third output and the second output to obtain second action intention sub-information; and adding the first action intention sub-information and the second action intention sub-information to obtain the action intention code of the person relative to the first object in the reference image.

7. the feature representation of the person in the reference image is obtained by multiplying a bounding box of the person in the reference image and the feature of the reference image; The method of claim 2 , wherein the feature representation of the first object in the reference image is obtained by multiplying a bounding box of the first object in the reference image and the feature of the reference image.

8. the step of transferring the action intention information to the image to be detected according to the action intention information of the person with respect to the first object in the reference image and the feature amount of the image to be detected, and segmenting a second object capable of complementing the action intention information from the image to be detected, performing a correlation calculation on the action intention information of the person with respect to each position of the feature amount of the first object in the reference image and the image to be detected, and normalizing the correlation calculation to obtain a weight for each position; and multiplying the weight of each position and the feature amount of the image to be detected, and adding a result of the multiplication and the feature amount of the image to be detected, in order to obtain the second object that is segmented from the image to be detected and that can complement the action intention information.

9. performing a reconstruction on a first feature of the second object by using a set of bases, the set of bases being capable of capturing common features between different objects having the same affordance; determining a second feature of the second object according to the first feature of the second object and the reconstructed first feature of the second object; The method of claim 1 , further comprising: outputting an image of the second object according to the second feature of the second object.

10. The method for obtaining the set of bases comprises: The method includes the steps of: randomly initializing a set of bases by using a preset optimization algorithm; iteratively updating the set of bases by reducing different information between training images and the training images after correlation operation using the set of bases; and regarding the updated set of bases as a set of learned bases that can capture the common feature amount between different objects with one same affordance; The method of claim 9 , wherein the optimization algorithm comprises an expectation-maximum algorithm or a gradient descent algorithm.

11. A device for detecting object affordances, comprising: Memory and a processor coupled to the memory, An apparatus, wherein the processor is configured to perform the method for detecting object affordances according to any one of claims 1 to 10 based on instructions stored in the memory.

12. A device for detecting object affordances, comprising: A feature extraction module, acquiring a reference image and an image to be detected, the reference image comprising a person and a first object on which the person performs an action; extracting a feature amount of the reference image and a feature amount of the image to be detected; a feature extraction module configured to: an intention learning module configured to extract action intention information of the person with respect to the first object in the reference image according to the feature amount of the reference image; an intention transfer module configured to transfer the action intention information to the image to be detected according to the action intention information of the person with respect to the first object in the reference image and the feature amount of the image to be detected, and to segment a second object from the image to be detected that can complement the action intention information.

13. A synergistic enhancement module, comprising: Reconstructing a first feature of the second object using a set of bases; The set of bases can capture common features between different objects having one same affordance, and determining a second feature of the second object according to the first feature of the second object and the reconstructed first feature of the second object; a synergistic reinforcement module configured to: The apparatus of claim 12 , further comprising: a decoding module configured to output an image of the second object according to the second feature of the second object.

14. A non-transitory computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the method for detecting object affordances according to any one of claims 1 to 10.

Images