Method, device, equipment and storage medium for generating availability detection model

Abstract

The present application discloses a method, device, equipment and storage medium for generating an availability detection model, and relates to the field of artificial intelligence technology. A specific implementation of the method includes: obtaining a feature map of a first-person perspective and a set of feature maps of a third-person perspective; inputting a set of feature maps of a third-person perspective into a preset availability feature mining model to obtain availability features; inputting the feature map of the first-person perspective and the availability features into an initial availability detection model respectively to obtain a prediction value corresponding to the feature map of the first-person perspective and a prediction value corresponding to the availability features; training the initial availability detection model using at least one of the following determined target loss functions to obtain an availability detection model: a first loss function, a second loss function, and a third loss function.

Description

Technical Field

[0001] The present application relates to the field of computer technology, specifically to the field of artificial intelligence technology, and in particular to methods, devices, equipment and storage media for generating an availability detection model. Background Art

[0002] Visual affordance treats affordance as a computer vision problem based on images or videos, and uses machine learning related techniques to solve these challenges. Perceiving and reasoning about the possible interactions of local areas of objects is the key to transforming passive perception systems into embodied intelligent systems that actively interact and perceive the environment. Detecting the affordance area of ​​objects in weakly supervised scenarios has also attracted the attention of researchers.

[0003] Existing weakly supervised affordance detection work is basically based on constructing a mapping relationship between object appearance and affordance category. Affordance is different from the semantic attributes of objects. It is a dynamic attribute closely related to the environment and the operator. Existing methods are difficult to adapt to the dynamic attributes of affordance, to dynamically changing environments, and to generalize to new and unseen objects. Summary of the invention

[0004] The embodiments of the present application provide a method, apparatus, device and storage medium for generating an availability detection model.

[0005] In a first aspect, an embodiment of the present application provides a method for generating an availability detection model, the method comprising: obtaining a feature map of a first-person perspective and a set of feature maps of a third-person perspective; inputting a set of feature maps of a third-person perspective into a preset availability feature mining model to obtain availability features; inputting the feature map of the first-person perspective and the availability features into an initial availability detection model respectively to obtain a prediction value corresponding to the feature map of the first-person perspective and a prediction value corresponding to the availability features; training the initial availability detection model using a target loss function determined by at least one of the following to obtain an availability detection model: a first loss function, a second loss function, and a third loss function, wherein the first loss function is determined based on the following steps: according to the first A first loss function is determined by the predicted value corresponding to the feature map of the first-person perspective and its corresponding availability category label, and the cross entropy of the predicted value corresponding to the availability feature and its corresponding availability category label; a second loss function is determined based on the following steps: an L2 loss function is determined based on the predicted value corresponding to the feature map of the first-person perspective and its corresponding availability category label, and the predicted value corresponding to the availability feature and its corresponding availability category label; a third loss function is determined based on the following steps: a first sym-occurrence matrix is ​​obtained by multiplying the predicted value corresponding to the feature map of the first-person perspective and its transpose; and a second sym-occurrence matrix is ​​obtained by multiplying the predicted value corresponding to the availability feature and its transpose; and the third loss function is determined based on the cross entropy of the first sym-occurrence matrix and the second sym-occurrence matrix.

[0006] In some embodiments, a set of feature maps from a third-person perspective are input into a preset availability feature mining model to obtain availability features, including: reshaping a set of feature maps from a third-person perspective to generate a multidimensional matrix, wherein the rows of the multidimensional matrix are the same as the channel dimensions of a set of feature maps from a third-person perspective, and the columns of the multidimensional matrix are N*h*w, N is the number of images from a set of third-person perspectives, and h and w are the length and width of a set of feature maps from a third-person perspective, respectively; decomposing the multidimensional matrix into a basis matrix W and a coefficient matrix H; iteratively updating the basis matrix W and the coefficient matrix H to perform a non-negative matrix decomposition of the multidimensional matrix into the product of an updated basis matrix W and an updated coefficient matrix H; reshaping the updated basis matrix W and the updated coefficient matrix H to obtain a feature map after the reshape operation; and performing feature fusion based on a set of feature maps from a third-person perspective and the feature map after the reshape operation to obtain availability features.

[0007] In some embodiments, the second loss function is determined based on the following steps: inputting a feature map of the first-person perspective into a preset mapping layer, mapping the feature map of the first-person perspective into a feature space corresponding to a set of feature maps of the third-person perspective; inputting a set of feature maps of the third-person perspective and a feature map of the first-person perspective into preset convolutional layers, respectively, and passing the obtained results through pooling layers respectively; determining the L2 loss function based on the pooled results.

[0008] In some embodiments, the initial availability detection model is trained using at least one of the following target loss functions to obtain the availability detection model: a first loss function, a second loss function, and a third loss function, including: the initial availability detection model is trained using at least one of the following target loss functions to obtain the availability detection model: the first loss function and its corresponding weight, the second loss function and its corresponding weight, and the third loss function and its corresponding weight.

[0009] In some embodiments, obtaining a feature map of a first-person perspective and a set of feature maps of a third-person perspective includes: obtaining an image from a first-person perspective and a set of images from a third-person perspective, wherein the image from the first-person perspective and the set of images from the third-person perspective include objects of the same category; inputting the image from the first-person perspective and the set of images from the third-person perspective into a preset feature extractor, respectively, to obtain a feature map of the first-person perspective and a set of feature maps of the third-person perspective.

[0010] In a second aspect, an embodiment of the present application provides a method for identifying an availability category, the method comprising: obtaining an image from a first-person perspective to be predicted; inputting the image from the first-person perspective to be predicted into a preset feature extractor to obtain a feature map of the first-person perspective to be predicted; inputting the feature map of the first-person perspective to be predicted into the availability detection model described in the first aspect to obtain an availability category corresponding to the image from the first-person perspective to be predicted.

[0011] In a third aspect, an embodiment of the present application provides a device for generating an availability detection model, the device comprising: a feature map acquisition module, configured to acquire a feature map of a first-person perspective and a set of feature maps of a third-person perspective; a feature acquisition module, configured to input a set of feature maps of a third-person perspective into a preset availability feature mining model to obtain availability features; a prediction value acquisition module, configured to input the feature map of the first-person perspective and the availability features into an initial availability detection model, respectively, to obtain a prediction value corresponding to the feature map of the first-person perspective and a prediction value corresponding to the availability features; a model training module, configured to train the initial availability detection model using a target loss function determined by at least one of the following to obtain an availability detection model: a first loss function, a second loss function, a third loss function , wherein the first loss function is determined based on the following steps: determining the first loss function according to the predicted value corresponding to the feature map of the first-person perspective and its corresponding availability category label, and the cross entropy of the predicted value corresponding to the availability feature and its corresponding availability category label; the second loss function is determined based on the following steps: determining the L2 loss function according to the predicted value corresponding to the feature map of the first-person perspective and its corresponding availability category label, and the predicted value corresponding to the availability feature and its corresponding availability category label; the third loss function is determined based on the following steps: obtaining a first symbiosis matrix by multiplying the predicted value corresponding to the feature map of the first-person perspective and its transpose; and obtaining a second symbiosis matrix by multiplying the predicted value corresponding to the availability feature and its transpose; determining the third loss function according to the cross entropy of the first symbiosis matrix and the second symbiosis matrix.

[0012] In some embodiments, the feature acquisition module is further configured to: reshape a set of feature maps from a third-person perspective to generate a multidimensional matrix, wherein the rows of the multidimensional matrix are the same as the channel dimensions of the set of feature maps from a third-person perspective, and the columns of the multidimensional matrix are N*h*w, N is the number of images from a set of third-person perspectives, and h and w are the length and width of a set of feature maps from a third-person perspective, respectively; decompose the multidimensional matrix into a basis matrix W and a coefficient matrix H; iteratively update the basis matrix W and the coefficient matrix H to perform a non-negative matrix decomposition of the multidimensional matrix into the product of an updated basis matrix W and an updated coefficient matrix H; reshape the updated basis matrix W and the updated coefficient matrix H to obtain a feature map after the reshape operation; perform feature fusion based on a set of feature maps from a third-person perspective and the feature map after the reshape operation to obtain availability features.

[0013] In some embodiments, the device also includes: a feature mapping module, configured to input a feature map of the first-person perspective into a preset mapping layer, and map the feature map of the first-person perspective into a feature space corresponding to a set of feature maps of the third-person perspective; a feature processing module, configured to input a set of feature maps of the third-person perspective and a feature map of the first-person perspective into preset convolutional layers, respectively, and pass the obtained results through pooling layers, respectively; a function determination module, configured to determine the L2 loss function based on the pooled result.

[0014] In some embodiments, the model training module is further configured to: train the initial availability detection model using a target loss function determined by at least one of the following to obtain an availability detection model: a first loss function and its corresponding weight, a second loss function and its corresponding weight, and a third loss function and its corresponding weight.

[0015] In some embodiments, the feature map acquisition module is further configured to: acquire an image from a first-person perspective and a group of images from a third-person perspective, wherein the image from the first-person perspective and the group of images from the third-person perspective include objects of the same category; input the image from the first-person perspective and the group of images from the third-person perspective into a preset feature extractor, respectively, to obtain a feature map from the first-person perspective, and a group of feature maps from the third-person perspective.

[0016] In a fourth aspect, an embodiment of the present application provides an apparatus image acquisition module for identifying availability categories, which is configured to acquire an image from a first-person perspective to be predicted; a feature map extraction module, which is configured to input the image from the first-person perspective to be predicted into a preset feature extractor, to obtain a feature map of the first-person perspective to be predicted; and a category prediction module, which is configured to input the feature map of the first-person perspective to be predicted into the availability detection model described in the first aspect, to obtain an availability category corresponding to the image from the first-person perspective to be predicted.

[0017] In a fifth aspect, an embodiment of the present application provides an electronic device comprising at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in the first aspect or the second aspect.

[0018] In a sixth aspect, an embodiment of the present application provides a non-transitory computer-readable storage medium storing computer instructions, characterized in that the computer instructions are used to enable a computer to execute the method described in the first aspect or the second aspect.

[0019] The embodiments of the present application provide a method, apparatus, device and storage medium for generating an availability detection model, which trains an initial availability detection model to obtain an availability detection model using a target loss function determined by at least one of the following: a first loss function, a second loss function, and a third loss function, to train the availability detection model. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0021] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0022] Figure 2 is a flowchart of an embodiment of a method for generating an availability detection model according to the present application;

[0023] Figure 3 is a schematic diagram of generating the first loss function;

[0024] Figure 4 It is a schematic diagram to obtain the affordance characteristics;

[0025] Figure 5 is an application scenario diagram of the method for generating an availability detection model according to the present application;

[0026] Figure 6 is a flowchart of an embodiment of a method for identifying affordance categories according to the present application;

[0027] Figure 7 is an application scenario diagram of the method for identifying affordance categories according to the present application;

[0028] Figure 8 is a structural schematic diagram of an embodiment of an apparatus for generating an availability detection model according to the present application;

[0029] Fig. 9 is a structural schematic diagram of an embodiment of a device for identifying an availability category according to the present application;

[0030] Fig.10 It is a schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present application. DETAILED DESCRIPTION

[0031] The present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It should also be noted that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.

[0032] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0033] Figure 1 An exemplary system architecture 100 is shown to which an embodiment of a method for generating an availability detection model or an apparatus for generating an availability detection model of the present application can be applied.

[0034] like Figure 1 As shown, system architecture 100 may include terminal devices 101 and 102, network 103 and server 104. Network 103 is used to provide a medium for communication links between terminal devices 101 and 102 and server 104. Network 103 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0035] Users can use terminal devices 101 and 102 to interact with server 104 via network 103, such as target reporting.

[0036] The terminal devices 101 and 102 may be hardware or software. When the terminal devices 101 and 102 are hardware, they may be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When the terminal devices 101 and 102 are software, they may be implemented as multiple software or software modules (for example, for providing distributed services), or as a single software or software module. No specific limitation is made here.

[0037] The server 104 can provide various services. For example, the server 104 can obtain a target report; input the target report into a pre-trained text recognition model to obtain text structure information corresponding to the target report; obtain corresponding standard text information from a preset association relationship library based on the text structure information; and generate a standard report based on the standard text information.

[0038] It should be noted that the server 104 can be hardware or software. When the server 104 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server; or the server 104 can be an electronic product that interacts with the user through one or more methods such as a keyboard, touchpad, touch screen, remote control, voice interaction or handwriting device, such as a PC (Personal Computer), a mobile phone, a smart phone, a PDA (Personal Digital Assistant), a wearable device, a PPC (Pocket PC), a tablet computer, a smart car machine, a smart TV, a smart speaker, a tablet computer, a laptop portable computer and a desktop computer, etc. When the server 104 is software, it can be implemented as multiple software or software modules (for example, to provide distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0039] It should be noted that the method for generating an availability detection model provided in the embodiment of the present application is generally executed by the server 104 , and accordingly, the device for generating an availability detection model is generally set in the server 104 .

[0040] It should be understood that Figure 1 The number of electronic devices, networks and servers in the embodiment is only for illustration. Any number of electronic devices, networks and servers may be provided as required.

[0041] Continue to refer Figure 2 , which shows a flowchart 200 of an embodiment of a method for generating an availability detection model according to the present application. The method for generating an availability detection model may include the following steps:

[0042] Step 201, obtaining a feature map of a first-person perspective and a set of feature maps of a third-person perspective.

[0043] In this embodiment, the execution subject of the method for generating an availability detection model (for example Figure 1 The server 104 shown in FIG. Figure 1 The network 103 shown) from the terminal device (eg Figure 1 The servers 101 and 102 shown in the figure) or obtain a first-person perspective feature map and a set of third-person perspective feature maps from a local; or, an execution subject of the method for generating an availability detection model (e.g. Figure 1The terminal devices 101 and 102 shown in the figure can obtain a first-person perspective feature map and a set of third-person perspective feature maps from the local. The feature map of the first-person perspective may include features collected from the user's own perspective (i.e., the first-person perspective). The feature map of the third-person perspective may include features collected from the third-person perspective.

[0044] It should be noted that the number of a set of feature maps of the third-person perspective can be set according to the prediction accuracy of the affordance detection model or by the user according to work requirements.

[0045] Step 202: Input a set of feature maps of the third-person perspective into a preset affordance feature mining model to obtain affordance features.

[0046] In this embodiment, the execution subject may input a set of feature maps from a third-person perspective into a preset availability feature mining model to obtain availability features. The availability feature mining model may be used to extract availability features from a set of feature maps from a third-person perspective. The availability features may refer to user actions that an object in a set of images from a third-person perspective can withstand, that is, a user may apply different actions to different functional parts of an object in a set of images from a third-person perspective, thereby realizing the user's cognitive operation on an object in a set of images from a third-person perspective.

[0047] It should be noted that the above-mentioned objects may be people, plants, backgrounds, etc. in the image. The above-mentioned users may be real users, virtual robots, real robots, etc.

[0048] In this embodiment, the above-mentioned availability feature mining model can be determined based on the following steps: obtaining a feature map of the first-person perspective and a corresponding availability category label, as well as an availability feature and a corresponding availability category label; then, using the feature map of the first-person perspective and the corresponding availability category label, as well as the availability feature and the corresponding availability category label for training to obtain an availability feature mining model. During training, the execution subject can use the feature map of the first-person perspective and the availability feature as inputs of the availability feature mining model, and use the feature map of the first-person perspective and the corresponding availability category label, as well as the availability feature and the corresponding availability category label as expected outputs to obtain an availability feature mining model. The above-mentioned initial availability detection model can be a probability model, a classification model or other classifiers in existing technology or future development technology. For example, the initial availability detection model can include any of the following: a decision tree model (XGBoost), a logistic regression model (LR), and a deep neural network model (DNN).

[0049] Step 203, inputting the feature map of the first-person perspective and the availability feature into the initial availability detection model respectively, and obtaining the prediction value corresponding to the feature map of the first-person perspective and the prediction value corresponding to the availability feature.

[0050] In this embodiment, the above-mentioned execution entity can input the feature map of the first-person perspective and the availability feature into the initial availability detection model respectively to obtain the prediction value corresponding to the feature map of the first-person perspective and the prediction value corresponding to the availability feature.

[0051] It should be noted that the initial affordance detection model may be an affordance detection model that has not started training or a model that has been trained for one stage. The prediction value may be the result obtained by the initial affordance detection model predicting the feature map of the first-person perspective and the affordance feature respectively.

[0052] Step 204, training the initial availability detection model using a target loss function determined by at least one of the following to obtain an availability detection model: a first loss function, a second loss function, and a third loss function, wherein the first loss function is determined based on the following steps: determining the first loss function based on the predicted value corresponding to the feature map of the first-person perspective and its corresponding availability category label, and the cross entropy of the predicted value corresponding to the availability feature and its corresponding availability category label; the second loss function is determined based on the following steps: determining the L2 loss function based on the predicted value corresponding to the feature map of the first-person perspective and its corresponding availability category label, and the predicted value corresponding to the availability feature and its corresponding availability category label; the third loss function is determined based on the following steps: obtaining a first symbiosis matrix by multiplying the predicted value corresponding to the feature map of the first-person perspective and its transpose; and obtaining a second symbiosis matrix by multiplying the predicted value corresponding to the availability feature and its transpose; determining the third loss function based on the cross entropy of the first symbiosis matrix and the second symbiosis matrix.

[0053] In this embodiment, the execution entity may train the initial availability detection model using at least one of the first loss function, the second loss function, and the third loss function to obtain the availability detection model.

[0054] In an example, first, for third-person perspective images, they can be obtained from COCO (a large image dataset) and HICO datasets, and retrieved according to action and object (i.e., objects in the image) categories. For first-person perspective images, 3755 images can be obtained. After that, each third-person perspective image can be labeled with an affordance label according to 36 commonly used affordance categories. At the same time, for first-person perspective images, labels are assigned according to the affordances they may belong to. The training process only requires affordance category labels as supervision. During the testing process, pixel-level labels are required to evaluate the performance of the model. Referring to the annotation method of saliency detection, points of different densities are marked according to the probability of interaction in the local area of ​​the object, and smoothed with Gaussian blur. After normalization, affordance heatmaps are obtained as labels.

[0055] Correspondingly, in this example, training the availability detection model may include the following steps:

[0056] Step 1: The loss function in the training process may include at least one of the following: the cross entropy loss of the classification results of the two branches in the first part (i.e., the first loss function), the L2 loss of the feature representation of the two branches calculated in the second part (i.e., the second loss function), and the loss of the co-occurrence matrix alignment in the affordance co-occurrence relationship retention strategy calculated in the third part (i.e., the third loss function); the three losses are multiplied by different hyperparameters (e.g., weights) respectively, and then added together to obtain the total loss of the training (i.e., the target loss function). For example, the hyperparameter of the loss weight of the first part is 1, the number of parameters of the loss weight of the second part is 0.5, and the hyperparameter of the loss weight of the third part is 0.5.

[0057] Step 2: The parameters of the availability detection model are solved by using the gradient descent method based on SGD (Stochastic gradient descent). After each iteration, the error between the predicted value and the true label (i.e., the availability category label) is calculated, and back-propagation is used to the network to calculate the gradient and update the network parameters. The initial learning rate is 1e-3.

[0058] Step 3: Initialize weights, load data, model, optimizer, and perform training. Iterate until the iteration stop condition is met, such as 35 cycles or the difference between the predicted value and the true value is less than the preset threshold. The batch size is 14.

[0059] Here, the first loss function can be determined based on the cross entropy between the predicted value corresponding to the feature map of the first-person perspective and its corresponding availability category label, and the predicted value corresponding to the availability feature and its corresponding availability category label.

[0060] The second loss function may be an L2 loss function determined based on the predicted value corresponding to the feature map of the first-person perspective and the corresponding availability category label, and the predicted value corresponding to the availability feature and the corresponding availability category label. The L2 loss function, also known as the minimum square error, minimizes the sum of the squares of the difference between the target value (i.e., the availability category label) and the predicted value.

[0061] The above-mentioned third loss function can be determined based on the following steps: multiplying the predicted value corresponding to the feature map of the first-person perspective with its transpose to obtain a first co-occurrence matrix; and multiplying the predicted value corresponding to the availability feature with its transpose to obtain a second co-occurrence matrix; and determining the third loss function based on the cross entropy of the first co-occurrence matrix and the second co-occurrence matrix.

[0062] In one example, an affordance symbiosis retention strategy is introduced to enhance the network's ability to perceive and locate affordance areas.

[0063] exist Figure 3 In the above example, the predicted values ​​of the two branches are multiplied by their transpose to obtain the co-occurrence matrices of the two branches and calculate the aligned cross entropy loss L. ACP , the symbiosis matrix of the first-person perspective can be aligned with the symbiosis relationship extracted from the third-person perspective.

[0064] The predicted value s corresponding to the feature map of the third-person perspective and the predicted value g corresponding to the feature map of the first-person perspective can be calculated as follows:

[0065]

[0066] P=pp T , Q=qq T

[0067]

[0068] Among them, P is the transposition of the feature map corresponding to the third-person perspective, and Q is the transposition of the feature map corresponding to the first-person perspective.

[0069] The method for generating an availability detection model provided in an embodiment of the present application trains an initial availability detection model to obtain an availability detection model using a target loss function determined by at least one of the following: a first loss function, a second loss function, and a third loss function to train the availability detection model.

[0070] In some optional implementations of the present embodiment, a set of feature maps from a third-person perspective are input into a preset availability feature mining model to obtain availability features, including: reshaping a set of feature maps from a third-person perspective to generate a multidimensional matrix, wherein the rows of the multidimensional matrix are the same as the channel dimensions of a set of feature maps from a third-person perspective, and the columns of the multidimensional matrix are N*h*w, N is the number of images from a set of third-person perspectives, and h and w are the length and width of a set of feature maps from a third-person perspective, respectively; decomposing the multidimensional matrix into a basis matrix W and a coefficient matrix H; iteratively updating the basis matrix W and the coefficient matrix H to perform a non-negative matrix decomposition of the multidimensional matrix into the product of an updated basis matrix W and an updated coefficient matrix H; reshaping the updated basis matrix W and the updated coefficient matrix H to obtain feature maps after the reshape operation; and performing feature fusion based on a set of feature maps from a third-person perspective and the feature maps after the reshape operation to obtain availability features.

[0071] In this implementation, a set of feature maps from a third-person perspective is reshaped to generate a multidimensional matrix, wherein the rows of the multidimensional matrix are the same as the channel dimensions of the set of feature maps from a third-person perspective, and the columns of the multidimensional matrix are N*h*w, where N is the number of images from a third-person perspective, and h and w are the length and width of a set of feature maps from a third-person perspective, respectively; then, the multidimensional matrix is ​​decomposed into a base matrix W and a coefficient matrix H; the base matrix W and the coefficient matrix H are iteratively updated to perform non-negative matrix decomposition (NMF, a matrix decomposition optimization algorithm) of the multidimensional matrix as the product of an updated base matrix W and an updated coefficient matrix H; then, the updated base matrix W and the updated coefficient matrix H are reshaped to obtain a feature map after the reshape operation; then, feature fusion is performed based on a set of feature maps from a third-person perspective and the feature map after the reshape operation to obtain availability features.

[0072] In an example, the interaction between a user and an object is represented as X=M+E, where X represents the user-object interaction from a third-person perspective, M represents the affordance feature, and E represents the difference caused by user habits. The low-rank matrix decomposition and reconstruction technology can be used to minimize the difference E caused by user habits and obtain the affordance feature M determined by the intrinsic structure of the object.

[0073] exist Figure 4In the above, the feature map of the third-person perspective extracted is reduced in dimension through a convolutional layer, and then this set of features is concatenated and reshaped into a two-dimensional matrix. The rows of the matrix are consistent with the channel dimension of the feature map, and the columns of the matrix are N*h*w, where N is the number of third-person images, h and w are the length and width of the feature map, respectively, and reshaped into a c×Nhw matrix. Then, M is decomposed into two matrices W and H, that is, X=M+E=WH+E. Using the non-negative matrix decomposition technology, W and H are iteratively updated, and finally the optimized result M=WH is obtained. It is then reshaped to the original N*c*h*w, and then mapped to the residual space through a layer of convolution, and then added to the original input (i.e., the feature map of the third-person perspective) to obtain the affordance feature.

[0074] In some optional implementations of this embodiment, the second loss function is determined based on the following steps: inputting the feature map of the first-person perspective into a preset mapping layer, mapping the feature map of the first-person perspective into a feature space corresponding to a set of feature maps of the third-person perspective; inputting a set of feature maps of the third-person perspective and the feature map of the first-person perspective into preset convolutional layers, respectively, and passing the obtained results through pooling layers, respectively; and determining the L2 loss function based on the pooled results.

[0075] In this implementation, the execution subject may input a feature map of the first-person perspective into a preset mapping layer, map the feature map of the first-person perspective into a feature space corresponding to a set of feature maps of the third-person perspective; input a set of feature maps of the third-person perspective and a feature map of the first-person perspective into a preset convolutional layer respectively, and pass the obtained results through a pooling layer respectively; determine the L2 loss function according to the pooled results. The preset mapping layer may be used to map the feature map of the first-person perspective into a feature space corresponding to a set of feature maps of the third-person perspective, and the preset mapping layer may include at least two convolutional layers. The preset convolutional layer may be a convolutional layer in a preset mapping layer or a convolutional layer outside a preset mapping layer.

[0076] In one example, the feature map of the first-person perspective is fed into a mapping layer consisting of two convolutional layers to map it to the features of the third-person perspective. Figure 1Subsequently, a set of feature maps from the third-person perspective and a set of feature maps from the first-person perspective are fed into a convolutional layer respectively. For the output from the third-person perspective, N features are passed through a global average pooling layer respectively, and then the feature representation of the third-person perspective is obtained after averaging. For the output from the first-person perspective, it is fed into a global average pooling layer. In order to ensure that the knowledge of the availability features extracted from the feature map of the third-person perspective is transferred to the feature map of the first-person perspective, the features of the two branches (i.e., the feature map of the first-person perspective and a set of feature maps of the third-person perspective) are represented with an L2 loss constraint so that the feature map representation of the first-person perspective can be aligned with the feature map representation of the third-person perspective in the feature space. After that, the feature map representations of the two branches are fed into the fully connected layer respectively, and the classification results (i.e., the availability categories) are obtained respectively.

[0077] In this implementation, the feature map of the first-person perspective can be mapped to the feature space corresponding to the feature map of the third-person perspective, and then a group of feature maps of the third-person perspective and feature maps of the first-person perspective are respectively input into the preset convolutional layers, and the obtained results are respectively passed through the pooling layer to determine the L2 loss function based on the pooled results.

[0078] In some optional implementations of this embodiment, the target loss function is determined using at least one of the following: a first loss function and its corresponding weight, a second loss function and its corresponding weight, and a third loss function and its corresponding weight.

[0079] In this implementation, the above-mentioned execution entity can determine the target loss function by using at least one of the first loss function and its corresponding weight, the second loss function and its corresponding weight, and the third loss function and its corresponding weight.

[0080] It should be noted that the above-mentioned first loss function and its corresponding weight, the second loss function and its corresponding weight, and the third loss function and its corresponding weight can be set according to the prediction accuracy of the availability detection model or by the user according to specific work requirements.

[0081] In one example, the target loss function may be determined based on the first loss function and its corresponding weight, the second loss function and its corresponding weight, and the third loss function and its corresponding weight, wherein the weight corresponding to the first loss function may be 1, the weight corresponding to the second loss function may be 0.5, and the weight corresponding to the third loss function may be 0.5.

[0082] In this implementation, the target loss function can be determined based on the first loss function and its corresponding weight, the second loss function and its corresponding weight, and the third loss function and its corresponding weight.

[0083] In some optional implementations of the present embodiment, obtaining a feature map of the first-person perspective and a set of feature maps of the third-person perspective includes: obtaining an image of the first-person perspective and a set of images of the third-person perspective, wherein the image of the first-person perspective and the set of images of the third-person perspective include objects of the same category; inputting the image of the first-person perspective and the set of images of the third-person perspective into a preset feature extractor, respectively, to obtain a feature map of the first-person perspective and a set of feature maps of the third-person perspective.

[0084] In this implementation, the execution subject may obtain an image from a first-person perspective and a group of images from a third-person perspective, and input the image from the first-person perspective and the group of images from the third-person perspective into a preset feature extractor, respectively, to obtain a feature map from the first-person perspective corresponding to the image from the first-person perspective, and a group of feature maps from the third-person perspective corresponding to the group of images from the third-person perspective; wherein the image from the first-person perspective and the group of images from the third-person perspective include objects of the same category. The image from the first-person perspective may be an image captured from the user's own perspective (i.e., the first-person perspective). The image from the third-person perspective may be an image captured from the third-person perspective.

[0085] In one example, the image is captured from a first-person perspective.

[0086] It should be noted that the number of a set of third-person perspective images can be set according to the prediction accuracy of the affordance detection model or by the user according to work requirements.

[0087] In this implementation, the above-mentioned execution entity can perform feature extraction on the first-person perspective image and a group of third-person perspective images respectively according to a preset feature extractor to obtain a feature map of the first-person perspective and a feature map of the third-person perspective.

[0088] Continue to refer Figure 5 , which shows an application scenario diagram of the method for generating an availability detection model according to the present application. In the application scenario diagram, the method for generating an availability detection model may include:

[0089] In the first step, a set of third-person perspective images and first-person perspective images are input into a preset feature extractor.

[0090] In an embodiment, a group of third-person perspective images and first-person perspective images are respectively input into a preset feature extractor, and the objects (i.e., subjects) contained in the images of the two branches belong to the same affordance category. The purpose is to extract affordance features from the interaction of a group of feature maps from the third-person perspective, and then migrate them to the feature maps from the first-person perspective.

[0091] In one example, the preset feature extractor may be Resnet50, and may also include but is not limited to VGG series, DenseNet series, GoogleNet, Inception, ShuffleNet, etc.

[0092] In the second step, the feature map of the third-person perspective is input into the preset affordance feature mining model to extract affordance-related features from various human-object interactions under the third-person perspective.

[0093] In one example: the user-object interaction is represented as X=M+E, where X represents the user-object interaction from the third-person perspective, M represents the affordance feature, and E represents the difference caused by user habits. The low-rank matrix decomposition and reconstruction technology can be used to minimize the difference E caused by user habits and obtain the affordance feature M determined by the intrinsic structure of the object. After that, the feature map extracted from the third-person perspective is reduced in dimension through a convolutional layer, and then this set of features is concatenated and reshaped into a two-dimensional matrix. The rows of the matrix are consistent with the channel dimension of the feature map, and the columns of the matrix are N*h*w, where N is the number of third-person images, h and w are the length and width of the feature map, respectively, and reshaped into a c×Nhw matrix. Then, M is decomposed into two matrices W and H, that is, X=M+E=WH+E. Using the non-negative matrix decomposition technology, W and H are iteratively updated, and finally the optimized result M=WH is obtained. It is then reshaped to the original N*c*h*w, and then mapped to the residual space through a layer of convolution, and then added to the original input (i.e., the feature map of the third-person perspective) to obtain the affordance feature.

[0094] Here, the update of W and H can adopt VQ, CD, etc.

[0095] The third step is to input the feature map of the first-person perspective into the preset mapping layer and map it to the feature map of the third-person perspective. Figure 1 Then, the feature map of the third-person perspective and the feature map of the first-person perspective are sent to the preset convolution layer respectively. For the output of the third-person perspective, N features are passed through the global average pooling layer respectively, and then the feature representation of the third-person perspective is obtained after averaging. For the output of the first-person perspective, it is sent to the global average pooling layer. In order to ensure that the knowledge related to the availability extracted by the third-person perspective is transferred to the first-person perspective, we constrain the feature representation of the two branches with an L2 loss so that the feature representation of the first-person can be aligned with the feature representation of the third-person in the feature space. Then we send the feature representation of the two branches to the fully connected layer respectively to obtain the classification results.

[0096] The fourth step is to enhance the network’s ability to perceive and locate affordance areas through affordance symbiotic relationship retention strategies.

[0097] In the fifth step, in order to detect the affordance area of ​​objects from the third-person perspective images, an Affordance Grounding Dataset (AGD20K) was collected for training and evaluating the affordance detection model.

[0098] Continue to refer Figure 6 , which shows a flowchart 600 of an embodiment of a method for identifying an affordance category according to the present application. The method for identifying an affordance category may include the following steps:

[0099] Step 601: Obtain an image of the first-person perspective to be predicted.

[0100] In this embodiment, the execution subject of the method for identifying the affordance category (eg Figure 1 The server 104 shown in FIG. Figure 1 The network 103 shown) from the terminal device (eg Figure 1 The servers 101 and 102 shown in the figure) or obtain the first-person perspective image to be predicted from the local; or, the execution subject of the method for generating the availability detection model (for example Figure 1 The terminal devices 101 and 102 shown in the figure can obtain the first-person perspective image to be predicted locally. The first-person perspective image to be predicted can be used to identify its availability category label by the availability detection model.

[0101] Step 602: Input the image of the first-person perspective to be predicted into a preset feature extractor to obtain a feature map of the first-person perspective to be predicted.

[0102] In this embodiment, the execution subject may input the first-person perspective image into a preset feature extractor to obtain a feature map of the first-person perspective to be predicted, for example, a Resnet50 feature extractor.

[0103] Step 603: Input the feature map of the first-person perspective to be predicted into a pre-trained availability detection model to obtain the availability category corresponding to the image of the first-person perspective to be predicted.

[0104] In this embodiment, the execution subject may input the feature map of the first-person perspective to be predicted into a pre-trained availability detection model to obtain the availability category corresponding to the image of the first-person perspective to be predicted.

[0105] In this embodiment, the above-mentioned pre-trained availability detection model can be Figure 2The method shown in Figure 2 is used to train the affordance detection model.

[0106] Continue to refer Figure 7 , which shows an application scenario diagram of the method for identifying an availability category according to the present application. In the application scenario diagram, the method for identifying an availability category may include:

[0107] The first-person perspective image to be predicted and the affordance category label are input into the affordance detection model, and the Class Activation Map (CAM) visualization technology, such as Grad-CAM, Grad-CAM++, XGrad-CAM, etc., is used to output the area on the object (i.e., the object in the first-person perspective image to be predicted) that can complete the interaction. The specific steps are as follows:

[0108] In the first step, the first-person perspective image to be predicted and the affordance label are input, and the relevant area with affordance in the object is output.

[0109] In the second step, the first-person perspective image and the affordance label (i.e., the affordance category label) are fed into the trained affordance detection model, and then the class activation technique is used to obtain the affordance category of the predicted affordance area. For example, the feature map of the first-person perspective to be predicted is sequentially passed through the mapping layer, the convolution layer, the average pooling layer, and the fully connected layer (classifier) ​​to output the affordance category, and then the class activation technique is used to output the affordance category of the predicted affordance area.

[0110] Further references Figure 8 As an implementation of the methods shown in the above figures, the present application discloses an embodiment of a device for generating an availability detection model. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0111] like Figure 8As shown, an embodiment of the present application provides a device 800 for generating an availability detection model, and the device 800 includes: a feature map acquisition module 801, a feature acquisition module 802, a prediction value acquisition module 803 and a model training module 804. Among them, the feature map acquisition module 801 is configured to acquire a feature map of the first-person perspective and a set of feature maps of the third-person perspective; the feature acquisition module 802 is configured to input a set of feature maps of the third-person perspective into a preset availability feature mining model to obtain availability features; the prediction value acquisition module 803 is configured to input the feature map of the first-person perspective and the availability features into the initial availability detection model respectively, and obtain the prediction value corresponding to the feature map of the first-person perspective and the prediction value corresponding to the availability features; the model training module 804 is configured to train the initial availability detection model using at least one of the following target loss functions to obtain the availability detection model: a first loss function, a second loss function, and a third loss function, wherein the first loss function is based on The first loss function is determined in the following steps: a first loss function is determined according to the predicted value corresponding to the feature map of the first-person perspective and its corresponding availability category label, and the cross entropy of the predicted value corresponding to the availability feature and its corresponding availability category label; the second loss function is determined based on the following steps: an L2 loss function is determined according to the predicted value corresponding to the feature map of the first-person perspective and its corresponding availability category label, and the predicted value corresponding to the availability feature and its corresponding availability category label; the third loss function is determined based on the following steps: a first sym-occurrence matrix is ​​obtained by multiplying the predicted value corresponding to the feature map of the first-person perspective and its transpose; and a second sym-occurrence matrix is ​​obtained by multiplying the predicted value corresponding to the availability feature and its transpose; the third loss function is determined based on the cross entropy of the first sym-occurrence matrix and the second sym-occurrence matrix.

[0112] In the present embodiment, in the device 800 for generating an availability detection model, the specific processing of the feature map acquisition module 801, the feature acquisition module 802, the prediction value acquisition module 803 and the model training module 804 and the technical effects thereof can be referred to in Figure 2 This corresponds to step 201 to step 204 in the embodiment.

[0113] In some embodiments, the feature acquisition module 802 is further configured to: reshape a set of feature maps from a third-person perspective to generate a multidimensional matrix, wherein the rows of the multidimensional matrix are the same as the channel dimensions of the set of feature maps from a third-person perspective, and the columns of the multidimensional matrix are N*h*w, N is the number of images from a third-person perspective, and h and w are the length and width of a set of feature maps from a third-person perspective, respectively; decompose the multidimensional matrix into a basis matrix W and a coefficient matrix H; iteratively update the basis matrix W and the coefficient matrix H to perform a non-negative matrix decomposition of the multidimensional matrix into the product of an updated basis matrix W and an updated coefficient matrix H; reshape the updated basis matrix W and the updated coefficient matrix H to obtain a feature map after the reshape operation; perform feature fusion based on a set of feature maps from a third-person perspective and the feature map after the reshape operation to obtain availability features.

[0114] In some embodiments, the device also includes: a feature mapping module, configured to input a feature map of the first-person perspective into a preset mapping layer, and map the feature map of the first-person perspective into a feature space corresponding to a set of feature maps of the third-person perspective; a feature processing module, configured to input a set of feature maps of the third-person perspective and a feature map of the first-person perspective into preset convolutional layers, respectively, and pass the obtained results through pooling layers, respectively; a function determination module, configured to determine the L2 loss function based on the pooled result.

[0115] In some embodiments, the model training module 804 is further configured to: train the initial availability detection model using a target loss function determined by at least one of the following to obtain an availability detection model: a first loss function and its corresponding weight, a second loss function and its corresponding weight, and a third loss function and its corresponding weight.

[0116] In some embodiments, the feature map acquisition module 801 is further configured to: acquire an image from a first-person perspective and a group of images from a third-person perspective, wherein the image from the first-person perspective and the group of images from the third-person perspective include objects of the same category; input the image from the first-person perspective and the group of images from the third-person perspective into a preset feature extractor, respectively, to obtain a feature map from the first-person perspective, and a group of feature maps from the third-person perspective.

[0117] Further references Fig. 9 As an implementation of the methods shown in the above figures, the present application discloses an embodiment of a device for generating an availability detection model. Figure 6 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0118] like Fig. 9As shown, an embodiment of the present application provides a device 900 for generating an availability detection model, and the device 900 includes: an image acquisition module 901, a feature map extraction module 902, and a category prediction module 903. Among them, the image acquisition module 901 is configured to acquire an image of the first-person perspective to be predicted; the feature map extraction module 902 is configured to input the image of the first-person perspective to be predicted into a preset feature extractor to obtain a feature map of the first-person perspective to be predicted; the category prediction module 903 is configured to input the feature map of the first-person perspective to be predicted into a pre-trained availability detection model to obtain an availability category corresponding to the image of the first-person perspective to be predicted.

[0119] In the apparatus 900 for generating an availability detection model, the specific processing of the image acquisition module 901, the feature map extraction module 902, and the category prediction module 903 and the technical effects thereof can be referred to in Figure 6 This corresponds to steps 601 to 603 in the embodiment.

[0120] like Fig.10 , is a block diagram of an electronic device for a method of generating an availability detection model or a method of identifying an availability category according to an embodiment of the present application. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein.

[0121] like Fig.10 As shown, the electronic device includes: one or more processors 1001, a memory 1002, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are interconnected using different buses and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the electronic device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In other embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple electronic devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Fig.10 A processor 1001 is taken as an example.

[0122] The memory 1002 is the non-transitory computer-readable storage medium provided in the present application. The memory stores instructions executable by at least one processor to enable at least one processor to execute the method for generating an availability detection model or the method for identifying an availability category provided in the present application. The non-transitory computer-readable storage medium of the present application stores computer instructions, which are used to enable a computer to execute the method for generating an availability detection model or the method for identifying an availability category provided in the present application.

[0123] The memory 1002 is a non-transient computer-readable storage medium that can be used to store non-transient software programs, non-transient computer executable programs and modules, such as the program instructions / modules corresponding to the method for generating an affordance detection model or the method for identifying an affordance category in the embodiments of the present application (for example, the attached Figure 8 The feature map acquisition module 801, feature acquisition module 802, prediction value acquisition module 803 and model training module 804 shown in the figure; or Fig. 9 The processor 1001 executes various functional applications and data processing of the server by running the non-transient software programs, instructions and modules stored in the memory 1002, that is, the method for generating an availability detection model or the method for identifying an availability category in the above method embodiment is implemented.

[0124] The memory 1002 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the blockchain-based information processing electronic device, etc. In addition, the memory 1002 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory 1002 may optionally include a memory remotely arranged relative to the processor 1001, and these remote memories may be connected to the blockchain-based information processing electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0125] The electronic device of the method for generating an affordance detection model or the method for identifying an affordance category may further include: an input device 1003 and an output device 1004. The processor 1001, the memory 1002, the input device 1003 and the output device 1004 may be connected via a bus or other means. Fig.10 The example of connecting through bus is taken in the following.

[0126] The input device 1003 can receive input digital or character information, and generate key signal input related to user settings and function control of blockchain-based information processing electronic devices, such as touch screens, keypads, mice, trackpads, touchpads, indicator rods, one or more mouse buttons, trackballs, joysticks and other input devices. The output device 1004 may include a display device, an auxiliary lighting device (e.g., LED) and a tactile feedback device (e.g., a vibration motor), etc. The display device may include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display and a plasma display. In some embodiments, the display device may be a touch screen.

[0127] Various implementations of the systems and techniques described herein can be realized in digital electronic circuit systems, integrated circuit systems, dedicated ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0128] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for programmable processors and can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or means (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0129] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0130] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0131] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

[0132] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this application can be executed in parallel, sequentially or in different orders, as long as the expected results of the technical solution disclosed in this application can be achieved, and this document is not limited here.

[0133] The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.

Claims

1. A method for generating an affordance detection model, comprising: Obtain a feature map of a first-person perspective and a set of feature maps of a third-person perspective; Inputting the set of third-person perspective feature maps into a preset affordance feature mining model to obtain affordance features; Inputting the feature map of the first-person perspective and the availability feature into an initial availability detection model respectively, to obtain a prediction value corresponding to the feature map of the first-person perspective and a prediction value corresponding to the availability feature; The initial availability detection model is trained using a target loss function determined by at least one of the following: a first loss function, a second loss function, and a third loss function, wherein the first loss function is determined based on the following steps: determining the first loss function based on the predicted value corresponding to the feature map of the first-person perspective and the corresponding availability category label, and the cross entropy of the predicted value corresponding to the availability feature and the corresponding availability category label; the second loss function is determined based on the following steps: determining the L2 loss function based on the predicted value corresponding to the feature map of the first-person perspective and the corresponding availability category label, and the predicted value corresponding to the availability feature and the corresponding availability category label; the third loss function is determined based on the following steps: obtaining a first symbiosis matrix by multiplying the predicted value corresponding to the feature map of the first-person perspective and its transpose; and obtaining a second symbiosis matrix by multiplying the predicted value corresponding to the availability feature and its transpose; and determining the third loss function based on the cross entropy of the first symbiosis matrix and the second symbiosis matrix.

2. The method according to claim 1, wherein: The step of inputting the set of third-person perspective feature maps into a preset affordance feature mining model to obtain affordance features includes: Reshape the set of feature maps from the third-person perspective to generate a multidimensional matrix, wherein the rows of the multidimensional matrix are the same as the channel dimensions of the set of feature maps from the third-person perspective, and the columns of the multidimensional matrix are N*h*w, where N is the number of images from a set of third-person perspectives, and h and w are the length and width of the set of feature maps from the third-person perspectives, respectively; Decomposing the multidimensional matrix into a basis matrix W and a coefficient matrix H; Iteratively updating the base matrix W and the coefficient matrix H to perform non-negative matrix decomposition on the multidimensional matrix into the product of the updated base matrix W and the updated coefficient matrix H; Reshape the updated basis matrix W and the updated coefficient matrix H to obtain the feature map after the reshape operation; Availability features are obtained by performing feature fusion based on a set of feature maps from the third-person perspective and the feature maps after the reshape operation.

3. The method according to claim 1, wherein: The second loss function is determined based on the following steps: Inputting the feature map of the first-person perspective into a preset mapping layer, and mapping the feature map of the first-person perspective into a feature space corresponding to the set of feature maps of the third-person perspective; Inputting the set of feature maps of the third-person perspective and the feature map of the first-person perspective into a preset convolutional layer respectively, and passing the obtained results through a pooling layer respectively; According to the pooled result, the L2 loss function is determined.

4. The method according to any one of claims 1 to 3, wherein: The initial availability detection model is trained using a target loss function determined by at least one of the following to obtain an availability detection model: a first loss function, a second loss function, and a third loss function, including: The initial availability detection model is trained using at least one of the following target loss functions to obtain an availability detection model: a first loss function and its corresponding weight, a second loss function and its corresponding weight, and a third loss function and its corresponding weight.

5. The method according to claim 1, wherein: The step of obtaining a feature map of a first-person perspective and a set of feature maps of a third-person perspective includes: Acquire an image from a first-person perspective and a group of images from a third-person perspective, wherein the image from the first-person perspective and the group of images from the third-person perspective include objects of the same category; The image from the first-person perspective and the group of images from the third-person perspective are respectively input into a preset feature extractor to obtain a feature map of the first-person perspective and a feature map of the group of images from the third-person perspective.

6. A method for identifying affordance categories, comprising: Obtain an image of the first-person perspective to be predicted; Inputting the image of the first-person perspective to be predicted into a preset feature extractor to obtain a feature map of the first-person perspective to be predicted; The feature map of the first-person perspective to be predicted is input into the availability detection model as described in any one of claims 1 to 5 to obtain the availability category corresponding to the image of the first-person perspective to be predicted.

7. An apparatus for generating an affordance detection model, comprising: A feature map acquisition module is configured to acquire a feature map of a first-person perspective and a set of feature maps of a third-person perspective; A feature acquisition module is configured to input the set of feature maps of the third-person perspective into a preset affordance feature mining model to obtain affordance features; a prediction value obtaining module, configured to input the feature map of the first-person perspective and the availability feature into an initial availability detection model respectively, to obtain a prediction value corresponding to the feature map of the first-person perspective and a prediction value corresponding to the availability feature; The model training module is configured to train the initial availability detection model using a target loss function determined by at least one of the following to obtain an availability detection model: a first loss function, a second loss function, and a third loss function, wherein the first loss function is determined based on the following steps: determining the first loss function based on the predicted value corresponding to the feature map of the first-person perspective and the corresponding availability category label, and the cross entropy of the predicted value corresponding to the availability feature and the corresponding availability category label; the second loss function is determined based on the following steps: determining the L2 loss function based on the predicted value corresponding to the feature map of the first-person perspective and the corresponding availability category label, and the predicted value corresponding to the availability feature and the corresponding availability category label; the third loss function is determined based on the following steps: obtaining a first symbiosis matrix by multiplying the predicted value corresponding to the feature map of the first-person perspective and its transpose; and obtaining a second symbiosis matrix by multiplying the predicted value corresponding to the availability feature and its transpose; and determining the third loss function based on the cross entropy of the first symbiosis matrix and the second symbiosis matrix.

8. The device according to claim 7, wherein: The feature acquisition module is further configured to: Reshape the set of feature maps from the third-person perspective to generate a multidimensional matrix, wherein the rows of the multidimensional matrix are the same as the channel dimensions of the set of feature maps from the third-person perspective, and the columns of the multidimensional matrix are N*h*w, where N is the number of images from a set of third-person perspectives, and h and w are the length and width of the set of feature maps from the third-person perspectives, respectively; Decomposing the multidimensional matrix into a basis matrix W and a coefficient matrix H; Iteratively updating the base matrix W and the coefficient matrix H to perform non-negative matrix decomposition on the multidimensional matrix into the product of the updated base matrix W and the updated coefficient matrix H; Reshape the updated basis matrix W and the updated coefficient matrix H to obtain the feature map after the reshape operation; Availability features are obtained by performing feature fusion based on a set of feature maps from the third-person perspective and the feature maps after the reshape operation.

9. The device according to claim 7, further comprising: a feature mapping module, configured to input the feature map of the first-person perspective into a preset mapping layer, and map the feature map of the first-person perspective into a feature space corresponding to the feature map of the set of third-person perspectives; A feature processing module, configured to input the set of feature maps of the third-person perspective and the feature map of the first-person perspective into a preset convolution layer respectively, and pass the obtained results through a pooling layer respectively; The function determination module is configured to determine the L2 loss function according to the pooled result.

10. The device according to any one of claims 7 to 9, wherein: The model training module is further configured to: The initial availability detection model is trained using at least one of the following target loss functions to obtain an availability detection model: a first loss function and its corresponding weight, a second loss function and its corresponding weight, and a third loss function and its corresponding weight.

11. The device according to claim 7, wherein: The feature map acquisition module is further configured to: Acquire an image from a first-person perspective and a group of images from a third-person perspective, wherein the image from the first-person perspective and the group of images from the third-person perspective include objects of the same category; The image from the first-person perspective and the group of images from the third-person perspective are respectively input into a preset feature extractor to obtain a feature map of the first-person perspective and a feature map of the group of images from the third-person perspective.

12. An apparatus for identifying an affordance category, comprising: An image acquisition module, configured to acquire an image of a first-person perspective to be predicted; A feature map extraction module is configured to input the image of the first-person perspective to be predicted into a preset feature extractor to obtain a feature map of the first-person perspective to be predicted; The category prediction module is configured to input the feature map of the first-person perspective to be predicted into the availability detection model as described in any one of claims 1-5 to obtain the availability category corresponding to the image of the first-person perspective to be predicted.

13. An electronic device, comprising: one or more processors; a storage device having one or more programs stored thereon; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

14. A non-transitory computer-readable storage medium storing computer instructions, having a computer program stored thereon, wherein: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images