Method for training model for detecting interaction, device for detecting interaction and storage medium
By introducing a combined architecture of multi-decoder and information fusion modules in the HOID model, the problem of insufficient performance of traditional detection algorithms in direct contact interaction scenarios is solved, and the accuracy of human-item interaction detection is significantly improved.
Patent Information
- Application Number
- CN202311160188.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-09-07
- Publication Date
- 2025-03-07
AI Technical Summary
The traditional human-item interaction detection algorithm has insufficient performance in scenarios where people and objects have direct contact and interaction, making it difficult to improve detection accuracy.
An improved HOID model architecture is proposed, including a visual feature extractor, a first decoder, a second decoder, an information fusion module and a third decoder. Through the combination of multiple decoders and information fusion modules, the model's detection ability of human-item interaction is improved.
Through the improved model architecture, the accuracy of human-item interaction detection is significantly improved, especially in scenarios where people and objects have direct contact.
Smart Images
Figure CN119580025A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure generally relates to computer vision, and more particularly, to a method for training a model for detecting human-object interaction, an apparatus for detecting human-object interaction, and a storage medium. Background Art
[0002] With the development of computer science and artificial intelligence, it is becoming more and more common and effective to use computers to run artificial intelligence models based on neural networks to realize information processing. Computer vision is an important application field of artificial intelligence models.
[0003] One branch of computer vision technology is human-object interaction detection (HOID: Human-Object Interaction Detection; or HOI detection). Human-object interaction detection is a technology that uses computer vision to determine whether there is an interaction between a person and an object in an image or video sequence and the type of interaction. HOID includes: person detection, interaction detection, and object detection. Person detection includes detecting the position of the person participating in the interaction in the image. Interaction detection includes detecting the type of interaction involved in the interaction between a person and an object in the image. Object detection includes detecting the classification and position of the objects involved in the interaction between a person and an object in the image. The HOID model based on a neural network can give the position of the person, the position of the object, the classification of the object, and the classification of the interaction in the detection image. Human-object interaction detection can be widely used in the fields of personnel behavior analysis, shopper interest analysis, supermarket anti-theft detection, and athlete foul detection. For example, in the task of shopper interest analysis, the human-object interaction detection model can be used to detect the action of "grabbing goods from the shelf" in supermarket surveillance videos.
[0004] Non-patent literature 1: Aixi Zhang et al., "Mining the Benefits of Two-stage and One-stage HOI Detection", 35 th Conference on Neural Information Processing Systems (NeurIPS2021), Sydney, Australia, 12 pages in total.
[0005] In non-patent document 1, Aixi Zhang proposed a cascade disentangling network (CDN) for HOID, in which the proposed one-stage network architecture disentangles human-object detection and interaction classification in a cascade manner. The CDN includes a human-object pair decoder and an interaction decoder.
[0006] In scenarios where people and objects have direct contact and interaction, the performance of traditional human-object interaction detection algorithms needs to be improved. Summary of the invention
[0007] A brief overview of the present disclosure will be given below in order to provide a basic understanding of certain aspects of the present disclosure. It should be understood that this overview is not an exhaustive overview of the present disclosure. It is not intended to identify key or important parts of the present disclosure, nor is it intended to limit the scope of the present disclosure. Its purpose is simply to give certain concepts in a simplified form as a prelude to a more detailed description discussed later.
[0008] The inventors studied the architecture of the conventional HOID model and proposed a HOID model with an improved architecture, in the hope of improving the performance of the model in some aspects.
[0009] According to one aspect of the present disclosure, a computer-implemented method for training a model for detecting human-object interactions is provided. The model includes a visual feature extractor, a first decoder, a second decoder, an information fusion module, and a third decoder. The method includes: extracting visual features from a sample image using a visual feature extractor, wherein the sample image includes an image region indicating an object contacted by a person; using a first decoder to determine a predicted position of an object, a predicted classification of an object, and a predicted action classification of an action performed by a person on an object based on visual features and an initial query vector, and generating a first query vector; using a second decoder to determine a predicted position of a local body of a person in contact with an object based on the first query vector and visual features, and generating a second query vector related to the local body; using an information fusion module to generate a third query vector by fusing the first query vector and the second query vector; using a third decoder to determine a predicted position of a person based on visual features and the third query vector; and adjusting parameters of the model based on the predicted position of the object, the predicted classification of the object, the predicted action classification, and the predicted position of the person.
[0010] According to one aspect of the present disclosure, a device for detecting human-object interaction is provided. The device includes: a memory having instructions stored thereon; and at least one processor connected to the memory and configured to execute the instructions to: extract visual features from a detection image using a visual feature extractor, wherein the detection image includes an image region indicating an object contacted by a person; determine a predicted position of the object, a predicted classification of the object, and a predicted action classification of an action performed by a person on the object based on the visual features and an initial query vector using a first decoder, and generate a first query vector; generate a second query vector related to a local body of the person contacting the object using a second decoder based on the first query vector and the visual features; generate a third query vector by fusing the first query vector and the second query vector using an information fusion module; and determine a predicted position of the person based on the visual features and the third query vector using a third decoder.
[0011] According to another aspect of the present disclosure, a computer-readable storage medium storing a program is provided. When the program is executed by a computer, the computer implements the following operations: extracting visual features from a detection image using a visual feature extractor, wherein the detection image includes an image region indicating an object contacted by a person; using a first decoder to determine a predicted position of an object, a predicted classification of an object, and a predicted action classification of an action performed by a person on an object based on the visual features and an initial query vector, and generating a first query vector; using a second decoder to generate a second query vector related to a local body of a person contacting the object based on the first query vector and the visual features; using an information fusion module to generate a third query vector by fusing the first query vector and the second query vector; and using a third decoder to determine a predicted position of a person based on the visual features and the third query vector.
[0012] The beneficial effects of the method, device, and storage medium of the present disclosure include: improving the accuracy of human-object interaction detection. BRIEF DESCRIPTION OF THE DRAWINGS
[0013] The following description of the embodiments of the present disclosure will help to more easily understand the above and other purposes, features and advantages of the present disclosure with reference to the accompanying drawings. The accompanying drawings are only for the purpose of illustrating the principles of the present disclosure. The sizes and relative positions of the elements in the accompanying drawings are not necessarily drawn to scale. The same reference numerals may represent the same features. In the accompanying drawings:
[0014] Figure 1 An exemplary flow chart of a method for training a model for detecting human-object interaction according to an embodiment of the present disclosure is shown;
[0015] Figure 2 An exemplary architecture of a model for human-object interaction detection according to one embodiment of the present disclosure is shown;
[0016] Figure 3 An exemplary flow chart of a method for detecting human-object interaction according to an embodiment of the present disclosure is shown;
[0017] Figure 4 An exemplary block diagram showing an apparatus for detecting human-object interaction according to an embodiment of the present disclosure; and
[0018] Figure 5 An exemplary block diagram of an information processing device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION
[0019] Exemplary embodiments of the present disclosure are described below in conjunction with the accompanying drawings. For the sake of clarity and conciseness, not all features of the actual embodiments are described in the specification. However, it should be understood that many implementation-specific decisions may be made in the process of developing any such actual embodiments in order to achieve the specific goals of the developer, and these decisions may vary from embodiment to embodiment.
[0020] It is also necessary to explain here that, in order to avoid obscuring the contents of the present disclosure due to unnecessary details, only the device structure closely related to the scheme according to the contents of the present disclosure is shown in the accompanying drawings, while other details that are not closely related to the contents of the present disclosure are omitted.
[0021] It should be understood that the present disclosure is not limited to the described implementation forms due to the following description with reference to the accompanying drawings. In this article, where feasible, the embodiments can be combined with each other, features between different embodiments can be replaced or borrowed, and one or more features can be omitted in one embodiment.
[0022] Computer program code for performing operations of various aspects of the embodiments of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, Smalltalk, C++, and the like, and conventional procedural programming languages such as the "C" programming language or similar programming languages.
[0023] The method of the present disclosure can be implemented by a circuit having a corresponding functional configuration, wherein the circuit includes a circuit for a processor.
[0024] One aspect of the present disclosure relates to a method for training a model intM for detecting human-object interactions. The method can be implemented by a computer. Depending on the training set, the trained model intM can be used for personnel behavior analysis, shopper interest analysis, supermarket anti-theft detection, and athlete foul detection, etc. Figure 1An exemplary flow chart of a method 100 for training a model intM for detecting human-object interaction according to an embodiment of the present disclosure is shown. Figure 2 The structure of the model intM for human-object interaction detection according to an embodiment of the present disclosure is shown, wherein the output and output of each component during model training are also exemplarily shown.
[0025] like Figure 2 As shown in , the model intM for human-object interaction detection includes a visual feature extractor Ext, a first decoder oiD, a second decoder ptD, an information fusion module fM and a third decoder hD.
[0026] In operation Op101, a visual feature extractor Ext is used to extract a visual feature Fv from a sample image sIm. The sample image sIm includes an image region indicating an object touched by a person. The sample image sIm comes from a training sample set. The training sample set includes a plurality of labeled images. For example, the sample image is a photograph taken at the moment when a shopper (corresponding to a "person") in a supermarket pulls out a commodity (corresponding to an "interaction action" of a specific classification) from a shelf with one hand.
[0027] In operation Op103, the first decoder oiD is used to determine the predicted position oLp of the object, the predicted classification oCp of the object, and the predicted action classification aCp of the action performed by the person on the object based on the visual feature Fv and the initial query vector qV0, and generate a first query vector qV1. The predicted position oLp of the object can be represented by the bounding box of the object detected by the model. That is, the first decoder oiD can determine the predicted position of the object by determining the bounding box of the object. The initial query vector qV0 is automatically generated by a module built into the model, which can be gradually determined (optimized) through learning. The first decoder oiD can also be called an "object-interaction pair decoder". The first query vector qV1 is generated by the final output of the first decoder oiD, which contains information about the object-interaction pair. During the training process, the first query vector qV1 will change synchronously with the changes in the model parameters.
[0028] In operation Op105, the second decoder ptD is used to determine the predicted position pLp of the local body of the person in contact with the object based on the first query vector qV1 and the visual feature Fv, and generate a second query vector qV2 related to the local body. The second decoder ptD can determine the predicted position pLp by determining the bounding box of the local body. The second decoder ptD can also be called a "local body decoder". For example, in the sample image, the customer bends his right arm and is holding the bottle of "XX bottled beverage" with his right hand. The detected bounding box of the local body may include the bottle, the customer's right hand, the customer's right forearm, the customer's right upper arm, and part of the customer's right shoulder. The second query vector qV2 is generated by the final output of the second decoder ptD and contains information about the local body. During the training process, the second query vector qV2 will change synchronously with the changes in the model parameters. The target range of the local body decoder will be scaled according to the actions and potential interaction areas that need to be paid attention to in the application environment.
[0029] In operation Op107, the information fusion module fM is used to generate a third query vector qV3 by fusing the first query vector qV1 and the second query vector qV2.
[0030] In operation Op109, a predicted position hLp of the person is determined based on the visual features Fv and the third query vector qV3 using a third decoder hD. The third decoder hD may determine the predicted position hLp by determining a bounding box of the person. The third decoder hD may also be referred to as a “human detection decoder”.
[0031] In operation Op111, the parameters of the model are adjusted based on the predicted position oLp of the item, the predicted classification oCp of the item, the predicted action classification aCp, and the predicted position hLp of the person. In one example, the loss may be determined based on the predicted position oLp of the item, the predicted classification oCp of the item, the predicted action classification aCp, and the predicted position hLp of the person, and the parameters of the model may be adjusted based on the determined loss.
[0032] Operation Op111 may also include adjusting parameters of the model intM based on the predicted position pLp of the local body.
[0033] It can be understood that when the model intM is iteratively trained using the sample set, each round of training may include operations Op101 to Op111. When the training reaches a predetermined condition, the training may be terminated. The final model may be used to detect human-object interactions.
[0034] In order to further improve the performance of the model, the model can be trained using two training sample sets. In one example, method 100 includes training the model using a first training sample set; and fine-tuning the trained model using a second training sample set. The first training sample set includes more sample images than the second training sample set, and the first training sample set includes images for multiple interaction scenarios. All images in the second training sample set are images for a single interaction scene. In one example, the single interaction scene is a scene corresponding to the interaction between customers and goods in an offline shopping place. In one example, the single interaction scene is a ball game scene. In other words, the model can be first trained using a larger general data set so that the model can gain a general understanding of various scenarios, and then the model can be trained using a smaller data set involving a specific scenario to fine-tune the model.
[0035] The information fusion module fM may be configured to generate a third query vector using a modified channel attention mechanism, a spatial attention mechanism, or a residual connection mechanism.
[0036] One aspect of the present disclosure relates to a method for detecting human-object interactions using a trained model IntM. Figure 3 An exemplary flow chart of a method 300 for detecting human-object interaction according to an embodiment of the present disclosure is shown. The method 300 is similar to the method for training a model. The method 300 includes: in operation Op301, using a visual feature extractor Ext to extract a visual feature Fv from a detection image dIm (also referred to as an "input image", "test image" or "image to be detected"), wherein the detection image includes an image region indicating an object contacted by a person; in operation Op303, using a first decoder oiD to determine a predicted position oLp of the object, a predicted classification oCp of the object, and a predicted action classification aCp of the action performed by the person on the object based on the visual feature Fv and an initial query vector qV0, and generating into a first query vector qV1; in operation Op305, using a second decoder ptD, based on the first query vector qV1 and the visual feature Fv, a second query vector qV2 related to the local body of the person in contact with the object is generated; in operation Op307, using an information fusion module fM, a third query vector qV3 is generated by fusing the first query vector qV1 and the second query vector qV2; in operation Op309, using a third decoder hD, the predicted position hLp of the person is determined based on the visual feature Fv and the third query vector qV3.
[0037] One aspect of the present disclosure relates to an apparatus for detecting human-object interaction. Figure 4 The device for detecting human-object interaction is described exemplarily.
[0038] Figure 4An exemplary block diagram of an apparatus 400 for detecting human-object interaction according to an embodiment of the present disclosure is shown.
[0039] The device 400 includes: a memory 401 on which instructions Inst are stored; and at least one processor 403, which is connected to the memory and configured to execute the instructions to: extract visual features from a detection image using a visual feature extractor, wherein the detection image includes an image area indicating an object contacted by a person; determine the predicted position of the object, the predicted classification of the object, and the predicted action classification of the action performed by the person on the object based on the visual features and the initial query vector using a first decoder, and generate a first query vector; generate a second query vector related to the local body of the person in contact with the object using a second decoder based on the first query vector and the visual features; generate a third query vector by fusing the first query vector and the second query vector using an information fusion module; and determine the predicted position of the person based on the visual features and the third query vector using a third decoder. The instruction Inst may have a corresponding relationship with the method 100 and the method 300. The device 400 is a device for detecting human-object interaction in an image using the trained model intM provided by the method 100.
[0040] One aspect of the present disclosure provides a non-transitory computer-readable storage medium storing a program. When the program is executed by a computer, the computer implements the following operations: extracting visual features from a detection image using a visual feature extractor, wherein the detection image includes an image region indicating an object contacted by a person; using a first decoder to determine the predicted position of the object, the predicted classification of the object, and the predicted action classification of the action performed by the person on the object based on the visual features and the initial query vector, and generating a first query vector; using a second decoder to generate a second query vector related to the local body of the person in contact with the object based on the first query vector and the visual features; using an information fusion module to generate a third query vector by fusing the first query vector and the second query vector; and using a third decoder to determine the predicted position of the person based on the visual features and the third query vector. The program may have a corresponding relationship with method 100 and method 300. For further configuration of the program, reference may be made to the description of method 100 and method 300 in the present disclosure.
[0041] According to one aspect of the present disclosure, an information processing device is also provided.
[0042] The present disclosure also relates to an information processing device. Figure 5 is an exemplary block diagram of an information processing device 500 according to one embodiment of the present disclosure. Figure 5In the embodiment, a central processing unit (CPU) 501 performs various processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 to a random access memory (RAM) 503. In the RAM 503, data and the like required when the CPU 501 performs various processes are also stored as needed.
[0043] The CPU 501, the ROM 502, and the RAM 503 are connected to one another via a bus 504. An input / output interface 505 is also connected to the bus 504.
[0044] The following components are connected to the input / output interface 505: an input device 506 including a soft keyboard, etc.; an output device 507 including a display such as a liquid crystal display (LCD), etc. and a speaker, etc.; a storage device 508 such as a hard disk; and a communication device 509 including a network interface card such as a LAN card, a modem, etc. The communication device 509 performs communication processing via a network such as the Internet, a local area network, a mobile network, or a combination thereof.
[0045] A drive 510 is also connected to the input / output interface 505 as necessary. A removable medium 511 such as a semiconductor memory or the like is mounted on the drive 510 as necessary, so that a program read therefrom is installed to the storage device 508 as necessary.
[0046] The CPU 501 may execute a program corresponding to a method for training a model for detecting human-object interaction, or a program for a method for detecting human-object interaction.
[0047] The inventors compared the performance of the model intM of the present disclosure and the model in non-patent document 1. As shown in Table 1, the model of the present disclosure has a significant improvement in detection accuracy, where "normal scene" corresponds to a simple scene with fewer people in the image, and "complex scene" corresponds to a scene with more people in the image.
[0048] Table 1 Comparison of model performance
[0049]
[0050] The model for detecting human-object interaction according to the present disclosure includes a first decoder and a second decoder, that is, an object-interaction pair decoder and a local body decoder. The object-interaction pair decoder combines object detection and interaction detection into a single module. In human-object interaction detection, when an object is detected, the local body of the person in contact with the object is detected in most cases; the local body is related to the position of the person; therefore, when performing person detection, considering the local body information is conducive to improving the accuracy of person position prediction; the classification of interaction actions is highly correlated with objects, therefore, combining object detection and interaction detection into a single module is conducive to improving model performance.
[0051] The beneficial effects of the method, device, and storage medium of the present disclosure include: improving the accuracy of human-object interaction detection results.
[0052] As described above, according to the present disclosure, the principle of improving multi-target tracking has been disclosed. It should be noted that the effects of the scheme of the present disclosure are not necessarily limited to the above-mentioned effects, and in addition to or instead of the effects described in the previous paragraphs, any effect shown in this specification or other effects that can be understood from this specification can be achieved.
[0053] Although the present invention has been disclosed above through the description of the specific embodiments of the present invention, it should be understood that those skilled in the art can design various modifications (including, in the case of failure, the combination or replacement of features between the various embodiments), improvements or equivalents of the present invention within the scope of the appended claims. These modifications, improvements or equivalents should also be considered to be included in the protection scope of the present disclosure.
[0054] It should be emphasized that the term “include / comprises” when used herein refers to the presence of features, elements, steps or components, but does not exclude the presence or addition of one or more other features, elements, steps or components.
[0055] In addition, the methods of the embodiments of the present invention are not limited to being executed in the time sequence described in the specification or shown in the drawings, and may also be executed in other time sequences, in parallel or independently. Therefore, the execution order of the methods described in this specification does not limit the technical scope of the present invention.
[0056] Notes
[0057] The present disclosure includes, but is not limited to, the following schemes.
[0058] 1. A computer-implemented method for training a model for detecting human-object interactions, wherein the model comprises a visual feature extractor, a first decoder, a second decoder, an information fusion module, and a third decoder;
[0059] The method comprises:
[0060] extracting visual features from a sample image using the visual feature extractor, wherein the sample image includes an image region indicating an item touched by a person;
[0061] Determine, using the first decoder based on the visual features and the initial query vector, a predicted location of the object, a predicted classification of the object, and a predicted action classification of an action performed by the person on the object, and generate a first query vector;
[0062] Using the second decoder, based on the first query vector and the visual feature, determine a predicted position of a body part of the person in contact with the object, and generate a second query vector related to the body part;
[0063] Using the information fusion module, generating a third query vector by fusing the first query vector and the second query vector;
[0064] determining, using the third decoder, a predicted position of the person based on the visual features and the third query vector; and
[0065] The parameters of the model are adjusted based on the predicted location of the object, the predicted classification of the object, the predicted action classification, and the predicted location of the person.
[0066] 2. The method according to Note 1, wherein the first decoder is configured to determine the predicted position of the object by determining a bounding box of the object.
[0067] 3. The method according to Note 1, wherein the third decoder is configured to determine the predicted position of the person by determining a bounding box of the person.
[0068] 4. The method according to Note 1 also includes: adjusting the parameters of the model based on the predicted position of the local body.
[0069] 5. The method according to Note 1, wherein the method comprises training the model using a first training sample set; and
[0070] Fine-tune the trained model using the second training sample set;
[0071] The first training sample set includes more sample images than the second training sample set, the first training sample set includes images for multiple interaction scenes, and all images in the second training sample set are images for a single interaction scene.
[0072] 6. The method according to Note 5, wherein the single interaction scenario corresponds to a scenario of interaction between customers and merchandise in an offline shopping location.
[0073] 7. The method according to Note 1, wherein the single interaction scene is a ball game scene.
[0074] 8. The method according to Note 1, wherein the information fusion module is configured to generate a third query vector using a modified channel attention mechanism, a spatial attention mechanism or a residual connection mechanism.
[0075] 9. The method according to Note 1, wherein the second decoder is configured to determine the predicted position of the local body by determining a bounding box of the local body.
[0076] 10. The method according to Supplement 1, wherein adjusting the parameters of the model comprises:
[0077] determining a loss based on the predicted location of the item, the predicted classification of the item, the predicted action classification, and the predicted location of the person; and
[0078] Parameters of the model are adjusted based on the determined losses.
[0079] 11. A device for detecting human-object interaction, comprising:
[0080] a memory having instructions stored thereon; and
[0081] at least one processor connected to the memory and configured to execute the instructions to:
[0082] extracting visual features from a detection image using a visual feature extractor, wherein the detection image includes an image region indicating an item touched by a person;
[0083] Determine, using a first decoder, a predicted position of the object, a predicted classification of the object, and a predicted action classification of an action performed by the person on the object based on the visual features and the initial query vector, and generate a first query vector;
[0084] Using a second decoder, generating a second query vector related to a body part of the person in contact with the object based on the first query vector and the visual feature;
[0085] Using an information fusion module, generating a third query vector by fusing the first query vector and the second query vector; and
[0086] Using a third decoder, a predicted position of the person is determined based on the visual features and the third query vector.
[0087] 12. The apparatus according to Note 11, wherein the first decoder is configured to determine the predicted position of the object by determining a bounding box of the object.
[0088] 13. The apparatus according to Note 11, wherein the third decoder is configured to determine the predicted position of the person by determining a bounding box of the person.
[0089] 14. The device according to Note 11, wherein the detection image is an image obtained in an offline shopping location and indicating the interaction between a customer and a product.
[0090] 15. The device according to Note 11, wherein the detected image is an image acquired in a ball game scene.
[0091] 16. An apparatus according to Note 11, wherein the information fusion module is configured to generate a third query vector using a modified channel attention mechanism, a spatial attention mechanism, or a residual connection mechanism.
[0092] 17. The apparatus according to Note 11, wherein the second decoder is configured to determine a bounding box of the local body.
[0093] 18. A non-transitory computer-readable storage medium storing a program, wherein when the program is executed by a computer, the computer implements the following operations:
[0094] extracting visual features from a detection image using a visual feature extractor, wherein the detection image includes an image region indicating an item touched by a person;
[0095] Determine, using a first decoder, a predicted position of the object, a predicted classification of the object, and a predicted action classification of an action performed by the person on the object based on the visual features and the initial query vector, and generate a first query vector;
[0096] Using a second decoder, generating a second query vector related to a body part of the person in contact with the object based on the first query vector and the visual feature;
[0097] Using an information fusion module, generating a third query vector by fusing the first query vector and the second query vector; and
[0098] Using a third decoder, a predicted position of the person is determined based on the visual features and the third query vector.
[0099] 19. A non-transitory computer-readable storage medium according to Note 18, wherein the information fusion module is configured to generate a third query vector using a modified channel attention mechanism, a spatial attention mechanism, or a residual connection mechanism.
[0100] 20. According to the non-transitory computer-readable storage medium of Note 18, the second decoder is configured to determine a bounding box of the partial body.
Claims
1. A computer-implemented method for training a model for detecting human-object interactions, characterized in that The model includes a visual feature extractor, a first decoder, a second decoder, an information fusion module and a third decoder; The method comprises: extracting visual features from a sample image using the visual feature extractor, wherein the sample image includes an image region indicating an item touched by a person; Determine, using the first decoder based on the visual features and the initial query vector, a predicted location of the object, a predicted classification of the object, and a predicted action classification of an action performed by the person on the object, and generate a first query vector; Using the second decoder, based on the first query vector and the visual feature, determine a predicted position of a body part of the person in contact with the object, and generate a second query vector related to the body part; Using the information fusion module, generating a third query vector by fusing the first query vector and the second query vector; determining, using the third decoder, a predicted position of the person based on the visual features and the third query vector; and The parameters of the model are adjusted based on the predicted location of the object, the predicted classification of the object, the predicted action classification, and the predicted location of the person.
2. The method according to claim 1, wherein: The first decoder is configured to determine a predicted position of the item by determining a bounding box of the item.
3. The method according to claim 1, wherein: The third decoder is configured to determine a predicted position of the person by determining a bounding box of the person.
4. The method according to claim 1, further comprising: Parameters of the model are adjusted based on the predicted position of the local body.
5. The method according to claim 1, wherein: The method includes training the model using a first training sample set; and Fine-tune the trained model using the second training sample set; The first training sample set includes more sample images than the second training sample set, the first training sample set includes images for multiple interaction scenes, and all images in the second training sample set are images for a single interaction scene.
6. The method according to claim 5, wherein: The single interaction scenario corresponds to a scenario of interaction between customers and commodities in an offline shopping place.
7. The method according to claim 5, wherein: The single interaction scene is a ball game scene.
8. The method according to claim 1, wherein: The information fusion module is configured to generate a third query vector using a modified channel attention mechanism, a spatial attention mechanism, or a residual connection mechanism.
9. A device for detecting human-object interaction, characterized in that: include: a memory having instructions stored therein; as well as at least one processor connected to the memory and configured to execute the instructions to: extracting visual features from a detection image using a visual feature extractor, wherein the detection image includes an image region indicating an item touched by a person; Determine, using a first decoder, a predicted position of the object, a predicted classification of the object, and a predicted action classification of an action performed by the person on the object based on the visual features and the initial query vector, and generate a first query vector; Using a second decoder, generating a second query vector related to a body part of the person in contact with the object based on the first query vector and the visual feature; Using an information fusion module, generating a third query vector by fusing the first query vector and the second query vector; and Using a third decoder, a predicted position of the person is determined based on the visual features and the third query vector.
10. A non-transitory computer-readable storage medium storing a program, characterized in that: When the program is executed by a computer, the computer implements the following operations: extracting visual features from a detection image using a visual feature extractor, wherein the detection image includes an image region indicating an item touched by a person; Determine, using a first decoder, a predicted position of the object, a predicted classification of the object, and a predicted action classification of an action performed by the person on the object based on the visual features and the initial query vector, and generate a first query vector; Using a second decoder, generating a second query vector related to a body part of the person in contact with the object based on the first query vector and the visual feature; Using an information fusion module, generating a third query vector by fusing the first query vector and the second query vector; and Using a third decoder, a predicted position of the person is determined based on the visual features and the third query vector.