Character interaction detection method and device based on large model, equipment and medium
Through a large-model-based human interaction detection method, a backbone network and decoder combination are used to perform multi-scale feature extraction and action category probability prediction, which solves the problem of low human interaction detection accuracy and achieves higher detection accuracy.
Patent Information
- Application Number
- CN202510812574.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-17
- Publication Date
- 2025-09-26
AI Technical Summary
In the prior art, due to the large number of people and objects in the image and their small sizes, it is difficult to determine the main people, objects, and the actions between people and objects, which leads to low accuracy in human interaction detection.
A large-model-based human interaction detection method is adopted. By obtaining training data, using the backbone network, image codec, human decoder and object decoder, combined with multi-scale feature maps and encoded image features, the position and action category probability of human and object are predicted, and the decoder parameters are updated through the loss function to improve the detection accuracy.
The accuracy of human interaction detection has been improved, especially the detection performance in rare and non-rare categories, which significantly improves the detection effect of the model in processing complex scenes.
Smart Images
Figure CN120707950A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of object detection technology, and more specifically, to a large-scale model-based human interaction detection method, device, equipment, and medium. Background Art
[0002] The neural network-based object detection algorithm can quickly detect people and objects in images, but it cannot detect the motion relationship between people and objects.
[0003] Existing technology uses human interaction detection algorithms to detect the motion relationships between people and objects surrounding them in an image. These algorithms first detect objects and people, then predict the motion relationships between them based on the detected people and objects. However, in some image data, the number of people and objects is large and small, making it difficult to identify the primary people, objects, and the motions between them. This, in turn, leads to low human interaction detection accuracy. Summary of the Invention
[0004] In order to overcome the above-mentioned defects, the present application is proposed to provide a solution or at least partially solve the technical problem that the large number of people and objects makes it difficult to determine the main people, objects and actions between people and objects, which in turn leads to low accuracy in human interaction detection.
[0005] In a first aspect, the present application provides a method for detecting human interaction based on a large model, comprising:
[0006] Obtain the image to be detected;
[0007] Inputting the image to be detected into a preset model to obtain an interactive action;
[0008] Wherein, obtaining the preset model includes:
[0009] In each round, training data is obtained; wherein the training data includes image data, the position of people, the position of objects, and the interactive actions of people;
[0010] Inputting the image data into the backbone network to obtain a multi-scale feature map;
[0011] Inputting the multi-scale feature map into an image codec to obtain encoded image features and reference detection frame information;
[0012] Obtaining query data according to the encoded image features and the reference detection frame information;
[0013] Inputting the query data into a human body decoder to obtain human body feature code and human body position information;
[0014] Inputting the query data into an object decoder to obtain object feature code, object location information and object category probability;
[0015] Obtaining an action category probability according to the human body feature code, the object feature code, the human body position information, and the object position information;
[0016] updating the parameters of the human decoder, the object decoder, and the action decoder according to the human position information, the object position information, the object category probability, the action category probability, and the human positions, object positions, and human interaction actions in the training data;
[0017] After determining that the preset conditions are met, a preset model is obtained.
[0018] In a technical solution of the above-mentioned large-model-based method for detecting human interaction, inputting the image data into a backbone network to obtain a multi-scale feature map includes:
[0019] Inputting the image data into the backbone network to obtain feature maps of different scales;
[0020] The multi-scale feature map is obtained according to the feature maps of different scales.
[0021] In a technical solution of the above-mentioned large-model-based method for detecting human interaction, inputting the feature map into an image codec to obtain encoded image features and reference detection frame information includes:
[0022] Dividing the feature map into a plurality of feature map blocks, and generating position information according to positions of the feature map blocks;
[0023] Convert the feature image block into a one-dimensional vector, and then convert the one-dimensional vector into an embedding vector;
[0024] The embedding vector and the position information are input into an image codec to obtain encoded image features and reference detection frame information; wherein the reference detection frame information includes the position of the reference detection frame and the probability that the reference detection frame belongs to the foreground.
[0025] In a technical solution of the above-mentioned large-model-based method for detecting human interaction, obtaining query data according to the encoded image features and the reference detection frame information includes:
[0026] sorting the reference detection frames according to the probability that the reference frames belong to the foreground, to obtain sorted reference detection frames;
[0027] Selecting a preset number of reference detection frames from the sorted reference detection frames;
[0028] For any reference detection frame in the reference detection frames of the preset data, the position of the reference detection frame and the encoded image feature corresponding to the reference detection frame are combined to obtain query data.
[0029] In a technical solution of the above-mentioned large-model-based human interaction detection method, inputting the query data into an object decoder to obtain object feature codes, object location information, and object category probabilities includes:
[0030] Inputting the query data into an object decoder to obtain an object feature code and an object reference position;
[0031] Adjusting the object reference position according to the object reference position and the query data to obtain object position information;
[0032] According to the query data, the object category probability is obtained.
[0033] In a technical solution of the above-mentioned large-model-based human interaction detection method, obtaining the action category probability based on the human feature code, the object feature code, the human position information, and the object position information includes:
[0034] Combining the human body feature code and the object feature code to obtain a combined feature code;
[0035] combining the human body position information and the object position information to obtain combined position information;
[0036] The combined feature code and the combined position information are input into an action decoder to obtain an action category probability.
[0037] In a technical solution of the above-mentioned large-model-based human interaction detection method, the updating of parameters of the human decoder, the object decoder, and the action decoder based on the human position information, object position information, object category probability, action category probability, and the human positions, object positions, and human interaction actions in the training data includes:
[0038] Obtaining a plurality of triplets according to the human body position information, the object position information, the object category probability, and the action category probability;
[0039] Obtaining a confidence score of the triplet according to the object category probability and the action category probability;
[0040] According to the preset confidence, the predicted triples are obtained;
[0041] Obtaining a loss value according to the predicted triplet, a position of a person in training data, a position of an object, a person interaction action, and a loss function;
[0042] According to the loss value, the parameters of the human body decoder, the object decoder and the action decoder are updated.
[0043] In a second aspect, the present application provides a large-model-based human interaction detection device, comprising:
[0044] An acquisition module, used for acquiring an image to be detected;
[0045] A prediction module, configured to input the image to be detected into a preset model to obtain an interactive action;
[0046] A training module is used to obtain training data in each round; wherein the training data includes image data, the position of people, the position of objects, and the interactive actions of people;
[0047] The training module is further configured to input the image data into a backbone network to obtain a multi-scale feature map;
[0048] The training module is further configured to input the multi-scale feature map into an image codec to obtain encoded image features and reference detection frame information;
[0049] The training module is further configured to obtain query data based on the encoded image features and the reference detection frame information;
[0050] The training module is further configured to input the query data into a human body decoder to obtain human body feature codes and human body position information;
[0051] The training module is further configured to input the query data into an object decoder to obtain object feature codes, object location information, and object category probabilities;
[0052] The training module is further configured to obtain an action category probability based on the human body feature code, the object feature code, the human body position information, and the object position information;
[0053] The training module is further configured to update parameters of the human decoder, the object decoder, and the action decoder based on the human position information, object position information, object category probability, action category probability, and the positions of people, objects, and human interaction actions in the training data;
[0054] The training module is further configured to obtain a preset model after determining that a preset condition is met.
[0055] In a third aspect, the present application provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the large model-based human interaction detection method described in any one of the first aspects through the computer program.
[0056] In a fourth aspect, the present application provides a computer-readable storage medium storing a plurality of program codes, wherein the program codes are suitable for being loaded and run by a processor to execute the large model-based human interaction detection method described in any one of the first aspects.
[0057] The present application provides a method, apparatus, device and medium for detecting human interaction based on a large model, the method specifically comprising: obtaining an image to be detected; inputting the image to be detected into a preset model to obtain interactive actions; wherein obtaining the preset model comprises: obtaining training data in each round; wherein the training data comprises image data, the position of a person, the position of an object, and the interactive actions of a person; inputting the image data into a backbone network to obtain a multi-scale feature map; inputting the multi-scale feature map into an image codec to obtain encoded image features and reference detection frame information; obtaining query data based on the encoded image features and the reference detection frame information; and inputting the query data into a human body decoder. The invention relates to a method for detecting human interaction by using a human decoder, a human body feature code and a human body position information; inputting the query data into an object decoder to obtain an object feature code, an object position information and an object category probability; obtaining an action category probability based on the human feature code, the object feature code, the human position information and the object position information; updating the parameters of the human decoder, the object decoder and the action decoder based on the human position information, the object position information, the object category probability, the action category probability and the human position, the object position and the human interaction action in the training data; obtaining a preset model after determining that the preset conditions are met, thereby improving the accuracy of human interaction detection. BRIEF DESCRIPTION OF THE DRAWINGS
[0058] The disclosure of this application will become more easily understood with reference to the accompanying drawings. Those skilled in the art will readily appreciate that these drawings are for illustrative purposes only and are not intended to limit the scope of protection of this application.
[0059] Additionally, like numerals are used to denote like parts throughout the drawings, wherein:
[0060] Figure 1 A flowchart of a first embodiment of a method for detecting human interaction based on a large model provided in an embodiment of the present application;
[0061] Figure 2 A flowchart of a second embodiment of a large-model-based method for detecting human interaction provided in an embodiment of the present application;
[0062] Figure 3 A flowchart of a third embodiment of a large-model-based method for detecting human interaction provided in an embodiment of the present application;
[0063] Figure 4 A flowchart of a fourth embodiment of a large-model-based method for detecting human interaction provided in an embodiment of the present application;
[0064] Figure 5 A flowchart of a fifth embodiment of a large-model-based method for detecting human interaction provided in an embodiment of the present application;
[0065] Figure 6 A flowchart of a sixth embodiment of a large-model-based method for detecting human interaction provided in an embodiment of the present application;
[0066] Figure 7 A flowchart of a seventh embodiment of a large-model-based method for detecting human interaction provided in an embodiment of the present application;
[0067] Figure 8 A structural diagram of a preset model provided in an embodiment of the present application;
[0068] Figure 9 A schematic structural diagram of a first embodiment of a large-model-based human interaction detection device provided in an embodiment of the present application;
[0069] Figure 10 This is a structural diagram of a first embodiment of an electronic device provided in an embodiment of the present application.
[0070] Reference Signs List :
[0071] 11: Acquisition module; 12: Prediction module; 13: Training module; 21: Processor; 22: Memory. DETAILED DESCRIPTION
[0072] Some embodiments of the present application are described below with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are only used to explain the technical principles of the present application and are not intended to limit the scope of protection of the present application.
[0073] In the description of this application, "module" and "processor" may include hardware, software, or a combination of both. A module may include hardware circuitry, various suitable sensors, communication ports, and memory. It may also include software components, such as program code, or a combination of software and hardware. A processor may be a central processing unit, a microprocessor, an image processor, a digital signal processor, or any other suitable processor. A processor has data and / or signal processing capabilities. A processor may be implemented in software, hardware, or a combination of both. Non-transitory computer-readable storage media include any suitable medium capable of storing program code, such as magnetic disks, hard disks, optical disks, flash memory, read-only memory, random access memory, etc. The term "A and / or B" refers to all possible combinations of A and B, such as only A, only B, or both A and B. The terms "at least one of A or B" or "at least one of A and B" have similar meanings to "A and / or B" and may include only A, only B, or both A and B. The singular forms "a" and "the" may also include the plural forms.
[0074] Here we first explain some terms involved in this application.
[0075] object365: An object detection dataset containing approximately 2 million images of objects in 365 categories.
[0076] COCO: One of the most commonly used object detection datasets, the detection part contains about 200,000 images in 80 categories.
[0077] VG: Visual Genome Dataset, a language and vision dataset proposed by Stanford in 2016, aims to better understand the world. It annotates a large number of objects and relationships in images, as well as question-and-answer pairs about the images. The annotations can also be used for the HOI task, and consist of approximately 110,000 images.
[0078] HICO-DET: It is one of the most commonly used datasets in the field of human interaction detection, containing 47,776 images, 80 types of objects, 117 types of actions, and 600 types of gerund combinations (objects and actions are combined in pairs, but some categories of combinations do not actually exist).
[0079] mAP: Used to measure the performance of object detection algorithms. Generally speaking, the average accuracy (mAP) is the weighted average of the average accuracy (AP) of all categories.
[0080] Rare mAP: In model evaluation, this measures the average precision of a model on rare classes. Rare classes are classes that appear less frequently in a dataset. By calculating the average precision on these rare classes, we can assess the model's performance when handling rare classes.
[0081] Non-rare mAP: In model evaluation, it is used to measure the average precision of the model on non-rare categories. By calculating the average precision of these non-rare categories, the performance of the model in handling non-rare categories can be evaluated.
[0082] Existing human interaction detection algorithms based on detection transformers are effective for detecting large objects, but struggle with small ones. Training these algorithms also requires significant time and computational resources. Furthermore, training these algorithms faces challenges due to the limited availability of publicly available datasets and the difficulty of labeling them. These issues all contribute to the low accuracy of detecting human-object interactions.
[0083] Based on this, in order to solve the above technical problems, the present application provides a new large-model-based character interaction detection method to improve the accuracy of character interaction action detection.
[0084] The following specific embodiments describe in detail the technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.
[0085] Figure 1 This is a flow chart of a first embodiment of a method for detecting human interaction based on a large model provided in the present application. Figure 1 Specifically, the method includes:
[0086] Step S1: Acquire the image to be detected.
[0087] Step S2: Input the image to be detected into the preset model to obtain an interactive action.
[0088] In this embodiment, the interactive action is text information including people, objects and actions, for example: he is playing basketball.
[0089] The step of obtaining the preset model in step S2 includes:
[0090] Step S21: In each round, training data is obtained; wherein the training data includes image data, positions of people, positions of objects, and interactive actions of people.
[0091] In this embodiment, 500,000 labeled data are selected from the object365, COCO, and VG datasets as training data.
[0092] Step S22: Input the image data into the backbone network to obtain a multi-scale feature map.
[0093] In this embodiment, the backbone network is any network having a convolutional neural network structure.
[0094] In this embodiment, for example, the backbone network can be Restnet50, swin-tiny, swin-large or transform neural network.
[0095] Preferably, swim-large is used as the backbone network.
[0096] In this embodiment, multiple feature maps are generated after feature extraction. The multiple feature maps may be feature maps of different scales in different convolutional layers.
[0097] Step S23: Input the multi-scale feature map into the image codec to obtain encoded image features and reference detection frame information.
[0098] In this embodiment, the image codec is a neural network with a transform structure, which includes a transform encoder and a transform decoder, which includes a deformable self-attention mechanism and has a global receptive field, and can realize global context relationship modeling.
[0099] In this embodiment, the encoded image feature is a 256-dimensional vector, and the reference frame information is the position of the human body in the image.
[0100] In this embodiment, one encoded image feature corresponds to one reference detection frame information.
[0101] Step S24: Obtain query data based on the encoded image features and the reference detection frame information.
[0102] In this embodiment, the encoded image features and the reference detection frame information are combined to obtain query data.
[0103] In this embodiment, for example, there are 900 encoded image features and corresponding reference detection frame information, so the number of query data is also 900.
[0104] Step S25: input the query data into the human body decoder to obtain the human body feature code and human body position information.
[0105] In this embodiment, the human body decoder is a multi-layer transform structure with a deformable self-attention mechanism and a cross-attention mechanism, which can more effectively model the context of the current generation position.
[0106] Preferably, the human body codec is a 6-layer transform structure.
[0107] In this embodiment, the human body decoder is used to detect only humans, thereby avoiding interference from objects in the image and improving the accuracy of human body position detection.
[0108] Step S26: Input the query data into the object decoder to obtain the object feature code, object location information and object category probability.
[0109] In this embodiment, the object decoder includes a multi-layer transform structure and a fully connected network. The multi-layer transform structure can obtain object feature coding and object position information. The object feature coding is input into the fully connected network to obtain the object category probability.
[0110] In this embodiment, for example, the softmax cross entropy loss function of the fully connected network is used to obtain the object category and the probability that the object belongs to the category.
[0111] Step S27: Obtaining the action category probability according to the human body feature code, the object feature code, the human body position information and the object position information.
[0112] In this embodiment, human feature encoding, object feature encoding, human position information, and object position information are input into a fully connected layer, and a loss function is used to obtain the action category probability.
[0113] Step S28: Update the parameters of the human decoder, object decoder and action decoder according to the human position information, object position information, object category probability, action category probability and the human position, object position and human interaction action in the training data.
[0114] Step S29: After determining that the preset conditions are met, a preset model is obtained.
[0115] In this embodiment, the preset condition may be the number of training times or the loss value.
[0116] In this embodiment, for example, after determining that a preset number of training times has been reached, a preset model is obtained.
[0117] In this embodiment, an image to be detected is obtained; the image to be detected is input into a preset model to obtain an interactive action; wherein obtaining the preset model includes: obtaining training data in each round; wherein the training data includes image data, a person's position, an object's position, and a person's interactive action; inputting the image data into a backbone network to obtain a multi-scale feature map; inputting the multi-scale feature map into an image codec to obtain encoded image features and reference detection frame information; obtaining query data based on the encoded image features and the reference detection frame information; inputting the query data into a human body decoder to obtain a human body feature code and human body position information; inputting the query data into an object decoder to obtain an object feature code, object position information, and object category probability; obtaining an action category probability based on the human body feature code, object feature code, human position information, and object position information; updating the parameters of the human body decoder, object decoder, and action decoder based on the human position information, object position information, object category probability, action category probability, and the person's position, object position, and person's interactive action in the training data; after determining that the preset conditions are met, the preset model is obtained. Compared with the prior art, due to the number of any objects, the As for the low accuracy of detection of interactive actions between people and objects caused by too many factors, the present application obtains training data in each round; wherein the training data includes image data, the position of people, the position of objects, and interactive actions of people; the image data is input into the backbone network to obtain a multi-scale feature map; the multi-scale feature map is input into the image codec to obtain encoded image features and reference detection frame information; query data is obtained based on the encoded image features and the reference detection frame information; the query data is input into the human body decoder to obtain human body feature coding and human body position information; the query data is input into the object decoder to obtain object feature coding, object position information and object category probability; the action category probability is obtained based on the human body feature coding, object feature coding, human position information and object position information; the parameters of the human body decoder, object decoder and action decoder are updated based on the human position information, object position information, object category probability, action category probability and the position of people, object position and interactive actions of people in the training data; after determining that the preset conditions are met, a preset model is obtained, and then the image to be detected is input into the preset model to obtain interactive actions, thereby improving the accuracy of human interaction detection.
[0118] Figure 2 This is a flow chart of a second embodiment of a large model-based character interaction detection method provided in the present application. Based on the above embodiment, Figure 2 Specifically, one implementation of step S22 includes:
[0119] Step S201: Input the image data into the backbone network to obtain feature maps of different scales.
[0120] In this embodiment, the backbone network is a feature extraction network having a multi-layer convolutional neural network. The image data is input into the multi-layer convolutional neural network, and the scale of the feature map is gradually reduced, thereby obtaining feature maps of different scales.
[0121] In this embodiment, for example, image data is input into the backbone network to obtain feature maps of four different scales.
[0122] Step S202: Obtain a multi-scale feature map based on feature maps of different scales.
[0123] In this embodiment, feature maps of different scales are combined to obtain a multi-scale feature map.
[0124] In this embodiment, feature maps of different scales include large-scale feature maps and small-scale feature maps, wherein the large-scale feature map can be used to detect small objects, and the small-scale feature map can be used to detect large objects, and thus the multi-scale feature map can accurately detect objects of different sizes.
[0125] In this embodiment, image data is input into the backbone network to obtain feature maps of different scales; based on the feature maps of different scales, multi-scale feature maps are obtained, and then features of different scales can be used to accurately detect objects of different sizes.
[0126] Figure 3 This is a flow chart of a third embodiment of a method for detecting human interaction based on a large model provided in the present application. Based on the above embodiment, Figure 3 As shown, specifically, one implementation of step S23 includes:
[0127] Step S301: Divide the feature map into multiple feature blocks, and generate position information according to the positions of the feature blocks.
[0128] In this embodiment, the feature map is divided evenly according to a preset number of feature blocks, and then during the division process, after a feature block is divided, position information is generated according to the position of the feature block in the feature map.
[0129] In this embodiment, for example, the feature map is evenly divided into 4 feature blocks, and the position information of each feature block is the coordinates of the first point in the upper left corner of the feature block in the feature map.
[0130] Step S302: Convert the feature image block into a one-dimensional vector, and then convert the one-dimensional vector into an embedding vector.
[0131] In this embodiment, the data of each row of the two-dimensional feature block are connected end to end to obtain a one-dimensional vector, and then the one-dimensional vector is converted into an embedding vector.
[0132] Step S303: Input the embedding vector and position information into the image codec to obtain encoded image features and reference detection frame information; wherein the reference detection frame information includes the position of the reference detection frame and the probability that the reference detection frame belongs to the foreground.
[0133] In this embodiment, the image codec is a transform structure, which includes a transform encoder and a transform decoder.
[0134] In this embodiment, feature extraction is performed on the embedding vector and the position information of the corresponding feature block in the transform encoder to obtain the encoded image features, and then the encoded image features are input into the transform decoder, and through the fully connected layer in the transform decoder, the position of the reference detection frame and the probability that the reference detection frame belongs to the foreground are obtained.
[0135] In this embodiment, the position of the reference detection frame includes the coordinates of the upper left corner of the reference detection frame in the image and the width and height of the reference detection frame.
[0136] In this embodiment, the foreground of the image includes people and objects, while the background of the image is information unrelated to people and objects. Therefore, a reference detection frame belonging to the foreground needs to be obtained.
[0137] In this embodiment, the feature map is divided into multiple feature blocks, and position information is generated according to the position of the feature blocks; the feature blocks are converted into one-dimensional vectors, and then the one-dimensional vectors are converted into embedding vectors; the embedding vectors and the position information are input into the image codec to obtain encoded image features and reference detection frame information; wherein the reference detection frame information includes the position of the reference detection frame and the probability that the reference detection frame belongs to the foreground, thereby distinguishing the foreground and background in the image, and improving the detection accuracy of people and objects.
[0138] Figure 4 This is a flow chart of a fourth embodiment of a large model-based human interaction detection method provided in the present application. Based on the above embodiments, Figure 4 Specifically, one implementation of step S24 includes:
[0139] Step S401: sorting the reference detection frames according to the probability that the reference frames belong to the foreground to obtain sorted reference detection frames.
[0140] In this embodiment, multiple reference detection frames all have a probability of belonging to the foreground, and the greater the probability, the more likely the reference detection frame is a person or an object. Therefore, it is necessary to sort the probabilities of the reference frames belonging to the foreground from large to small to obtain the sorted reference detection frames.
[0141] Step S402: Select a preset number of reference detection frames from the sorted reference detection frames.
[0142] In this embodiment, for example, the preset number is 900.
[0143] Step S403: For any reference detection frame in the reference detection frames of the preset data, the position of the reference detection frame and the encoded image features corresponding to the reference detection frame are combined to obtain query data.
[0144] In this embodiment, the position of the reference detection frame is converted into a one-dimensional position vector, the encoded image features corresponding to the reference frame are converted into a one-dimensional feature vector, and the one-dimensional position vector and the one-dimensional feature vector are spliced to obtain query data.
[0145] In this embodiment, the reference detection frames are sorted according to the probability that the reference detection frames belong to the foreground to obtain sorted reference detection frames; a preset number of reference detection frames are selected from the sorted reference detection frames; for any reference detection frame in the reference detection frames of the preset data, the position of the reference detection frame and the encoded image features corresponding to the reference detection frame are combined to obtain query data.
[0146] Figure 5 This is a flow chart of a fifth embodiment of a method for detecting human interaction based on a large model provided in an embodiment of the present application. Based on the above embodiments, Figure 5 As shown, specifically, one implementation of step S26 includes:
[0147] Step S501: Input query data into the object decoder to obtain the object feature code and object reference position.
[0148] In this embodiment, the query data is input into the object decoder to obtain a 256*1-dimensional object feature code and a 4*1-dimensional object reference position.
[0149] Step S502: According to the object reference position and the query data, the object reference position is adjusted to obtain the object position information.
[0150] In this embodiment, the object reference position is adjusted in combination with the object feature coding to obtain the object position information.
[0151] Step S503: Obtain object category probability based on the query data.
[0152] In this embodiment, the fully connected network and loss function in the object decoder are used to classify objects and obtain object category probabilities.
[0153] In this embodiment, the object decoder is used to detect the object, thereby eliminating the interference caused by people in the image and improving the accuracy of object detection.
[0154] In this embodiment, the query data is input into the object decoder to obtain the object feature code and the object reference position; according to the object reference position and the query data, the object reference position is adjusted to obtain the object position information; according to the query data, the object category probability is obtained, and the objects in the image can be accurately classified and the position detected.
[0155] Figure 6 This is a flow chart of a sixth embodiment of a large model-based person interaction detection method provided in the present application. Based on the above embodiments, Figure 6 Specifically, one implementation of step S27 includes:
[0156] Step S601: combining the human body feature code and the object feature code to obtain a combined feature code.
[0157] In this embodiment, the corresponding points of the human body feature code and the object feature code are added together to obtain a combined feature code.
[0158] In this embodiment, for example, human body feature encoding is 256*1 dimension, object feature encoding is 256*1 dimension, and combination feature encoding is 256*1 dimension.
[0159] Step S602: combining the human body position information and the object position information to obtain combined position information.
[0160] In this embodiment, the human body position information and the object position information are stacked to obtain combined position information.
[0161] In this embodiment, for example, the human body position information is a 4*1 dimensional vector, and the object position information is a 4*1 dimensional vector. After stacking the human body problem information and the object position information, the combined position information is an 8*1 dimensional vector.
[0162] Step S603: input the combined feature code and the combined position information into the action decoder to obtain the action category probability.
[0163] In this embodiment, the combined feature code and the combined position information are input into the action decoder, and the combined feature code is classified through the fully connected layer in the action decoder to obtain the action category probability.
[0164] In this embodiment, the human feature code and the object feature code are combined to obtain a combined feature code; the human position information and the object position information are combined to obtain a combined position information; the combined feature code and the combined position information are input into the action decoder to obtain the action category probability, thereby improving the accuracy of human interaction detection.
[0165] Figure 7 This is a flow chart of a seventh embodiment of a method for detecting human interaction based on a large model provided in an embodiment of the present application. Based on the above embodiments, Figure 7 As shown, specifically, a specific implementation of step S28 includes:
[0166] Step S701: obtaining multiple triplets based on human body position information, object position information, object category probability, and action category probability.
[0167] In this embodiment, multiple sets of human body position information, object position information, object category probabilities, and action category probabilities are combined to obtain multiple triplets.
[0168] In this embodiment, for example, one of the three primitive groups is {human body position information; object position information; object category probability; action category probability}.
[0169] Step S702: Obtain the confidence of the triplet based on the object category probability and the action category probability.
[0170] In this embodiment, the object category probability is multiplied by the action category probability to obtain the confidence of the triplet.
[0171] Step S703: Obtain a predicted triplet according to a preset confidence level.
[0172] In this embodiment, a screening triplet is obtained according to a non-maximum suppression algorithm, and then a triplet whose confidence in the screening triplet is greater than a preset confidence is used as a predicted triplet.
[0173] Step S704: Obtain a loss value based on the predicted triplet, the position of the person in the training data, the position of the object, the character interaction action, and the loss function.
[0174] In this embodiment, for example, the loss function may be a cross entropy loss function.
[0175] Step S705: Update the parameters of the human body decoder, object decoder, and action decoder according to the loss value.
[0176] In this embodiment, the parameters of the human body decoder, object decoder, and action decoder are updated using a gradient descent algorithm based on the loss value.
[0177] In this embodiment, multiple triplets are obtained based on human body position information, object position information, object category probability, and action category probability; the confidence of the triplets is obtained based on the object category probability and the action category probability; based on the preset confidence, a predicted triplet is obtained; based on the predicted triplet, the human position, object position, character interaction action and loss function in the training data, a loss value is obtained; based on the loss value, the parameters of the human body decoder, object decoder and action decoder are updated.
[0178] The method provided in this application is experimented with other human interaction detection algorithms on the HICO-DET dataset. All human interaction detection algorithms in the table use swin-large as the feature extraction network. The comparison results are shown in Table 1.
[0179] Table 1 Comparison results
[0180] method mAP rare mAP non-rare mAP FGA-HOI 37.18 30.71 39.11 DiffHOI 41.50 39.96 43.62 RLIPv2 45.09 43.23 45.64 This application 45.33 44.22 45.67
[0181] It can be seen from the experimental results in Table 1 that the experimental results of the large model-based human interaction detection method proposed in this application are all higher than those of other human interaction detection methods.
[0182] Figure 8 The structural diagram of the preset model provided in the embodiment of the present application is as follows: Figure 8 shown.
[0183] In the structure diagram of the preset model, the backbone network extracts features from the image and obtains a multi-scale feature map. Here, four feature maps of different scales are listed in a graphical manner.
[0184] Furthermore, the present application also provides a large model-based human interaction detection device.
[0185] Figure 9 This is a structural diagram of a first embodiment of a large model-based human interaction detection device provided in the present application. Figure 9As shown, the large-model-based human interaction detection device in an embodiment of the present application mainly includes an acquisition module 11, a prediction module 12, and a training module 13, wherein one or more of the above modules can be combined into one module. In some embodiments, the acquisition module 11 can be configured to acquire an image to be detected. The prediction module 12 can be configured to input the image to be detected into a preset model to obtain interactive actions. The training module 13 can be configured to acquire training data in each round; wherein the training data includes image data, the location of people, the location of objects, and human interactive actions. The training module 13 can also be configured to input the image data into the backbone network to obtain a multi-scale feature map. The training module 13 can also be configured to input the multi-scale feature map into the image codec to obtain encoded image features and reference detection frame information. The training module 13 can also be configured to obtain query data based on the encoded image features and reference detection frame information. The training module 13 can also be configured to input the query data into the human body decoder to obtain human body feature encoding and human body position information. The training module 13 can also be configured to input query data into the object decoder to obtain object feature codes, object location information, and object category probabilities. The training module 13 can also be configured to obtain action category probabilities based on the human feature codes, object feature codes, human location information, and object location information. The training module 13 can also be configured to update the parameters of the human decoder, object decoder, and action decoder based on the human location information, object location information, object category probabilities, action category probabilities, and the human positions, object positions, and human interaction actions in the training data. The training module 13 can also be configured to obtain a preset model after determining that a preset condition has been met.
[0186] The above-mentioned large model-based human interaction detection device is used to perform Figures 1 to 7 The embodiment of the large-model-based character interaction detection method shown in the figure has similar technical principles, technical problems solved and technical effects produced. Those skilled in the art can clearly understand that for the convenience and conciseness of description, the specific working process and related instructions of the large-model-based character interaction detection device can refer to the contents described in the embodiment of the large-model-based character interaction detection method whose execution subject is the large-model-based character interaction detection device, and will not be repeated here.
[0187] It will be understood by those skilled in the art that all or part of the processes in the method for implementing the above embodiment of the present application can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, it can implement the steps of each of the above method embodiments. The computer program includes computer program code, which can be in source code form, object code form, executable file or some intermediate form. The computer-readable storage medium can include: any entity or device, medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory, random access memory, electric carrier signal, telecommunication signal and software distribution medium that can carry the computer program code. It should be noted that the content contained in the computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable storage media do not include electric carrier signals and telecommunication signals.
[0188] Furthermore, the present application also provides an electronic device.
[0189] Figure 10 This is a structural diagram of an electronic device embodiment 1 provided in the present application. Figure 10 As shown, the electronic device includes at least one processor 21 and a memory 22, and the memory 22 can be configured to store and execute the above Figures 1 to 7 The processor 21 can be configured to execute the program in the memory 22 of the large model-based human interaction detection method of the illustrated embodiment, including but not limited to executing a program of the large model-based human interaction detection method of the above-mentioned method embodiment. For ease of explanation, only the parts related to the embodiment of the present application are shown. For specific technical details not disclosed, please refer to the method section of the embodiment of the present application. The electronic device can be a control device device formed by various electronic devices.
[0190] Furthermore, the present application also provides a computer-readable storage medium. In a computer-readable storage medium embodiment according to the present application, the computer-readable storage medium can be configured to store the execution of the above method. Figures 1 to 7 The program of the large model-based character interaction detection method of the illustrated embodiment can be loaded and run by a processor to implement the above-mentioned large model-based character interaction detection method. For ease of explanation, only the parts related to the embodiment of the present application are shown. For specific technical details not disclosed, please refer to the method part of the embodiment of the present application. The computer-readable storage medium can be a storage device formed by various electronic devices. Optionally, the computer-readable storage medium in the embodiment of the present application is a non-transitory computer-readable storage medium.
[0191] Furthermore, it should be understood that since the configuration of each module is merely for the purpose of illustrating the functional units of the apparatus of the present application, the physical devices corresponding to these modules may be the processor itself, or a portion of the software in the processor, a portion of the hardware, or a combination of software and hardware. Therefore, the number of modules in the figure is merely illustrative.
[0192] Those skilled in the art will appreciate that the various modules in the device can be adaptively split or merged. Such splitting or merging of specific modules will not cause the technical solution to deviate from the principles of this application. Therefore, the technical solutions after splitting or merging will fall within the scope of protection of this application.
[0193] Thus far, the technical solutions of the present application have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is readily understood by those skilled in the art that the scope of protection of the present application is obviously not limited to these specific embodiments. Without departing from the principles of the present application, those skilled in the art may make equivalent changes or substitutions to the relevant technical features, and the technical solutions after such changes or substitutions will fall within the scope of protection of the present application.
Claims
1. A method for detecting human interaction based on a large model, characterized in that: include: Obtain the image to be detected; Inputting the image to be detected into a preset model to obtain an interactive action; Wherein, obtaining the preset model includes: In each round, training data is obtained; wherein the training data includes image data, the position of people, the position of objects, and the interactive actions of people; Inputting the image data into the backbone network to obtain a multi-scale feature map; Inputting the multi-scale feature map into an image codec to obtain encoded image features and reference detection frame information; Obtaining query data according to the encoded image features and the reference detection frame information; Inputting the query data into a human body decoder to obtain human body feature code and human body position information; Inputting the query data into an object decoder to obtain object feature code, object location information and object category probability; Obtaining an action category probability according to the human body feature code, the object feature code, the human body position information, and the object position information; updating the parameters of the human decoder, the object decoder, and the action decoder according to the human position information, the object position information, the object category probability, the action category probability, and the human positions, object positions, and human interaction actions in the training data; After determining that the preset conditions are met, a preset model is obtained.
2. The method for detecting human interaction based on a large model according to claim 1, characterized in that: The image data is input into the backbone network to obtain a multi-scale feature map, including: Inputting the image data into the backbone network to obtain feature maps of different scales; The multi-scale feature map is obtained according to the feature maps of different scales.
3. The method for detecting human interaction based on a large model according to claim 1, wherein: Inputting the feature map into an image codec to obtain encoded image features and reference detection frame information includes: Dividing the feature map into a plurality of feature map blocks, and generating position information according to positions of the feature map blocks; Convert the feature image block into a one-dimensional vector, and then convert the one-dimensional vector into an embedding vector; The embedding vector and the position information are input into an image codec to obtain encoded image features and reference detection frame information; wherein the reference detection frame information includes the position of the reference detection frame and the probability that the reference detection frame belongs to the foreground.
4. The method for detecting human interaction based on a large model according to claim 1, wherein: The obtaining query data according to the encoded image features and the reference detection frame information includes: sorting the reference detection frames according to the probability that the reference frames belong to the foreground, to obtain sorted reference detection frames; Selecting a preset number of reference detection frames from the sorted reference detection frames; For any reference detection frame in the reference detection frames of the preset data, the position of the reference detection frame and the encoded image feature corresponding to the reference detection frame are combined to obtain query data.
5. The method for detecting human interaction based on a large model according to claim 2, characterized in that: Inputting the query data into the object decoder to obtain object feature code, object location information and object category probability includes: Inputting the query data into an object decoder to obtain an object feature code and an object reference position; Adjusting the object reference position according to the object reference position and the query data to obtain object position information; According to the query data, the object category probability is obtained.
6. The method for detecting human interaction based on a large model according to claim 1, characterized in that: The obtaining of the action category probability according to the human body feature code, the object feature code, the human body position information, and the object position information includes: Combining the human body feature code and the object feature code to obtain a combined feature code; combining the human body position information and the object position information to obtain combined position information; The combined feature code and the combined position information are input into an action decoder to obtain an action category probability.
7. The method for detecting human interaction based on a large model according to claim 1, characterized in that: The updating process of the parameters of the human decoder, the object decoder, and the action decoder according to the human position information, the object position information, the object category probability, the action category probability, and the human position, the object position, and the human interaction action in the training data comprises: Obtaining a plurality of triplets according to the human body position information, the object position information, the object category probability, and the action category probability; Obtaining a confidence score of the triplet according to the object category probability and the action category probability; According to the preset confidence, the predicted triples are obtained; Obtaining a loss value according to the predicted triplet, a position of a person in training data, a position of an object, a person interaction action, and a loss function; According to the loss value, the parameters of the human body decoder, the object decoder and the action decoder are updated.
8. A large-scale model-based human interaction detection device, characterized in that: include: An acquisition module, used for acquiring an image to be detected; A prediction module, configured to input the image to be detected into a preset model to obtain an interactive action; A training module is used to obtain training data in each round; wherein the training data includes image data, the position of people, the position of objects, and the interactive actions of people; The training module is further configured to input the image data into a backbone network to obtain a multi-scale feature map; The training module is further configured to input the multi-scale feature map into an image codec to obtain encoded image features and reference detection frame information; The training module is further configured to obtain query data based on the encoded image features and the reference detection frame information; The training module is further configured to input the query data into a human body decoder to obtain human body feature codes and human body position information; The training module is further configured to input the query data into an object decoder to obtain object feature codes, object location information, and object category probabilities; The training module is further configured to obtain an action category probability based on the human body feature code, the object feature code, the human body position information, and the object position information; The training module is further configured to update parameters of the human decoder, the object decoder, and the action decoder based on the human position information, object position information, object category probability, action category probability, and the positions of people, objects, and human interaction actions in the training data; The training module is further configured to obtain a preset model after determining that a preset condition is met.
9. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to execute the large model-based human interaction detection method according to any one of claims 1 to 7 through the computer program.
10. A computer-readable storage medium storing a plurality of program codes, characterized in that: The program code is suitable for being loaded and run by a processor to execute the large model-based human interaction detection method according to any one of claims 1 to 7.