Intent Object Prediction Method Based on Multi-Path Network Structure and Multi-Deep Supervision Mechanism

Abstract

The present invention discloses a method for predicting an intention object based on a multi-path network structure and a multi-depth supervision mechanism, comprising the following steps: obtaining an initial image sample, obtaining a visual attention probability map through a human body attention module according to the initial image sample, detecting key points of the hand in the initial image sample according to the initial image sample and performing Gaussian filtering to obtain a hand position probability map, and extracting object contour information according to the initial image sample through a global semantics and local contour module; performing fusion based on spatial attention on the visual attention probability map, the hand position probability map and the object contour information, and then performing deep supervision processing to obtain an accurate probability map of the next short-term activity object of the initial image sample, thereby completing the prediction of the intention object. The present invention integrates global semantics and local contour information, clues of human body attention and hand key points based on a deep supervision mechanism, and predicts the next short-term activity object in an ego-centric image.

Description

Technical Field

[0001] The present invention belongs to the technical field of intention object prediction, and particularly relates to an intention object prediction method based on a multi-path network structure and a multi-depth supervision mechanism. Background Art

[0002] Predicting the objects that humans are about to interact with and detecting human intentions are important tasks for intelligent systems, which have great significance for human-machine collaborative work or human-machine interaction. For example, when a disabled person wants to pick up a book on the table, the robot detects the disabled person's intention and responds before they act, passing the book over. It can help the disabled person complete some daily activities in life. In the past, Bertasius et al. proposed the concept of action-objects, that is, to capture the objects of a person's conscious visual or tactile interaction. They used first-person RGB and DHG images as the input of the network, encoding two-dimensional visual appearance and three-dimensional spatial cues respectively. Then, the fully convolutional RGB path uses visual appearance cues, while the fully convolutional DHG path uses 3D spatial information to detect action objects. The information of the two paths is combined through a joint path, and the joint path also realizes first-person coordinate embedding, and then outputs a per-pixel action object probability map.

[0003] Furnari et al. believe that when a user performs a specific task, the way he moves and interacts with the environment is affected by his goals and the expected interaction behaviors with the objects. Therefore, the shape of the trajectories and their positions in the frames help to predict the next active object and distinguish them from those that will remain passive. So they predict the next active object by analyzing trajectory segments of a fixed length within a time-sliding window.

[0004] Bertasius et al. tried to unsupervised learn important objects from first-person videos. They speculated that important objects can be detected in an unsupervised situation because important objects show common visual semantics and spatial layouts. They proposed a VSN (Visual Spatial Network) composed of a spatial and a visual path. The network first guesses the positions of important objects in the first-person image and uses the MCG projection scheme to propose important object segmentation masks, and these masks are used as monitoring signals to train the visual path. The predictions from the visual path are projected through MCG and transmitted to the spatial path. Then, the spatial path learns the "positions" of important objects in the first-person image. This alternating cross-channel supervision scheme is repeated to obtain the final prediction result.

[0005] Next active object prediction under the first vision, visual cues are an indispensable part. When humans want to touch or obtain an object, their attention will focus on that object. Therefore, human attention indicates the possible position of the next active object. In addition, the position of the human hand is also directly related to the location of the next active object, because humans manipulate objects with their hands. After observing the results of some networks, we found that the edges of the probability maps output by some networks are relatively rough and cannot well segment the contour of the next active object. A network model with a multi-path network structure and a multi-depth monitoring mechanism is proposed. This model takes the ego-centric image as input and outputs the probability map of the position of the next active object. Specifically, the model has three paths. One path generates the probability map of human attention, one path generates the probability map of hand key points, and another path extracts global semantics and local contour information. The output probability maps of the three paths are fused as the input of the fusion module. And in the network, a deep supervision branch is added after the output of the human attention module and the spatial attention of the fusion module to supervise the backbone network. Finally, the fusion module outputs the probability map of the next active object. Summary of the Invention

[0006] The purpose of the present invention is to propose an intention object prediction method based on a multi-path network structure and a multi-depth supervision mechanism, so that intelligent devices can predict the objects that humans want to grasp or touch, and help disabled or injured people complete tasks.

[0007] To achieve the above purpose, the present invention provides an intention object prediction method based on a multi-path network structure and a multi-depth supervision mechanism, including the following steps:

[0008] Obtain an initial image sample, and obtain a visual attention probability map according to the initial image sample;

[0009] After performing hand detection on the initial image sample, obtain a hand position probability map through Gaussian filtering;

[0010] Extract object contour information from the initial image sample through a global semantics and local contour module;

[0011] Based on the visual attention probability map, the hand position probability map and the object contour information, perform spatial attention fusion, and then perform deep supervision processing to obtain the accurate probability map of the next short-term active object of the initial image sample, and complete the intention object prediction.

[0012] Optionally, the method for obtaining a visual attention probability map according to the initial image sample specifically includes:

[0013] Process the initial image sample using an improved backbone network, connect the layers in the encoder network and the layers in the improved decoder network to obtain the decoded visual attention probability map; several skip layers are added to the improved backbone network, and 2D Dropblocks are added to the improved decoder network to prevent overfitting.

[0014] Optionally, the method for obtaining the hand position probability map by Gaussian filtering according to the initial image sample specifically includes:

[0015] Detect the position of the middle finger fingertip point in the initial image sample, assign a value to the position of the middle finger fingertip point and perform Gaussian filtering to obtain the hand position probability map.

[0016] Optionally, the method for obtaining the hand position probability map by Gaussian filtering according to the initial image sample further includes that when the position of the middle finger fingertip point in the initial image sample cannot be detected, the method for obtaining the hand position probability map specifically includes: using the statistical average value of the middle finger fingertip points detected by two experimental data sets as the Gaussian kernel points, defining a Gaussian filter according to the Gaussian kernel points, and combining the assignment to obtain the hand position probability map.

[0017] Optionally, the method for extracting object contour information from the initial image sample through global semantics and local contours specifically includes: segmenting each object in the initial image sample through the detectron2 network, and obtaining the contour information of each object to predict the contour information of the next active object.

[0018] Optionally, the method for performing spatial attention fusion based on the visual attention probability map, the hand position probability map, and the object contour information specifically includes:

[0019] First, the output F of the human attention module v , the probability map F of the hand key points h , the global semantic information and local contour information F s are batch-normalized through three independent convolutional layers, transformed into subspaces of the same dimension, and then added together to obtain F c . Subsequently, it is input into spatial attention for target localization, and transformation or weight acquisition is performed to find the most important part of the target for processing. Then, the output F of the spatial attention a is refined to output the probability map of the position where the next active object is located.

[0020] Optionally, obtaining the accurate probability map of the next short-term active object of the initial image sample specifically includes: iteratively training the probability map processed by the spatial attention based on the deep supervision mechanism to obtain a loss function, and then performing deep supervision mechanism processing according to the loss function to obtain the accurate probability map of the next short-term active object of the initial image sample.

[0021] Optionally, the calculation of the loss function includes:

[0022]

[0023] where y n is the label, and p n is the probability of predicting the next active object.

[0024] Technical effects of the present invention: The present invention discloses an intention object prediction method based on a multi-path network structure and a multi-deep supervision mechanism. A new multi-path network structure with a deep supervision mechanism is used to predict the next active object. The hand key points are used as the Gaussian kernel center, and a new Gaussian filtering mechanism is adopted to generate the position probability map of the hand key points. The contour information of the objects in the input image is extracted to increase the accuracy of the network. The effectiveness of the network deep supervision module improves the directness of the learning process in the middle part of the network. The network is more fully trained, improving the training speed and the prediction accuracy of the intention object. Description of the Drawings

[0025] The drawings constituting a part of this application are used to provide a further understanding of this application. The schematic embodiments of this application and their descriptions are used to explain this application and do not constitute an improper limitation to this application. In the drawings:

[0026] Figure 1 is a schematic structural diagram of the intention object prediction method based on the multi-path network structure and the multi-deep supervision mechanism according to the embodiment of the present invention;

[0027] Figure 2 is a Gaussian filter graph generated by hand key points according to the embodiment of the present invention, where (a) is the input picture, (b) is the hand key point detection, and (c) is the probability map of the middle finger fingertip;

[0028] Figure 3 is a schematic diagram of the experimental results in the ADL dataset according to the embodiment of the present invention;

[0029] Figure 4 is a schematic diagram of the experimental results in the EPIC-KITCHENS dataset according to the embodiment of the present invention. Detailed Embodiments

[0030] It should be noted that, without conflict, the embodiments in the present application and the features in the embodiments can be combined with each other. The present application will be described in detail below with reference to the drawings and in combination with the embodiments.

[0031] It should be noted that the steps shown in the flowchart of the drawings can be executed in a computer system such as a set of computer-executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than here.

[0032] As Figure 1 shown, in this embodiment, an intention object prediction method based on a multi-path network structure and a multi-depth supervision mechanism is provided, including the following steps:

[0033] Obtain an initial image sample, obtain a visual attention probability map according to the initial image sample through a human body attention module, obtain a hand position probability map through Gaussian filtering after detecting the key points of the hand in the initial image sample according to the initial image sample, and extract object contour information according to the initial image sample through a global semantics and local contour module;

[0034] Based on the visual attention probability map, the hand position probability map, and the object contour information, perform fusion based on spatial attention, and then perform deep supervision processing to obtain an accurate probability map of the next short-term active object of the initial image sample, completing the intention object prediction.

[0035] Model architecture: After the image is input into the network, the human body attention part, the hand key point heat map part, and the global semantics and local contour information extraction part process the image respectively. The human body attention module provides visual cues, the hand key point heat map generates the positions of the hand key points, and the global semantics and local contour module extracts the contour information of the object, that is, the object edge. Their outputs are used as the inputs of the fusion module. The output of the human body attention, the output of the spatial attention of the fusion module, and the final output of the network are respectively calculated with the ground truth using a loss function and summed to obtain the loss function of the network. This part is the deep supervision module.

[0036] Human attention. The backbone network adopted by the human attention module is ResNet-18, and the output is the probability map of attention. It predicts the position of the next active object in the input image based on human attention. Since the next active object may be a door or a cup, and the size of the object we predict is uncertain, we added four skip connections in the network to connect the layers in the encoder network with the layers in the decoder network. In the decoder network, we added 2D DropBlocks[6] layers in the last two transposed convolutional layers instead of the traditional Dropout, because DropBlock prevents overfitting by deleting continuous regions from the feature map, and random block inactivation has a better inhibitory effect on overfitting than random point inactivation. The input of the human attention module is an RGB image, and the output Fv is a visual attention probability map with 64 channels.

[0037] Hand keypoint heatmap. This module generates a probability map of hand keypoints, which predicts the possible position of the next active object in the image based on the position clues of human hand keypoints. Through observation, we found that when a person grabs an object, the tip of the middle finger is often closer to the next active object. So we use the tip point of the middle finger of the 21 detected hand keypoints to generate the hand position probability map, as shown in Figure 2 (b). We regard the detected tip point of the middle finger as the center position of the Gaussian kernel, assign a value of 1 to this point, and use a two-dimensional Gaussian filtering mechanism to generate the hand position probability map, as shown in Figure 2 (c).

[0038] The size of the Gaussian kernel is determined according to the distance between the center position of the Gaussian kernel and the center point of the entire image. To prevent the Gaussian distribution from being too large or too small, when the distance is less than or greater than a certain range, sigma is set to a fixed value. Because if sigma is too large, the Gaussian distribution will become larger and may cover most of the image area. If the next active object we want to predict is just a small object, the hand information may have an adverse effect on our final prediction. When sigma is too small, the Gaussian distribution will become smaller and may cover only a small part of the image area, and finally only a small part of the next active object will be predicted.

[0039] In addition, if no hand is detected in the input image, we use the statistical average of the detected tip points of the middle finger in two experimental datasets as the Gaussian kernel point. The Gaussian filter is defined as follows:

[0040]

[0041] (u, v) is the coordinate of the tip of the middle finger. The size of sigma is:

[0042]

[0043] (center_x, center_y) is the center point of the input image. N is the number of Gaussian kernels detected in the input image, that is, the number of middle finger fingertips. min_x and min_y are the minimum limits of the horizontal and vertical distances of the center position of the Gaussian kernel from the horizontal and vertical coordinates of the center point of the entire image respectively; max_x and max_y are the maximum limits of the horizontal and vertical distances of the center position of the Gaussian kernel from the horizontal and vertical coordinates of the center point of the entire image respectively.

[0044] When predicting the next active object relying on global semantics and local contour information, visual cues, and hand key points, there will be a situation where a complete object cannot be predicted. For example, if the next active object is a mineral water bottle, the network only predicts a part of the water bottle and does not output the mineral water bottle as a whole. And we need the network to predict the next active object as a whole rather than a part of it. Therefore, we added object detection and segmentation. Semantic segmentation divides a given image into several visually meaningful or interesting regions. Using the detectron2 network, it can well segment each object in the picture, making the contour of the object clear. Therefore, when predicting the next active object, our network can better distinguish the edges of each object, rather than blurring the edge information between two objects or only predicting a part of the next active object. It can be said that in the entire network, the role of this part of the structure is to extract global semantic information and local contour information.

[0045] The fusion module fuses human attention, hand key point cues, global semantics, and local contour information to output an accurate probability map of the next short-term active object. When a human is performing an activity, the eyes search for an object, and the hand approaches the object. Vision and hand movements assist each other in approaching the next active object. Both of them locate the area of the next active object. The global semantics and local contour module provide the edge information of the object. Therefore, we fuse the three.

[0046] The architecture of the fusion module is as Figure 1 shown. First, the output F v of the human attention module, the probability map F h of the hand key points, the global semantic information, and the local contour information F s are batch-normalized through three independent convolutional layers and transformed into subspaces of the same dimension, and then added together to obtain F c . Subsequently, it is input into the spatial attention module. Spatial attention is added to the network because we only need to capture a part of the image. Spatial attention can locate the target, perform some transformations, or obtain weights to find the most important part of the image for processing. The spatial attention part is defined as follows:

[0047] F a = F c ⊙ [σ(f 7×7 ([AvgPool(F c );MaxPool(F c )]))] (3)

[0048] AvgPool represents average pooling, MaxPool represents max pooling, f 7×7 represents a 7×7 convolutional layer, σ represents a sigmoid activation layer, and ⊙ represents element-wise multiplication.

[0049] Then, the output F of the spatial attention is refined to output the final probability map. The process is as follows: a to output the final probability map. The process is as follows:

[0050] F o = f 3×3 (PReLU(BN(f 3×3 (PReLU(F a ))))) (4)

[0051] PReLU represents a PReLU activation layer, f 3×3 represents a 3×3 convolutional layer, and BN represents batch normalization layer.

[0052] Deep Supervision module. Deep Supervision is a technique that adds an auxiliary classifier as a network branch at some intermediate hidden layers of a deep neural network to supervise the main network, aiming to solve problems such as vanishing gradients and slow convergence speed in the training of deep neural networks.

[0053] In the network of this paper, there are two deep supervision branches. One is at the output of the human attention, i.e., F v and the other is at the output of the spatial attention, i.e., F a . Adding deep supervision at the output of the human attention is because our network is a multi-path network structure, and the output of the human attention enters the fusion module. Without adding deep supervision, it may be difficult to train the intermediate layer of the network, i.e., the human attention module. The same applies to adding deep supervision at the output of the spatial attention, mainly to make the training easier and more sufficient.

[0054] The loss function used is the cross-entropy loss function. It is defined as follows:

[0055]

[0056] y n is the label (the value of the next active object is 1, and the background is 0), p nis the probability of predicting the next activity object. The loss function of the network is the sum of the loss functions of the backbone network and the two deep supervision branches, which is calculated as follows:

[0057] Loss = L v + L a + L o (6)

[0058] L v ,L a ,L o are the results obtained by performing convolution on F v ,F a ,F o , changing the dimension and then calculating the loss function with the label.

[0059] We evaluated our method on the ADL and EPIC-KITCHENS datasets, which are two egocentric videos collected during human execution of unscripted daily activities. In these two public datasets, one image is sampled per second from the original video and all possible activity objects in the image are annotated with bounding boxes. The video resolution in the ADL dataset is 1280×960, while the video resolution in the EPIC-KITCHENS dataset is 1920×1080.

[0060] We screened the images sampled per second. When a person is reading a book or watching TV, this may be a long-duration activity. Similar to this, predicting the next activity object based on such a long-duration activity may have lost its practical significance. Finally, we selected the images in the past 1 - 6 s for the next activity object prediction, deleting the images that do not contain the next activity object and the images where the next activity object remains unchanged for more than 6 seconds. Finally, 6924 images remained in the ADL dataset and 11601 images remained in the EPIC-KITCHENS dataset. We allocated the training set and the test set according to a ratio of 2.7:1.

[0061] First, resize the input image to 3×224×320. When generating the hand position probability map, our weight = 320, height = 224, center_x = 160, center_y = 112, and set min_x = 20, min_y = 14, max_x = 40, max_y = 28. When two hands are detected in the input image, N is set to 2, otherwise N = 1. The statistical average of the middle finger fingertip coordinates detected in the EPIC-KITCHENS and ADL datasets is (173, 132), which is set as the coordinate of the Gaussian kernel center point when no hand is detected. To obtain a more reliable short-term probability map of the next active object, we filter several pixels using a score threshold, and we set the threshold to 0.6.

[0062] Select five baseline methods from the relevant fields, including the active object prediction baseline and the object detection baseline. The information of the baseline methods is shown in Table 1.

[0063] Table 1

[0064]

[0065] ⑴ Active object prediction:

[0066] a. Egonet is a joint two-stream network that takes first-person RGB and DHG images as inputs, encodes two-dimensional visual appearance and three-dimensional spatial cues respectively. Then the network globally integrates visual appearance (RGB) and 3D spatial layout (depth and height) cues, and then outputs the probability map of active objects for each pixel.

[0067] b. VSN (Visual Spatial Network) consists of a spatial and a visual path for unsupervised learning of active objects. One path learns common visual semantics, while the other path focuses on spatial location cues, and both paths act as recognition agents. The interaction between segmentation and recognition agents is achieved through an alternating cross-path monitoring scheme. This alternating cross-channel supervision scheme is repeated for several rounds.

[0068] c. Furnari et al. assume that an object detector trained on a set of N object categories is available. A tracker is used to associate detections related to the same object instance to generate object trajectories. A classifier is trained to distinguish trajectories that lead to object activation from all other trajectories, and trajectory segments of a fixed length within a time-sliding window are analyzed to predict the next active object. At each time step, the system analyzes the trajectories observed in the most recent frames to identify the next active object before an interaction actually occurs.

[0069] ⑵ Object detection:

[0070] a. CPD is a new cascaded partial decoder framework for fast and accurate salient object detection. On the one hand, the framework constructs a partial decoder which, for acceleration, discards the larger-resolution features of the shallower layers. On the other hand, CPD directly uses the generated saliency map to refine the features of the backbone network. This strategy effectively suppresses the interference factors in the features and significantly improves the feature representation ability.

[0071] b. The architecture of the U2 network is a two-layer nested U-shaped structure. Since it mixes receptive fields of different sizes in the residual U-blocks (RSUs), it can capture more context information from different scales, and due to the pooling operations used in these RSU blocks, it increases the depth of the entire architecture, which enables us to train a deep network from scratch without using the backbone of an image classification task.

[0072] For the probability map of the next active object output by the network, we can binarize it for pixel binary classification. To evaluate the performance of our model, we use the standard evaluation metrics for binary classification: balanced F-score (F1), average precision (AP), and pixel accuracy (PA). The F1 score is the harmonic mean of precision P and recall R: It measures the overall accuracy of our model. The higher the F1 score, the more robust the model. AP is the area under the precision-recall curve; PA is the average pixel classification accuracy: where N is the number of test images, and are the number of correctly predicted pixels and the total number of pixels in image i, respectively.

[0073] Table 2 shows the comparison of the performance of the baseline model and our model on the ADL and EPIC-KITCHENS datasets, as shown in Table 2.

[0074] Table 2

[0075]

[0076] To test the effects of the various components of the model, we also report the performance of the model under different configurations. The results and analysis are as follows:

[0077] ⑴ Comparison with the baseline:

[0078] The performance of our method and the baseline methods is shown in Table 2. Our method is significantly better than the baseline methods. Compared with the best-performing (Furnari) in the baseline methods, our method has increased by 12.68% (F1), 36.48% (AP), and 0.29% (PA) on the ADL dataset, and by 0.79% (F1), 19.51% (AP), and 2.35% (PA) on the EPIC dataset. This demonstrates the effectiveness of our network structure.

[0079] ⑵ Effectiveness of deep supervision:

[0080] When all other parts of the network are kept unchanged, removing the deep supervision module gives the following results. This can prove the effectiveness of the deep supervision module in our network, which improves the directness of the learning process in the middle part of the network, and the network is more fully trained.

[0081] ⑶ Influence of model components: "Our Model(V)" means that the fusion module only has Fv as input, and "Our Model(V+H)" means that the fusion module has both Fv and Fh as inputs, with the other parts of the network unchanged. Human attention scores highly in the experiment and is the most important model component. Even on the EPIC dataset, the F1 score is 0.08% higher than that of our complete model, but there is still a gap in other scores compared with the complete model. When the fusion module has both Fv and Fh as inputs, the score is not as high as when the fusion module only has Fv as input. We analyze that this is because there is no semantic segmentation at this time, and the contour information of the object is not clear. Sometimes, there may be a situation where the hand in the picture is not close enough to the next active object, that is, the hand position probability map only covers a part of the target, which will result in a lower score than when only Fv is used as input. When the fusion module has both Fv and Fs as inputs, the scores are sometimes higher and sometimes lower, but generally, it can prove the effectiveness of each component of our model.

[0082] ⑴ Comparison with the baseline:

[0083] When no hand is detected. In some cases, a person's hand is outside the image, or the hand detector fails to detect the hand in the image. In this case, we use the statistical average of the hand position as the center of the Gaussian filter to generate the hand position probability map. As can be seen from Figure 3 、 Figure 4 our model can also make good predictions for pictures where no hand is detected.

[0084] ⑵ Next active object adjacent to other objects:

[0085] When the distance between the next active object and other objects is relatively close, the difficulty of prediction increases, and the presence of other objects will interfere with the network. In this case, our network performs well and can accurately predict the location of the next active object, as Figure 3-4 shown Figure 3 The experimental results in the ADL dataset. The first and third rows are the network output results, and the second and fourth rows are the correct annotations. Figure 4 The experimental results in the ADL dataset. The first and third rows are the network output results, and the second and fourth rows are the correct annotations.

[0086] ⑶ Multiple next active objects are shown in the same image:

[0087] Sometimes, a person may interact with multiple objects simultaneously. For example, a person may hold different objects with both hands at the same time, or touch objects with the left and right hands successively, which are captured in a single frame of the image.

[0088] The present invention discloses an intention object prediction method based on a multi-path network structure and a multi-depth supervision mechanism. A new multi-path network structure with a depth supervision mechanism is used to predict the next active object. The hand key points are used as the centers of Gaussian kernels, and a new Gaussian filtering mechanism is adopted to generate the position probability map of the hand key points. The contour information of the objects in the input image is extracted to improve the accuracy of the network. The effectiveness of the network depth supervision module improves the directness of the learning process in the middle part of the network. The network is more fully trained, and the training speed and the prediction accuracy of the intention object are improved.

[0089] As mentioned above, the above are only the preferred specific embodiments of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by those skilled in the art within the technical scope disclosed in the present application should be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. An intention object prediction method based on a multi-path network structure and a multi-depth supervision mechanism, comprising the following steps: Obtain an initial image sample, and obtain a visual attention probability map according to the initial image sample; Obtain a hand position probability map by Gaussian filtering according to the initial image sample; Extract object contour information through global semantics and local contours according to the initial image sample; Based on the visual attention probability map, the hand position probability map, and the object contour information, perform fusion, and then perform deep supervision processing to obtain an accurate probability map of the next short-term active object of the initial image sample, completing the intention object prediction; The method for performing spatial attention fusion based on the visual attention probability map, the hand position probability map, and the object contour information specifically includes: First, the output F of the human attention module v , the probability map F of the hand key points h , the global semantic information and local contour information F s are batch-normalized through three independent convolutional layers, transformed into subspaces of the same dimension, and then added together to obtain F c . Subsequently, spatial attention is input for target localization, and transformation or weight acquisition is performed to find the most important part of the target for processing. Then, the output F of the spatial attention a is refined to output the probability map of the location of the next active object; Obtaining an accurate probability map of the next short-term active object of the initial image sample specifically includes: performing iterative training on the probability map after spatial attention processing based on a deep supervision mechanism to obtain a loss function, and then performing deep supervision mechanism processing according to the loss function to obtain an accurate probability map of the next short-term active object of the initial image sample.

2. The intention object prediction method based on a multi-path network structure and a multi-depth supervision mechanism according to claim 1, wherein, The method for obtaining a visual attention probability map according to the initial image sample specifically includes: Process the initial image sample using an improved backbone network, connect the layers in the encoder network and the layers in the improved decoder network to obtain the decoded visual attention probability map; several skip layers are added to the improved backbone network, and 2D Dropblocks are added to the improved decoder network to prevent overfitting.

3. The method for predicting an intent object based on a multi-path network structure and a multi-depth supervision mechanism according to claim 1, wherein The method for obtaining a hand position probability map according to the initial image sample specifically includes: Detect the position of the middle finger fingertip point in the initial image sample, assign a value to the position of the middle finger fingertip point, and perform Gaussian filtering to obtain the hand position probability map.

4. The method for predicting an intention object based on a multi-path network structure and a multi-depth supervision mechanism according to claim 3, wherein The method for obtaining a hand position probability map by Gaussian filtering according to the initial image sample further includes that when the position of the middle finger fingertip point in the initial image sample cannot be detected, obtaining the hand position probability map specifically includes: using the statistical average value of detecting the middle finger fingertip point through two experimental data sets as the Gaussian kernel point, defining a Gaussian filter according to the Gaussian kernel point, and combining the assignment to obtain the hand position probability map.

5. The intention object prediction method based on a multi-path network structure and a multi-depth supervision mechanism according to claim 1, characterized in that The method for extracting object contour information through global semantics and local contours according to the initial image sample specifically includes: segmenting each object in the initial image sample through the detectron2 network, and obtaining the contour information of each object for predicting the contour information of the next active object.

6. The intention object prediction method based on a multi-path network structure and a multi-depth supervision mechanism according to claim 1, characterized in that, The calculation of the loss function includes: Among them, y n is the label, and p n is the probability predicted for the next active object.

Images