Methods, apparatus, and electronic devices for detecting target objects in video.

Abstract

A method, apparatus, and electronic device for detecting target objects from video are disclosed. The method includes: acquiring a current feature map of the current frame and a previous feature map of a previous frame of the video; weighting the current feature map based on the correlation between the current feature map and the previous feature map to obtain an attention map; fusing the attention map with the current feature map to obtain a final feature map; and detecting the target object from the current frame based on the final feature map. This improves the accuracy of object detection.

Description

Technical Field

[0001] The present application relates to the field of video processing technology, and more specifically, to a method, device and electronic device for detecting a target object from a video. Background Art

[0002] In the process of video production, it is often necessary to remove the background area of ​​the current video and replace it with other scenes. The core algorithm here is the video object segmentation algorithm, which is used to segment the foreground area of ​​interest in the video, and then a simple image restoration algorithm can be used to synthesize the video under the new background.

[0003] Changing the video background is used in many scenarios. For example, in live broadcasting, there is often a need to change the background. However, in dynamic live broadcasting scenes, especially in outdoor live broadcasting scenes, since the background changes dynamically, it is impossible to segment the foreground area through simple algorithms such as background difference.

[0004] For video object segmentation, we first need to detect the foreground area of ​​interest in the video, such as a person or object in the video, and then segment it using a mask.

[0005] Therefore, in order to improve the effect of video object segmentation, it is desirable to provide an improved solution for detecting target objects from videos. Summary of the invention

[0006] In order to solve the above technical problems, this application is proposed. The embodiments of this application provide a method, device and electronic device for detecting a target object from a video, which obtains a final feature map from feature maps of a current frame and a previous frame of a video through an attention module, and detects the target object in the video from the final feature map, thereby improving the accuracy of object detection.

[0007] According to one aspect of the present application, a method for detecting a target object from a video is provided, comprising: obtaining a current feature map of a current frame of a video and a previous feature map of a previous frame; weighting the current feature map based on the correlation between the current feature map and the previous feature map to obtain an attention map; fusing the attention map with the current feature map to obtain a final feature map; and detecting the target object from the current frame based on the final feature map.

[0008] In the above-mentioned method for detecting a target object from a video, obtaining a current feature map of a current frame of a video and a previous feature map of a previous frame includes: obtaining a current frame and a previous frame of the video; obtaining the previous feature map from the previous frame through a convolutional neural network; and obtaining the current feature map from the current frame through the convolutional neural network.

[0009] In the above-mentioned method for detecting a target object from a video, weighting the current feature map based on the correlation between the current feature map and the previous feature map to obtain an attention map includes: passing the current feature map through a first convolution layer to obtain a first feature map; passing the previous feature map through a second convolution layer to obtain a second feature map; and obtaining an attention matrix for representing the correlation between the current feature map and the previous feature map based on the first feature map and the second feature map.

[0010] In the above-mentioned method for detecting target objects from videos, weighting the current feature map based on the correlation between the current feature map and the previous feature map to obtain an attention map further includes: passing the current feature map through a third convolutional layer to obtain a third feature map; and weighting the third feature map with the attention matrix to obtain the attention map.

[0011] In the above method for detecting a target object from a video, the previous frame is a frame before the current frame.

[0012] In the above-mentioned method for detecting a target object from a video, fusing the attention map with the current feature map to obtain a final feature map includes: weighting the attention map with a predetermined weight and fusing it with the current feature map to obtain the final feature map.

[0013] In the above method for detecting a target object from a video, the predetermined weight is obtained through training.

[0014] In the above-mentioned method for detecting target objects from videos, the training process of the predetermined weights includes: obtaining a current training feature map of a current frame of a training video and a previous training feature map of a previous frame; weighting the current training feature map based on the correlation between the current training feature map and the previous training feature map to obtain a training attention map; weighting the training attention map with an initial weight and fusing it with the current training feature map to obtain the final training feature map; calculating a loss function value based on the final training feature map; and, updating the initial weights by gradient descent based on the loss function value to obtain the predetermined weights.

[0015] In the above method for detecting a target object from a video, the detection result of the target object is further used for object tracking and / or object segmentation of the target object.

[0016] According to another aspect of the present application, a method for segmenting a target object in a video is provided, comprising: obtaining a current feature map of a current frame of the video and a previous feature map of a previous frame; weighting the current feature map based on the correlation between the current feature map and the previous feature map to obtain an attention map; fusing the attention map with the current feature map to obtain a final feature map; detecting a target object from the current frame based on the final feature map; and generating a mask based on the detection result of the target object to segment the target object.

[0017] In the target object segmentation method in the above video, fusing the attention map with the current feature map to obtain the final feature map includes: weighting the attention map with a predetermined weight and fusing it with the current feature map to obtain the final feature map.

[0018] In the target object segmentation method in the above-mentioned video, the predetermined weight is obtained through training, and the training process includes: obtaining a current training feature map of a current frame of a training video and a previous training feature map of a previous frame; weighting the current training feature map based on the correlation between the current training feature map and the previous training feature map to obtain a training attention map; weighting the training attention map with an initial weight and fusing it with the current training feature map to obtain the final training feature map; obtaining a detection result of the target object based on the final training feature map; obtaining a segmentation result of the target object based on the detection result of the target object; calculating a detection loss function value based on the detection result of the target object and a segmentation loss function value based on the segmentation result of the target object; and, updating the initial weight by gradient descent based on the detection loss function value and the segmentation loss function value to obtain the predetermined weight.

[0019] According to another aspect of the present application, a method for tracking a target object in a video is provided, comprising: obtaining a current feature map of a current frame of the video and a previous feature map of a previous frame; weighting the current feature map based on the correlation between the current feature map and the previous feature map to obtain an attention map; fusing the attention map with the current feature map to obtain a final feature map; detecting a target object from the current frame based on the final feature map; and tracking the target object based on the detection result of the target object.

[0020] In the target object tracking method in the above video, fusing the attention map with the current feature map to obtain the final feature map includes: weighting the attention map with a predetermined weight and fusing it with the current feature map to obtain the final feature map.

[0021] In the tracking method of the target object in the above-mentioned video, the predetermined weight is obtained through training, and the training process includes: obtaining a current training feature map of the current frame of the training video and a previous training feature map of the previous frame; weighting the current training feature map based on the correlation between the current training feature map and the previous training feature map to obtain a training attention map; fusing the training attention map with the current training feature map after weighting it with the initial weight to obtain the final training feature map; obtaining the detection result of the target object based on the final training feature map; obtaining the tracking result of the target object based on the detection result of the target object; calculating the detection loss function value based on the detection result of the target object and calculating the tracking loss function value based on the tracking result of the target object; and, updating the initial weight by gradient descent based on the detection loss function value and the tracking loss function value to obtain the predetermined weight.

[0022] According to another aspect of the present application, a video processing method is provided, including: obtaining a current feature map of a current frame of a video and a previous feature map of a previous frame; weighting the current feature map based on the correlation between the current feature map and the previous feature map to obtain an attention map; fusing the attention map with the current feature map to obtain a final feature map; and detecting a foreground area from the current frame based on the final feature map.

[0023] In the above video processing method, it further includes: segmenting the detected foreground area from the current frame.

[0024] In the above video processing method, it further includes: superimposing the segmented foreground area with the background image.

[0025] According to another aspect of the present application, a method for searching similar objects based on a video is provided, comprising: obtaining a current feature map of a current frame of a video and a previous feature map of a previous frame; weighting the current feature map based on the correlation between the current feature map and the previous feature map to obtain an attention map; fusing the attention map with the current feature map to obtain a final feature map; detecting a target object from the current frame based on the final feature map; and searching for similar objects similar to the target object.

[0026] In the above-mentioned method for searching similar objects based on videos, it further includes: obtaining an address link of the similar object; and connecting to the network location where the similar object is located based on the address link.

[0027] According to another aspect of the present application, a device for detecting a target object from a video is provided, comprising: an acquisition unit for acquiring a current feature map of a current frame of a video and a previous feature map of a previous frame; an attention unit for weighting the current feature map based on the correlation between the current feature map and the previous feature map to obtain an attention map; a processing unit for fusing the attention map with the current feature map to obtain a final feature map; and a detection unit for detecting the target object from the current frame based on the final feature map.

[0028] According to another aspect of the present application, a device for segmenting a target object in a video is provided, comprising: an acquisition unit, configured to acquire a current feature map of a current frame of the video and a previous feature map of a previous frame; an attention unit, configured to weight the current feature map based on the correlation between the current feature map and the previous feature map to obtain an attention map; a processing unit, configured to fuse the attention map with the current feature map to obtain a final feature map; a detection unit, configured to detect a target object from the current frame based on the final feature map; and a segmentation unit, configured to generate a mask based on the detection result of the target object to segment the target object.

[0029] According to another aspect of the present application, a device for tracking a target object in a video is provided, comprising: an acquisition unit, configured to acquire a current feature map of a current frame of the video and a previous feature map of a previous frame; an attention unit, configured to weight the current feature map based on the correlation between the current feature map and the previous feature map to obtain an attention map; a processing unit, configured to fuse the attention map with the current feature map to obtain a final feature map; a detection unit, configured to detect a target object from the current frame based on the final feature map; and a tracking unit, configured to track the target object based on the detection result of the target object.

[0030] According to another aspect of the present application, a video processing device is provided, including: an acquisition unit, used to acquire a current feature map of a current frame of a video and a previous feature map of a previous frame; an attention unit, used to weight the current feature map based on the correlation between the current feature map and the previous feature map to obtain an attention map; a processing unit, used to fuse the attention map with the current feature map to obtain a final feature map; and a detection unit, used to detect a foreground area from the current frame based on the final feature map.

[0031] According to another aspect of the present application, a device for searching for similar objects based on a video is provided, comprising: an acquisition unit for acquiring a current feature map of a current frame of a video and a previous feature map of a previous frame; an attention unit for weighting the current feature map based on the correlation between the current feature map and the previous feature map to obtain an attention map; a processing unit for fusing the attention map with the current feature map to obtain a final feature map; a detection unit for detecting a target object from the current frame based on the final feature map; and a search unit for searching for similar objects similar to the target object.

[0032] According to another aspect of the present application, an electronic device is provided, comprising: a processor; and a memory, in which computer program instructions are stored, and when the computer program instructions are executed by the processor, the processor executes the method for detecting a target object from a video as described above, the method for segmenting a target object in a video as described above, the method for tracking a target object in a video as described above, the video processing method as described above, and the method for searching for similar objects based on a video as described above.

[0033] According to another aspect of the present application, a computer-readable medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes the method for detecting a target object from a video as described above, the method for segmenting a target object in a video as described above, the method for tracking a target object in a video as described above, the video processing method as described above, and the method for searching for similar objects based on a video as described above.

[0034] The method, device and electronic device for detecting target objects from a video provided in the present application can obtain a final feature map from feature maps of a current frame and previous frames of a video through an attention module, and detect the target object in the video from the final feature map, thereby improving the accuracy of object detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0036] Figure 1 The flowchart of the method for detecting a target object from a video according to an embodiment of the present application is illustrated.

[0037] Figure 2A schematic diagram of a process of acquiring a feature map in a method for detecting a target object from a video according to an embodiment of the present application is illustrated.

[0038] Figure 3 A schematic diagram of an example of an architecture for implementing an attention mechanism in a method for detecting a target object from a video according to an embodiment of the present application is illustrated.

[0039] Figure 4 A schematic diagram of obtaining a final feature map in a method for detecting a target object from a video according to an embodiment of the present application is illustrated.

[0040] Figure 5 The figure illustrates a flow chart of a method for segmenting a target object in a video according to an embodiment of the present application.

[0041] Figure 6 The figure illustrates a flow chart of a method for tracking a target object in a video according to an embodiment of the present application.

[0042] Figure 7 The figure illustrates a flow chart of a video processing method according to an embodiment of the present application.

[0043] Figure 8 A schematic diagram illustrating an application scenario of a video processing method according to an embodiment of the present application is illustrated.

[0044] Fig. 9 The flowchart of the method for searching similar objects based on a video according to an embodiment of the present application is illustrated.

[0045] Fig.10 The figure shows a block diagram of a video object detection device according to an embodiment of the present application.

[0046] Fig.11 The figure illustrates a block diagram of a device for segmenting a target object in a video according to an embodiment of the present application.

[0047] Fig.12 A block diagram of a device for tracking a target object in a video according to an embodiment of the present application is illustrated.

[0048] Fig.13 The figure shows a block diagram of a video processing device according to an embodiment of the present application.

[0049] Fig.14 A block diagram of an apparatus for searching for similar objects based on a video according to an embodiment of the present application is illustrated.

[0050] Fig.15 A block diagram of an electronic device according to an embodiment of the present application is illustrated. DETAILED DESCRIPTION

[0051] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0052] Application Overview

[0053] In the common video object segmentation algorithm, the optical flow between the current image and the previous frame image is first estimated, the optical flow is used to map the object area of ​​the previous frame to the current image, and finally the image object segmentation algorithm is used to optimize the precise edge of the object.

[0054] However, the calculation of optical flow not only takes a long time, but is also not stable enough. It is difficult to obtain accurate optical flow for areas lacking texture. Moreover, this method is only suitable for the segmentation of a single object. For multi-target video object segmentation scenes, it cannot solve the overlap between multiple objects.

[0055] In addition, there are also methods that integrate object detection, tracking, and segmentation into an end-to-end system to achieve automatic segmentation of video object instances. However, these methods are generally based on the information of the current image during the detection and segmentation process, and do not use the segmentation results of the previous frame. However, since objects are usually in continuous positions in each consecutive frame of the video, the object information in the previous frame can greatly promote the segmentation of the object in the current image frame. Therefore, if only based on the information of the current image without using the segmentation results of the previous frame, it may lead to poor segmentation results.

[0056] In response to the above technical problems, the basic concept of the present application is to obtain an attention map through an attention module using the feature maps of the current frame and the previous frame of the video, and obtain a final feature map for detecting the target object based on the attention map.

[0057] Specifically, the method, device and electronic device for detecting target objects from a video provided in the present application first obtain a current feature map of a current frame of the video and a previous feature map of a previous frame, then weight the current feature map based on the correlation between the current feature map and the previous feature map to obtain an attention map, then fuse the attention map with the current feature map to obtain a final feature map, and finally detect the target object from the current frame based on the final feature map.

[0058] In this way, the method, device and electronic device for detecting target objects from a video provided by the present application can also utilize the image information of the previous frame when detecting the target object from the current frame, and combine it with the image information of the current frame, thereby being able to make good use of the time domain information in the video to detect the target object, thereby improving the detection accuracy of the target object.

[0059] Moreover, by using the attention map obtained by the attention module, the features of the object area in the feature map can be enhanced, thereby effectively utilizing the spatial information in the frame image and improving the detection accuracy of the target object.

[0060] Therefore, the method, device and electronic device for detecting target objects from videos provided in the present application can utilize both temporal information and spatial information related to the target objects in the video to promote the detection of target objects through a space-time attention network, thereby significantly improving the detection accuracy.

[0061] In the method, device and electronic device for detecting a target object from a video provided in the present application, the target object can be various types of objects that need to be detected in the video, such as people, vehicles, etc., or the target object can also be a specific scene containing people and other objects, etc. The present application does not intend to impose any limitations on this.

[0062] Therefore, the method, device and electronic device for detecting target objects from videos provided by the present application can be applied to various application scenarios for companies, individuals and / or public affairs, such as detection of specific vehicles or people in the city brain.

[0063] After introducing the basic principles of the present application, various non-limiting embodiments of the present application will be described in detail with reference to the accompanying drawings.

[0064] Exemplary Method 1

[0065] Figure 1 The flowchart of the method for detecting a target object from a video according to an embodiment of the present application is illustrated.

[0066] like Figure 1 As shown, the method for detecting a target object from a video according to an embodiment of the present application includes the following steps.

[0067] Step S110, obtaining a current feature map of a current frame of a video and a previous feature map of a previous frame. That is, as described above, in order to fully utilize the temporal information in the video, the current feature map of the current frame of the video and the previous feature map of the previous frame are obtained to detect the target object from the current frame.

[0068] Figure 2 A schematic diagram of a process of acquiring a feature map in a method for detecting a target object from a video according to an embodiment of the present application is illustrated.

[0069] like Figure 2As shown, for a video V, first obtain the current frame Ft and the previous frame Ft-1 of the target object to be detected. Here, the previous frame Ft-1 can be the previous frame of the current frame Ft, or it can be the previous frame with a predetermined time interval from the current frame Ft. That is, it is only required that the previous frame Ft-1 contains the time domain information required for detecting the target object, and it is not necessary to limit the previous frame Ft-1 to be directly adjacent to the current frame Ft in time.

[0070] Then, the current frame Ft and the previous frame Ft-1 are respectively passed through the convolutional neural network C N To obtain the current feature corresponding to the current frame Ft Figure X t and the previous feature corresponding to the previous frame Ft-1 Figure X t-1. Here, the convolutional neural network C N It can be a convolutional neural network used to extract image features in various video detection and segmentation algorithms, such as the convolutional neural network used in the classic Mask Region Convolution Neural Network (Mask R-CNN) algorithm. N It may also be other convolutional neural networks used in video object segmentation or video object instance segmentation.

[0071] Therefore, in the method for detecting a target object from a video according to an embodiment of the present application, obtaining a current feature map of a current frame of a video and a previous feature map of a previous frame includes: obtaining the current frame and the previous frame of the video; obtaining the previous feature map from the previous frame through a convolutional neural network; and obtaining the current feature map from the current frame through the convolutional neural network.

[0072] Step S120, weighting the current feature map based on the correlation between the current feature map and the previous feature map to obtain an attention map. That is, in the embodiment of the present application, an attention map is obtained based on the current feature map and the previous feature map through an attention mechanism.

[0073] Here, the attention mechanism uses a mechanism similar to human attention, focusing on the area that needs to be focused on in space, and then investing more attention resources in this area to obtain more detailed information of the target that needs to be focused on, while suppressing other useless information. It can be applied to various types of deep learning tasks such as natural language processing, image recognition, and speech recognition to select more critical information for the current task goal from a large amount of information.

[0074] Figure 3A schematic diagram of an example of an architecture for implementing an attention mechanism in a method for detecting a target object from a video according to an embodiment of the present application is illustrated.

[0075] like Figure 3 As shown, for the current feature obtained Figure X t and previous features Figure X t-1, respectively pass through the first convolution layer C1 and the second convolution layer C2 to obtain the first feature map X1 and the second feature map X2. Here, the first convolution layer C1 and the second convolution layer C2 are not limited to a single convolution layer, but can be a plurality of convolution layers stacked together.

[0076] Then, the first feature map X1 and the second feature map X2 are multiplied, that is, matrix multiplication is performed to obtain the multiplied feature map Figure X M , and the features Figure X M Use the Softmax activation function to obtain the attention matrix X A Here, the Softmax activation function is to transform the feature Figure X M The value of each position is mapped to the interval from 0 to 1, thus obtaining the feature Figure X M The weight value of each position of Figure X M The corresponding attention matrix.

[0077] In addition, the current feature Figure X t passes through the first convolutional layer C3 to obtain the third feature map X3, and the third feature map X3 is combined with the attention matrix X A Multiply, that is, perform matrix multiplication to obtain attention Figure X B .

[0078] In this way, the attention obtained by the attention module Figure X B On the other hand, you can combine the current features Figure X t and previous features Figure X The time domain information in t-1 is used to strengthen the current features Figure X The features of the object area in t can effectively utilize the current features Figure X The spatial information of the area corresponding to the target object in t, that is, the local features of the area corresponding to the target object in the current frame image can be effectively extracted.

[0079] Therefore, in the method for detecting a target object from a video according to an embodiment of the present application, weighting the current feature map based on the correlation between the current feature map and the previous feature map to obtain an attention map includes: passing the current feature map through a first convolution layer to obtain a first feature map; passing the previous feature map through a second convolution layer to obtain a second feature map; and obtaining an attention matrix for representing the correlation between the current feature map and the previous feature map based on the first feature map and the second feature map.

[0080] Furthermore, in the above-mentioned method for detecting target objects from videos, weighting the current feature map based on the correlation between the current feature map and the previous feature map to obtain an attention map further includes: passing the current feature map through a third convolutional layer to obtain a third feature map; and weighting the third feature map with the attention matrix to obtain the attention map.

[0081] Step S130, fusing the attention map with the current feature map to obtain a final feature map. Here, the attention map contains local features of the area corresponding to the target object in the current frame image, and the current feature map contains global features of the current frame image. Therefore, by fusing the attention map with the current feature map, for example, by weighting the attention map with a predetermined weight and adding it to the current feature map, a final feature map containing both global features and local features can be obtained.

[0082] Figure 4 FIG. 2 is a schematic diagram of obtaining a final feature map in a method for detecting a target object from a video according to an embodiment of the present application. Figure 4 As shown, for the attention obtained Figure X B , multiply it by the predetermined weight Gamma. Here, the predetermined weight Gamma is used to multiply each element in the attention map to obtain the weighted attention Figure X C Then, the weighted attention Figure X C With the current characteristics Figure X t addition, that is, adding at each element position to obtain the final feature Figure X F .

[0083] That is, in the method for detecting a target object from a video according to an embodiment of the present application, fusing the attention map with the current feature map to obtain a final feature map includes: weighting the attention map with a predetermined weight and fusing it with the current feature map to obtain the final feature map.

[0084] Moreover, in an embodiment of the present application, the predetermined weight is obtained through training. The training process of the predetermined weight includes: obtaining a current training feature map of a current frame of a training video and a previous training feature map of a previous frame; weighting the current training feature map based on the correlation between the current training feature map and the previous training feature map to obtain a training attention map; weighting the training attention map with an initial weight and fusing it with the current training feature map to obtain the final training feature map; calculating a loss function value based on the final training feature map; and updating the initial weight by gradient descent based on the loss function value to obtain the predetermined weight.

[0085] Step S140, detecting the target object from the current frame based on the final feature map. Here, the target object can be detected from the final feature map by various detection network architectures, for example, a region proposal network (RPN) can be used to obtain a region of interest corresponding to the target object in the current frame.

[0086] Furthermore, the detection result of the target object can be further applied to object tracking and / or object segmentation of the target object. For example, for the region of interest obtained by the region candidate network as described above, an image mask can be further used to perform image segmentation to obtain the target object segmented from the current frame.

[0087] That is, the method for detecting target objects from videos according to the embodiments of the present application can be applied to various network architectures for video detection, as well as network architectures for video object segmentation or video object instance segmentation, to process the acquired video frames to fully utilize both the temporal information and spatial information related to the target object in the video to promote the detection or segmentation of the target object, thereby improving the detection accuracy or segmentation accuracy.

[0088] That is, in the method for detecting a target object from a video according to an embodiment of the present application, the detection result of the target object is further used for object tracking and / or object segmentation of the target object.

[0089] For example, the 2019 paper Video Instance Segmentation (https: / / arxiv.org / abs / 1905.04804) proposed a Mask Track R-CNN algorithm, which improved the Mask R-CNN network, added a tracking part, and integrated detection, tracking, and segmentation in an end-to-end system to achieve automatic segmentation of video object instances. However, during the detection and segmentation process, the algorithm is only based on the information of the current image and does not use the segmentation results of the previous frame, so the segmentation effect is poor. On YouTube vos 2019, the segmentation accuracy is 0.28.

[0090] In contrast, if the method for detecting target objects from videos according to an embodiment of the present application is applied to the algorithm, the attention map is obtained by combining the information of the previous frame image with the information of the current frame image, and the features of the object area are enhanced by using the attention map. This improves the accuracy of object instance segmentation. On Youtube vos 2019, the segmentation accuracy is 0.4.

[0091] Exemplary Method 2

[0092] Figure 5 The figure illustrates a flow chart of a method for segmenting a target object in a video according to an embodiment of the present application.

[0093] like Figure 5 As shown, the method for segmenting a target object in a video according to an embodiment of the present application includes: S210, obtaining a current feature map of a current frame of the video and a previous feature map of a previous frame; S220, weighting the current feature map based on the correlation between the current feature map and the previous feature map to obtain an attention map; S230, fusing the attention map with the current feature map to obtain a final feature map; S240, detecting a target object from the current frame based on the final feature map; and, S250, generating a mask based on the detection result of the target object to segment the target object.

[0094] In the target object segmentation method in the above-mentioned video, obtaining the current feature map of the current frame of the video and the previous feature map of the previous frame includes: obtaining the current frame and the previous frame of the video; obtaining the previous feature map from the previous frame through a convolutional neural network; and obtaining the current feature map from the current frame through the convolutional neural network.

[0095] In the target object segmentation method in the above video, weighting the current feature map based on the correlation between the current feature map and the previous feature map to obtain an attention map includes: passing the current feature map through a first convolution layer to obtain a first feature map; passing the previous feature map through a second convolution layer to obtain a second feature map; and obtaining an attention matrix for representing the correlation between the current feature map and the previous feature map based on the first feature map and the second feature map.

[0096] In the target object segmentation method in the above-mentioned video, weighting the current feature map based on the correlation between the current feature map and the previous feature map to obtain the attention map further includes: passing the current feature map through a third convolutional layer to obtain a third feature map; and weighting the third feature map with the attention matrix to obtain the attention map.

[0097] In the above method for segmenting a target object in a video, the previous frame is a frame before the current frame.

[0098] In the target object segmentation method in the above video, fusing the attention map with the current feature map to obtain the final feature map includes: weighting the attention map with a predetermined weight and fusing it with the current feature map to obtain the final feature map.

[0099] In the target object segmentation method in the above-mentioned video, the predetermined weight is obtained through training, and the training process includes: obtaining a current training feature map of a current frame of a training video and a previous training feature map of a previous frame; weighting the current training feature map based on the correlation between the current training feature map and the previous training feature map to obtain a training attention map; weighting the training attention map with an initial weight and fusing it with the current training feature map to obtain the final training feature map; obtaining a detection result of the target object based on the final training feature map; obtaining a segmentation result of the target object based on the detection result of the target object; calculating a detection loss function value based on the detection result of the target object and a segmentation loss function value based on the segmentation result of the target object; and, updating the initial weight by gradient descent based on the detection loss function value and the segmentation loss function value to obtain the predetermined weight.

[0100] That is, in the method for segmenting a target object in a video according to an embodiment of the present application, the predetermined weight is obtained by training the detection result and the segmentation result of the target object in the video together. Therefore, in the actual training process, after obtaining the detection result and the segmentation result of the target object, the detection loss function value based on the detection result and the segmentation loss function value based on the segmentation result are used to jointly train the entire network for detection and segmentation.

[0101] Here, those skilled in the art can understand that other details of the method for segmenting target objects in a video according to an embodiment of the present application are exactly the same as the corresponding details of the method for detecting target objects from a video according to an embodiment of the present application previously described in the "Exemplary Method One" section, and will not be repeated here to avoid redundancy.

[0102] Exemplary Method Three

[0103] Figure 6 The figure illustrates a flow chart of a method for tracking a target object in a video according to an embodiment of the present application.

[0104] like Figure 6 As shown, the method for tracking a target object in a video according to an embodiment of the present application includes: S310, obtaining a current feature map of a current frame of the video and a previous feature map of a previous frame; S320, weighting the current feature map based on the correlation between the current feature map and the previous feature map to obtain an attention map; S330, fusing the attention map with the current feature map to obtain a final feature map; S340, detecting a target object from the current frame based on the final feature map; and, S350, tracking the target object based on the detection result of the target object.

[0105] In the tracking method of the target object in the above video, obtaining the current feature map of the current frame of the video and the previous feature map of the previous frame includes: obtaining the current frame and the previous frame of the video; obtaining the previous feature map from the previous frame through a convolutional neural network; and obtaining the current feature map from the current frame through the convolutional neural network.

[0106] In the tracking method of the target object in the above video, weighting the current feature map based on the correlation between the current feature map and the previous feature map to obtain an attention map includes: passing the current feature map through a first convolution layer to obtain a first feature map; passing the previous feature map through a second convolution layer to obtain a second feature map; and obtaining an attention matrix for representing the correlation between the current feature map and the previous feature map based on the first feature map and the second feature map.

[0107] In the tracking method of the target object in the above video, weighting the current feature map based on the correlation between the current feature map and the previous feature map to obtain the attention map further includes: passing the current feature map through a third convolutional layer to obtain a third feature map; and weighting the third feature map with the attention matrix to obtain the attention map.

[0108] In the above-mentioned method for tracking a target object in a video, the previous frame is a frame before the current frame.

[0109] In the target object tracking method in the above video, fusing the attention map with the current feature map to obtain the final feature map includes: weighting the attention map with a predetermined weight and fusing it with the current feature map to obtain the final feature map.

[0110] In the tracking method of the target object in the above-mentioned video, the predetermined weight is obtained through training, and the training process includes: obtaining a current training feature map of the current frame of the training video and a previous training feature map of the previous frame; weighting the current training feature map based on the correlation between the current training feature map and the previous training feature map to obtain a training attention map; fusing the training attention map with the current training feature map after weighting it with the initial weight to obtain the final training feature map; obtaining the detection result of the target object based on the final training feature map; obtaining the tracking result of the target object based on the detection result of the target object; calculating the detection loss function value based on the detection result of the target object and calculating the tracking loss function value based on the tracking result of the target object; and, updating the initial weight by gradient descent based on the detection loss function value and the tracking loss function value to obtain the predetermined weight.

[0111] That is, in the method for tracking a target object in a video according to an embodiment of the present application, the predetermined weight is obtained by training the detection result and the tracking result of the target object in the video together. Therefore, in the actual training process, after obtaining the detection result and the tracking result of the target object, the detection loss function value based on the detection result and the tracking loss function value based on the tracking result are used to jointly train the entire network for detection and tracking.

[0112] Here, those skilled in the art can understand that other details of the method for tracking a target object in a video according to an embodiment of the present application are exactly the same as the corresponding details of the method for detecting a target object from a video according to an embodiment of the present application previously described in the "Exemplary Method One" section, and will not be repeated here to avoid redundancy.

[0113] Exemplary Method Four

[0114] Figure 7 The figure illustrates a flow chart of a video processing method according to an embodiment of the present application.

[0115] like Figure 7As shown, the video processing method according to the embodiment of the present application includes: S410, obtaining a current feature map of a current frame of a video and a previous feature map of a previous frame; S420, weighting the current feature map based on the correlation between the current feature map and the previous feature map to obtain an attention map; S430, fusing the attention map with the current feature map to obtain a final feature map; and, S440, detecting a foreground area from the current frame based on the final feature map.

[0116] In the above video processing method, it further includes: segmenting the detected foreground area from the current frame.

[0117] In the above video processing method, it further includes: superimposing the segmented foreground area with the background image.

[0118] That is, the video processing method according to the embodiment of the present application can be applied to the video production process, and the video producer can use the above video processing method to remove the background area of ​​the current video and replace it with other scenes.

[0119] Figure 8 FIG. 1 is a schematic diagram illustrating an application scenario of a video processing method according to an embodiment of the present application. Figure 8 As shown, for the acquired initial video V1, the foreground area in the image frame, such as the person shown in the figure, can be detected by the video processing method according to the embodiment of the present application. Then, by segmenting the foreground area, an image of the person shown in the figure is obtained, and further superimposed with the background. For example, Figure 8 As shown, the person in the foreground area can be superimposed with the scenic spot in the background image, so as to obtain a final video V2 of the person in the scenic spot.

[0120] Therefore, the video processing method according to the embodiment of the present application can be used by various video service providers to assist users in producing user generated content (UGC) or professional user generated content (PUGC), thereby increasing user stickiness.

[0121] Moreover, the video processing method according to the embodiment of the present application can also be applied to live broadcast scenes. That is to say, there is also a need to change the background in live broadcast scenes. For example, in outdoor live broadcast scenes, the dynamically changing background can be removed and replaced with the background desired by the user.

[0122] In the above-mentioned video processing method, obtaining the current feature map of the current frame of the video and the previous feature map of the previous frame includes: obtaining the current frame and the previous frame of the video; obtaining the previous feature map from the previous frame through a convolutional neural network; and obtaining the current feature map from the current frame through the convolutional neural network.

[0123] In the above-mentioned video processing method, weighting the current feature map based on the correlation between the current feature map and the previous feature map to obtain an attention map includes: passing the current feature map through a first convolution layer to obtain a first feature map; passing the previous feature map through a second convolution layer to obtain a second feature map; and obtaining an attention matrix for representing the correlation between the current feature map and the previous feature map based on the first feature map and the second feature map.

[0124] In the above-mentioned video processing method, weighting the current feature map based on the correlation between the current feature map and the previous feature map to obtain the attention map further includes: passing the current feature map through a third convolutional layer to obtain a third feature map; and weighting the third feature map with the attention matrix to obtain the attention map.

[0125] In the above video processing method, the previous frame is a frame before the current frame.

[0126] In the above-mentioned video processing method, fusing the attention map with the current feature map to obtain the final feature map includes: weighting the attention map with a predetermined weight and fusing it with the current feature map to obtain the final feature map.

[0127] In the above video processing method, the predetermined weight is obtained through training.

[0128] In the above-mentioned video processing method, the training process of the predetermined weight includes: obtaining a current training feature map of a current frame of a training video and a previous training feature map of a previous frame; weighting the current training feature map based on the correlation between the current training feature map and the previous training feature map to obtain a training attention map; weighting the training attention map with an initial weight and fusing it with the current training feature map to obtain the final training feature map; calculating a loss function value based on the final training feature map; and, updating the initial weight by gradient descent based on the loss function value to obtain the predetermined weight.

[0129] Here, those skilled in the art can understand that other details of the video processing method according to the embodiment of the present application are exactly the same as the corresponding details in the method for detecting target objects from a video according to the embodiment of the present application described in the "Exemplary Method One" section, and will not be repeated here to avoid redundancy.

[0130] Exemplary Method Five

[0131] Fig. 9 The flowchart of the method for searching similar objects based on a video according to an embodiment of the present application is illustrated.

[0132] like Fig. 9 As shown, the method for searching similar objects based on a video according to an embodiment of the present application includes: S510, obtaining a current feature map of a current frame of a video and a previous feature map of a previous frame; S520, weighting the current feature map based on the correlation between the current feature map and the previous feature map to obtain an attention map; S530, fusing the attention map with the current feature map to obtain a final feature map; S540, detecting a target object from the current frame based on the final feature map; and, S550, searching for similar objects similar to the target object.

[0133] In the above-mentioned method for searching similar objects based on videos, it further includes: obtaining an address link of the similar object; and connecting to the network location where the similar object is located based on the address link.

[0134] That is, the method for searching similar objects based on video according to an embodiment of the present application can be applied to e-commerce websites to identify target objects in videos, such as specific items that users want to buy, and search for similar products on the e-commerce website to facilitate user purchases.

[0135] Here, according to the method for searching similar objects based on video in an embodiment of the present application, after searching for similar objects, the address link of the similar object can be further obtained and connected to its network location. For example, a link to a specific product on an e-commerce website can be obtained and jumped to the link to present the specific product to the user, thereby improving the convenience of user purchase.

[0136] In addition, if the similar object is in another video, the method for searching for similar objects based on a video according to an embodiment of the present application can also be connected to the location of the video, thereby presenting a specific video to the user.

[0137] In the above-mentioned method for searching similar objects based on videos, obtaining the current feature map of the current frame of the video and the previous feature map of the previous frame includes: obtaining the current frame and the previous frame of the video; obtaining the previous feature map from the previous frame through a convolutional neural network; and obtaining the current feature map from the current frame through the convolutional neural network.

[0138] In the above-mentioned method for searching similar objects based on videos, weighting the current feature map based on the correlation between the current feature map and the previous feature map to obtain an attention map includes: passing the current feature map through a first convolution layer to obtain a first feature map; passing the previous feature map through a second convolution layer to obtain a second feature map; and obtaining an attention matrix for representing the correlation between the current feature map and the previous feature map based on the first feature map and the second feature map.

[0139] In the above-mentioned method of searching for similar objects based on videos, weighting the current feature map based on the correlation between the current feature map and the previous feature map to obtain an attention map further includes: passing the current feature map through a third convolutional layer to obtain a third feature map; and weighting the third feature map with the attention matrix to obtain the attention map.

[0140] In the above-mentioned method for searching similar objects based on video, the previous frame is a frame before the current frame.

[0141] In the above-mentioned method for searching similar objects based on videos, fusing the attention map with the current feature map to obtain a final feature map includes: weighting the attention map with a predetermined weight and fusing it with the current feature map to obtain the final feature map.

[0142] In the above-mentioned method for searching similar objects based on videos, the predetermined weight is obtained through training.

[0143] In the above-mentioned method for searching similar objects based on videos, the training process of the predetermined weights includes: obtaining a current training feature map of a current frame of a training video and a previous training feature map of a previous frame; weighting the current training feature map based on the correlation between the current training feature map and the previous training feature map to obtain a training attention map; fusing the training attention map with the current training feature map after weighting it with an initial weight to obtain the final training feature map; calculating a loss function value based on the final training feature map; and updating the initial weights by gradient descent based on the loss function value to obtain the predetermined weights.

[0144] Here, those skilled in the art can understand that other details of the method for searching similar objects based on video according to an embodiment of the present application are exactly the same as the corresponding details of the method for detecting target objects from a video according to an embodiment of the present application described in the "Exemplary Method One" section, and will not be repeated here to avoid redundancy.

[0145] Exemplary device 1

[0146] Fig.10A block diagram of a device for detecting a target object from a video according to an embodiment of the present application is illustrated.

[0147] like Fig.10 As shown, according to an embodiment of the present application, the device 600 for detecting a target object from a video includes: an acquisition unit 610, used to acquire a current feature map of a current frame of a video and a previous feature map of a previous frame; an attention unit 620, used to weight the current feature map based on the correlation between the current feature map and the previous feature map to obtain an attention map; a processing unit 630, used to fuse the attention map with the current feature map to obtain a final feature map; and a detection unit 640, used to detect the target object from the current frame based on the final feature map.

[0148] In one example, in the above-mentioned device 600 for detecting a target object from a video, the acquisition unit 610 is used to: acquire a current frame and a previous frame of the video; obtain the previous feature map from the previous frame through a convolutional neural network; and obtain the current feature map from the current frame through the convolutional neural network.

[0149] In one example, in the above-mentioned device 600 for detecting a target object from a video, the attention unit 620 is used to: pass the current feature map through a first convolution layer to obtain a first feature map; pass the previous feature map through a second convolution layer to obtain a second feature map; and obtain an attention matrix for representing the correlation between the current feature map and the previous feature map based on the first feature map and the second feature map.

[0150] In one example, in the above-mentioned device 600 for detecting a target object from a video, the attention unit 620 is further used to: pass the current feature map through a third convolutional layer to obtain a third feature map; and weight the third feature map with the attention matrix to obtain the attention map.

[0151] In one example, in the apparatus 600 for detecting a target object from a video, the previous frame is a frame before the current frame.

[0152] In one example, in the above-mentioned device 600 for detecting a target object from a video, the processing unit 630 is used to: weight the attention map with a predetermined weight and fuse it with the current feature map to obtain the final feature map.

[0153] In one example, in the above-mentioned device 600 for detecting a target object from a video, the predetermined weight is obtained through training.

[0154] In one example, in the above-mentioned device 600 for detecting a target object from a video, the training process of the predetermined weight includes: obtaining a current training feature map of a current frame of a training video and a previous training feature map of a previous frame; weighting the current training feature map based on the correlation between the current training feature map and the previous training feature map to obtain a training attention map; weighting the training attention map with an initial weight and fusing it with the current training feature map to obtain the final training feature map; calculating a loss function value based on the final training feature map; and, updating the initial weight by gradient descent based on the loss function value to obtain the predetermined weight.

[0155] In one example, in the above-mentioned apparatus 600 for detecting a target object from a video, the detection result of the target object is further used for object tracking and / or object segmentation of the target object.

[0156] Here, those skilled in the art will appreciate that the specific functions and operations of the various units and modules in the apparatus 600 for detecting a target object from a video have been described in detail above. Figures 1 to 4 The method for detecting a target object from a video has been described in detail, and therefore, its repeated description will be omitted.

[0157] As described above, the device 600 for detecting a target object from a video according to an embodiment of the present application can be implemented in various terminal devices, such as a server for video processing, etc. In one example, the device 600 for detecting a target object from a video according to an embodiment of the present application can be integrated into a terminal device as a software module and / or a hardware module. For example, the device 600 for detecting a target object from a video can be a software module in the operating system of the terminal device, or can be an application developed for the terminal device; of course, the device 600 for detecting a target object from a video can also be one of the many hardware modules of the terminal device.

[0158] Alternatively, in another example, the apparatus 600 for detecting a target object from a video and the terminal device may be separate devices, and the apparatus 600 for detecting a target object from a video may be connected to the terminal device via a wired and / or wireless network and transmit interactive information in accordance with an agreed data format.

[0159] Exemplary device 2

[0160] Fig.11 The figure illustrates a block diagram of a device for segmenting a target object in a video according to an embodiment of the present application.

[0161] like Fig.11As shown, according to an embodiment of the present application, a device 700 for segmenting a target object in a video includes: an acquisition unit 710, used to acquire a current feature map of a current frame of a video and a previous feature map of a previous frame; an attention unit 720, used to weight the current feature map based on the correlation between the current feature map and the previous feature map to obtain an attention map; a processing unit 730, used to fuse the attention map with the current feature map to obtain a final feature map; a detection unit 740, used to detect a target object from the current frame based on the final feature map; and a segmentation unit 750, used to generate a mask based on the detection result of the target object to segment the target object.

[0162] In one example, in the device 700 for segmenting a target object in the above-mentioned video, the acquisition unit 710 is used to: acquire a current frame and a previous frame of the video; obtain the previous feature map from the previous frame through a convolutional neural network; and obtain the current feature map from the current frame through the convolutional neural network.

[0163] In one example, in the device 700 for segmenting a target object in the above-mentioned video, the attention unit 720 is used to: pass the current feature map through a first convolution layer to obtain a first feature map; pass the previous feature map through a second convolution layer to obtain a second feature map; and obtain an attention matrix for representing the correlation between the current feature map and the previous feature map based on the first feature map and the second feature map.

[0164] In one example, in the device 700 for segmenting the target object in the above video, the attention unit 720 is further used to: pass the current feature map through a third convolutional layer to obtain a third feature map; and weight the third feature map with the attention matrix to obtain the attention map.

[0165] In one example, in the apparatus 700 for segmenting a target object in a video, the previous frame is a frame before the current frame.

[0166] In one example, in the device 700 for segmenting a target object in the above video, the processing unit 730 is used to: weight the attention map with a predetermined weight and fuse it with the current feature map to obtain the final feature map.

[0167] In one example, in the device 700 for segmenting a target object in the above video, the predetermined weight is obtained through training.

[0168] In one example, in the device 700 for segmenting the target object in the above-mentioned video, the training process of the predetermined weight includes: obtaining a current training feature map of a current frame of a training video and a previous training feature map of a previous frame; weighting the current training feature map based on the correlation between the current training feature map and the previous training feature map to obtain a training attention map; weighting the training attention map with the initial weight and fusing it with the current training feature map to obtain the final training feature map; obtaining a detection result of the target object based on the final training feature map; obtaining a segmentation result of the target object based on the detection result of the target object; calculating a detection loss function value based on the detection result of the target object and a segmentation loss function value based on the segmentation result of the target object; and, updating the initial weight by gradient descent based on the detection loss function value and the segmentation loss function value to obtain the predetermined weight.

[0169] Here, those skilled in the art can understand that the specific functions and operations of the various units and modules in the target object segmentation device 700 in the above video have been described in detail in the above reference. Figure 5 The method for segmenting a target object in a video has been described in detail, and therefore, its repeated description will be omitted.

[0170] Exemplary device three

[0171] Fig.12 A block diagram of a device for tracking a target object in a video according to an embodiment of the present application is illustrated.

[0172] like Fig.12 As shown, according to an embodiment of the present application, a tracking device 800 for a target object in a video includes: an acquisition unit 810, used to acquire a current feature map of a current frame of a video and a previous feature map of a previous frame; an attention unit 820, used to weight the current feature map based on the correlation between the current feature map and the previous feature map to obtain an attention map; a processing unit 830, used to fuse the attention map with the current feature map to obtain a final feature map; a detection unit 840, used to detect a target object from the current frame based on the final feature map; and a tracking unit 850, used to track the target object based on the detection result of the target object.

[0173] In one example, in the tracking device 800 for a target object in the above-mentioned video, the acquisition unit 710 is used to: acquire a current frame and a previous frame of the video; obtain the previous feature map from the previous frame through a convolutional neural network; and obtain the current feature map from the current frame through the convolutional neural network.

[0174] In one example, in the tracking device 800 for the target object in the above video, the attention unit 720 is used to: pass the current feature map through a first convolution layer to obtain a first feature map; pass the previous feature map through a second convolution layer to obtain a second feature map; and obtain an attention matrix for representing the correlation between the current feature map and the previous feature map based on the first feature map and the second feature map.

[0175] In one example, in the tracking device 800 for the target object in the above video, the attention unit 720 is further used to: pass the current feature map through a third convolutional layer to obtain a third feature map; and weight the third feature map with the attention matrix to obtain the attention map.

[0176] In one example, in the device 800 for tracking a target object in the above video, the previous frame is a frame before the current frame.

[0177] In one example, in the tracking device 800 for the target object in the above video, the processing unit 730 is used to: weight the attention map with a predetermined weight and fuse it with the current feature map to obtain the final feature map.

[0178] In one example, in the target object tracking device 800 in the above video, the predetermined weight is obtained through training.

[0179] In one example, in the tracking device 800 for the target object in the above-mentioned video, the training process of the predetermined weight includes: obtaining a current training feature map of a current frame of a training video and a previous training feature map of a previous frame; weighting the current training feature map based on the correlation between the current training feature map and the previous training feature map to obtain a training attention map; fusing the training attention map with the current training feature map after weighting it with an initial weight to obtain the final training feature map; obtaining a detection result of the target object based on the final training feature map; obtaining a tracking result of the target object based on the detection result of the target object; calculating a detection loss function value based on the detection result of the target object and a tracking loss function value based on the tracking result of the target object; and, updating the initial weight by gradient descent based on the detection loss function value and the tracking loss function value to obtain the predetermined weight.

[0180] Here, those skilled in the art can understand that the specific functions and operations of the various units and modules in the target object tracking device 800 in the above video have been described in detail above. Figure 6 The tracking method of the target object in the video has been described in detail, and therefore, its repeated description will be omitted.

[0181] Exemplary Device Four

[0182] Fig.13 The figure shows a block diagram of a video processing device according to an embodiment of the present application.

[0183] like Fig.13 As shown, the video processing device 900 according to an embodiment of the present application includes: an acquisition unit 910, used to acquire a current feature map of a current frame of a video and a previous feature map of a previous frame; an attention unit 920, used to weight the current feature map based on the correlation between the current feature map and the previous feature map to obtain an attention map; a processing unit 930, used to fuse the attention map with the current feature map to obtain a final feature map; and a detection unit 940, used to detect a foreground area from the current frame based on the final feature map.

[0184] In one example, the video processing device 900 further includes: a segmentation unit, configured to segment the detected foreground area from the current frame.

[0185] In one example, the video processing device 900 further includes: a superimposing unit, configured to superimpose the segmented foreground area with the background image.

[0186] In one example, in the above-mentioned video processing device 900, the acquisition unit 910 is used to: acquire the current frame and the previous frame of the video; obtain the previous feature map from the previous frame through a convolutional neural network; and obtain the current feature map from the current frame through the convolutional neural network.

[0187] In one example, in the above-mentioned video processing device 900, the attention unit 920 is used to: pass the current feature map through a first convolution layer to obtain a first feature map; pass the previous feature map through a second convolution layer to obtain a second feature map; and obtain an attention matrix for representing the correlation between the current feature map and the previous feature map based on the first feature map and the second feature map.

[0188] In one example, in the above-mentioned video processing device 900, the attention unit 920 is further used to: pass the current feature map through a third convolutional layer to obtain a third feature map; and weight the third feature map with the attention matrix to obtain the attention map.

[0189] In one example, in the video processing apparatus 900 , the previous frame is a frame before the current frame.

[0190] In one example, in the above-mentioned video processing device 900, the processing unit 930 is used to: weight the attention map with a predetermined weight and fuse it with the current feature map to obtain the final feature map.

[0191] In one example, in the above-mentioned video processing device 900, the predetermined weight is obtained through training.

[0192] In one example, in the above-mentioned video processing device 900, the training process of the predetermined weight includes: obtaining a current training feature map of a current frame of a training video and a previous training feature map of a previous frame; weighting the current training feature map based on the correlation between the current training feature map and the previous training feature map to obtain a training attention map; fusing the training attention map with the current training feature map after weighting it with an initial weight to obtain the final training feature map; calculating a loss function value based on the final training feature map; and, updating the initial weight by gradient descent based on the loss function value to obtain the predetermined weight.

[0193] Here, those skilled in the art can understand that the specific functions and operations of the various units and modules in the above-mentioned video processing device 900 have been described in detail above. Figure 7 The video processing method has been described in detail in the description of the video processing method, and therefore, its repeated description will be omitted.

[0194] Exemplary device five

[0195] Fig.14 A block diagram of an apparatus for searching for similar objects based on a video according to an embodiment of the present application is illustrated.

[0196] like Fig.14 As shown, according to an embodiment of the present application, the device 1000 for searching similar objects based on a video includes: an acquisition unit 1010, used to acquire a current feature map of a current frame of a video and a previous feature map of a previous frame; an attention unit 1020, used to weight the current feature map based on the correlation between the current feature map and the previous feature map to obtain an attention map; a processing unit 1030, used to fuse the attention map with the current feature map to obtain a final feature map; a detection unit 1040, used to detect a target object from the current frame based on the final feature map; and a search unit 1050, used to search for similar objects similar to the target object.

[0197] In one example, the apparatus 1000 for searching similar objects based on videos further includes: a link unit, configured to obtain an address link of the similar object; and connect to a network location where the similar object is located based on the address link.

[0198] In one example, in the above-mentioned device 1000 for searching similar objects based on video, the acquisition unit 1010 is used to: acquire the current frame and the previous frame of the video; obtain the previous feature map from the previous frame through a convolutional neural network; and obtain the current feature map from the current frame through the convolutional neural network.

[0199] In one example, in the above-mentioned device 1000 for searching similar objects based on video, the attention unit 1020 is used to: pass the current feature map through a first convolution layer to obtain a first feature map; pass the previous feature map through a second convolution layer to obtain a second feature map; and obtain an attention matrix for representing the correlation between the current feature map and the previous feature map based on the first feature map and the second feature map.

[0200] In one example, in the above-mentioned device 1000 for searching similar objects based on video, the attention unit 1020 is further used to: pass the current feature map through a third convolutional layer to obtain a third feature map; and weight the third feature map with the attention matrix to obtain the attention map.

[0201] In one example, in the apparatus 1000 for searching similar objects based on a video, the previous frame is a frame before the current frame.

[0202] In one example, in the above-mentioned apparatus 1000 for searching similar objects based on video, the processing unit 1030 is used to: weight the attention map with a predetermined weight and fuse it with the current feature map to obtain the final feature map.

[0203] In one example, in the apparatus 1000 for searching similar objects based on videos, the predetermined weight is obtained through training.

[0204] In one example, in the above-mentioned device 1000 for searching similar objects based on a video, the training process of the predetermined weight includes: obtaining a current training feature map of a current frame of a training video and a previous training feature map of a previous frame; weighting the current training feature map based on the correlation between the current training feature map and the previous training feature map to obtain a training attention map; weighting the training attention map with an initial weight and fusing it with the current training feature map to obtain the final training feature map; calculating a loss function value based on the final training feature map; and, updating the initial weight by gradient descent based on the loss function value to obtain the predetermined weight.

[0205] Here, those skilled in the art can understand that the specific functions and operations of the various units and modules in the apparatus 1000 for searching similar objects based on video have been described in the above reference. Fig. 9The method for searching similar objects based on a video has been described in detail, and therefore, its repeated description will be omitted.

[0206] Exemplary Electronic Devices

[0207] Below, reference Fig.15 To describe an electronic device according to an embodiment of the present application.

[0208] Fig.15 A block diagram of an electronic device according to an embodiment of the present application is illustrated.

[0209] like Fig.15 As shown, the electronic device 10 includes one or more processors 11 and a memory 12 .

[0210] The processor 11 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 10 to perform desired functions.

[0211] The memory 12 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 11 may run the program instructions to implement the method of detecting a target object from a video, the method of segmenting a target object in a video, the method of tracking a target object in a video, the video processing method, the method of searching for similar objects based on a video, and / or other desired functions of the various embodiments of the present application described above. Various contents such as current feature maps, previous feature maps, attention maps, etc. may also be stored in the computer-readable storage medium.

[0212] In one example, the electronic device 10 may further include: an input device 13 and an output device 14, and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0213] The input device 13 may include, for example, a keyboard, a mouse, etc.

[0214] The output device 14 can output various information to the outside, including the detection results of the target object in the video, etc. The output device 14 can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0215] Of course, to simplify, Fig.15 Only some of the components related to the present application in the electronic device 10 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device 10 may also include any other appropriate components.

[0216] Exemplary computer program products and computer-readable storage media

[0217] In addition to the above-mentioned methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of a method for detecting a target object from a video, a method for segmenting a target object in a video, a method for tracking a target object in a video, a video processing method, and a method for searching for similar objects based on a video according to various embodiments of the present application described in the above “Exemplary Methods” section of this specification.

[0218] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages, such as Java, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on the first user computing device, partially on the first user computing device, as an independent software package, partially on the first user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0219] In addition, an embodiment of the present application may also be a computer-readable storage medium having computer program instructions stored thereon, wherein when the computer program instructions are executed by a processor, the processor executes the steps described in the above “Exemplary Method” section of this specification, including a method for detecting a target object from a video, a method for segmenting a target object in a video, a method for tracking a target object in a video, a video processing method, and a method for searching for similar objects based on a video according to various embodiments of the present application.

[0220] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0221] The basic principles of the present application are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present application are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present application. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, not for limitation, and the above details do not limit the present application to being implemented by adopting the above specific details.

[0222] The block diagrams of the devices, apparatuses, equipment, and systems involved in this application are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagram. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including", "comprising", "having", etc. are open words, referring to "including but not limited to", and can be used interchangeably with them. The words "or" and "and" used here refer to the words "and / or" and can be used interchangeably with them, unless the context clearly indicates otherwise. The words "such as" used here refer to the phrase "such as but not limited to", and can be used interchangeably with them.

[0223] It should also be noted that in the apparatus, device and method of the present application, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present application.

[0224] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present application. Therefore, the present application is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0225] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present application to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A method for detecting a target object from a video, comprising: Obtain a current feature map of a current frame and a previous feature map of a previous frame of the video; Passing the current feature map through a first convolutional layer to obtain a first feature map; Passing the previous feature map through a second convolutional layer to obtain a second feature map; Obtaining an attention matrix for representing a correlation between the current feature map and the previous feature map based on the first feature map and the second feature map; Passing the current feature map through a third convolutional layer to obtain a third feature map; as well as Weighting the third feature map by the attention matrix to obtain an attention map; Fusing the attention map with the current feature map to obtain a final feature map; as well as A target object is detected from the current frame based on the final feature map.

2. The method for detecting a target object from a video according to claim 1, wherein: Obtaining the current feature map of the current frame and the previous feature map of the previous frame of the video includes: Obtaining a current frame and a previous frame of the video; Obtaining the previous feature map from the previous frame by a convolutional neural network; and The current feature map is obtained from the current frame through the convolutional neural network.

3. The method for detecting a target object from a video according to claim 1, wherein: The previous frame is a frame preceding the current frame.

4. The method for detecting a target object from a video according to claim 1, wherein: Fusing the attention map with the current feature map to obtain a final feature map includes: The attention map is weighted with a predetermined weight and fused with the current feature map to obtain the final feature map.

5. The method for detecting a target object from a video according to claim 4, wherein: The predetermined weight is obtained through training.

6. The method for detecting a target object from a video according to claim 5, wherein: The training process of the predetermined weights includes: Obtain a current training feature map of a current frame and a previous training feature map of a previous frame of a training video; weighting the current training feature map based on a correlation between the current training feature map and the previous training feature map to obtain a training attention map; The training attention map is weighted with the initial weight and then merged with the current training feature map to obtain a final training feature map; a loss function value is calculated based on the final training feature map; and The initial weight is updated by gradient descent based on the loss function value to obtain the predetermined weight.

7. The method for detecting a target object from a video according to claim 1, wherein: The detection result of the target object is further used for object tracking and / or object segmentation of the target object.

8. A method for segmenting a target object in a video, comprising: Obtain a current feature map of a current frame and a previous feature map of a previous frame of the video; Passing the current feature map through a first convolutional layer to obtain a first feature map; Passing the previous feature map through a second convolutional layer to obtain a second feature map; Obtaining an attention matrix for representing a correlation between the current feature map and the previous feature map based on the first feature map and the second feature map; Passing the current feature map through a third convolutional layer to obtain a third feature map; as well as Weighting the third feature map by the attention matrix to obtain an attention map; Fusing the attention map with the current feature map to obtain a final feature map; Detecting a target object from the current frame based on the final feature map; as well as A mask is generated based on the detection result of the target object to segment the target object.

9. A method for tracking a target object in a video, comprising: Obtain a current feature map of a current frame and a previous feature map of a previous frame of the video; Passing the current feature map through a first convolutional layer to obtain a first feature map; Passing the previous feature map through a second convolutional layer to obtain a second feature map; Obtaining an attention matrix for representing a correlation between the current feature map and the previous feature map based on the first feature map and the second feature map; Passing the current feature map through a third convolutional layer to obtain a third feature map; as well as Weighting the third feature map by the attention matrix to obtain an attention map; Fusing the attention map with the current feature map to obtain a final feature map; Detecting a target object from the current frame based on the final feature map; as well as The target object is tracked based on the detection result of the target object.

10. A video processing method, comprising: Obtain a current feature map of a current frame and a previous feature map of a previous frame of the video; Passing the current feature map through a first convolutional layer to obtain a first feature map; Passing the previous feature map through a second convolutional layer to obtain a second feature map; Obtaining an attention matrix for representing a correlation between the current feature map and the previous feature map based on the first feature map and the second feature map; Passing the current feature map through a third convolutional layer to obtain a third feature map; as well as Weighting the third feature map by the attention matrix to obtain an attention map; Fusing the attention map with the current feature map to obtain a final feature map; as well as A foreground region is detected from the current frame based on the final feature map.

11. The video processing method according to claim 10, further comprising: The detected foreground region is segmented from the current frame.

12. A method for searching similar objects based on a video, comprising: Obtain a current feature map of a current frame and a previous feature map of a previous frame of the video; Passing the current feature map through a first convolutional layer to obtain a first feature map; Passing the previous feature map through a second convolutional layer to obtain a second feature map; Obtaining an attention matrix for representing a correlation between the current feature map and the previous feature map based on the first feature map and the second feature map; Passing the current feature map through a third convolutional layer to obtain a third feature map; as well as Weighting the third feature map by the attention matrix to obtain an attention map; Fusing the attention map with the current feature map to obtain a final feature map; Detecting a target object from the current frame based on the final feature map; as well as Search for similar objects that are similar to the target object.

13. A device for detecting a target object from a video, comprising: An acquisition unit, used to acquire a current feature map of a current frame and a previous feature map of a previous frame of a video; an attention unit, configured to pass the current feature map through a first convolutional layer to obtain a first feature map; Passing the previous feature map through a second convolutional layer to obtain a second feature map; Obtaining an attention matrix for representing a correlation between the current feature map and the previous feature map based on the first feature map and the second feature map; Passing the current feature map through a third convolutional layer to obtain a third feature map; and weighting the third feature map with the attention matrix to obtain an attention map; A processing unit, configured to fuse the attention map with the current feature map to obtain a final feature map; as well as A detection unit is used to detect a target object from the current frame based on the final feature map.

14. A device for segmenting a target object in a video, comprising: An acquisition unit, used to acquire a current feature map of a current frame and a previous feature map of a previous frame of a video; an attention unit, configured to pass the current feature map through a first convolutional layer to obtain a first feature map; Passing the previous feature map through a second convolutional layer to obtain a second feature map; Obtaining an attention matrix for representing a correlation between the current feature map and the previous feature map based on the first feature map and the second feature map; Passing the current feature map through a third convolutional layer to obtain a third feature map; and weighting the third feature map with the attention matrix to obtain an attention map; A processing unit, configured to fuse the attention map with the current feature map to obtain a final feature map; a detection unit, configured to detect a target object from the current frame based on the final feature map; as well as The segmentation unit is used to generate a mask based on the detection result of the target object to segment the target object.

15. A device for tracking a target object in a video, comprising: An acquisition unit, used to acquire a current feature map of a current frame and a previous feature map of a previous frame of a video; an attention unit, configured to pass the current feature map through a first convolutional layer to obtain a first feature map; Passing the previous feature map through a second convolutional layer to obtain a second feature map; Obtaining an attention matrix for representing a correlation between the current feature map and the previous feature map based on the first feature map and the second feature map; Passing the current feature map through a third convolutional layer to obtain a third feature map; and weighting the third feature map with the attention matrix to obtain an attention map; A processing unit, configured to fuse the attention map with the current feature map to obtain a final feature map; a detection unit, configured to detect a target object from the current frame based on the final feature map; as well as A tracking unit is used to track the target object based on the detection result of the target object.

16. A video processing device, comprising: An acquisition unit, used to acquire a current feature map of a current frame and a previous feature map of a previous frame of a video; an attention unit, configured to pass the current feature map through a first convolutional layer to obtain a first feature map; Passing the previous feature map through a second convolutional layer to obtain a second feature map; Obtaining an attention matrix for representing a correlation between the current feature map and the previous feature map based on the first feature map and the second feature map; Passing the current feature map through a third convolutional layer to obtain a third feature map; and weighting the third feature map with the attention matrix to obtain an attention map; A processing unit, configured to fuse the attention map with the current feature map to obtain a final feature map; as well as A detection unit is used to detect a foreground area from the current frame based on the final feature map.

17. A device for searching similar objects based on a video, comprising: An acquisition unit, used to acquire a current feature map of a current frame and a previous feature map of a previous frame of a video; an attention unit, configured to pass the current feature map through a first convolutional layer to obtain a first feature map; Passing the previous feature map through a second convolutional layer to obtain a second feature map; Obtaining an attention matrix for representing a correlation between the current feature map and the previous feature map based on the first feature map and the second feature map; Passing the current feature map through a third convolutional layer to obtain a third feature map; and weighting the third feature map with the attention matrix to obtain an attention map; A processing unit, configured to fuse the attention map with the current feature map to obtain a final feature map; a detection unit, configured to detect a target object from the current frame based on the final feature map; as well as The searching unit is used to search for similar objects similar to the target object.

18. An electronic device, comprising: processor; as well as A memory, in which computer program instructions are stored, and when the computer program instructions are executed by the processor, the processor executes the method for detecting a target object from a video as described in any one of claims 1 to 7, the method for segmenting a target object in a video as described in claim 8, the method for tracking a target object in a video as described in claim 9, the video processing method as described in claim 10 or 11, and the method for searching for similar objects based on a video as described in claim 12.

Images