Instance segmentation method, related device and storage medium
By using time-domain shift processing of time-domain memory features in the video instance segmentation model, information interaction between video image frames is achieved, and the problem of insufficient accuracy of instance segmentation in the prior art is solved, and the accuracy of video instance segmentation is improved sex.
Patent Information
- Application Number
- CN202210335171.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-31
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2042-03-31
AI Technical Summary
The existing video instance segmentation model is limited by the locality of the convolution operation when extracting features, resulting in inaccuracy of the instance segmentation results.
By acquiring the time domain memory features of the video image and performing time domain shift processing, information interaction between frames is realized, and instance segmentation is performed in combination with image features to improve the accuracy of segmentation.
This improves the accuracy of instance segmentation, avoids the inaccuracy problem caused by local feature extraction by traditional models, and achieves more accurate video instance segmentation.
Smart Images

Figure CN114782861B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing technology, and in particular to an instance segmentation method, related equipment and storage medium. Background Art
[0002] Instance segmentation is a crucial preprocessing for image recognition and computer vision, and is widely used in various fields. For example, instance segmentation can be used for tasks such as object recognition, object detection, and object search. At present, when performing video instance segmentation, the video instance segmentation model is mainly based on the traditional convolutional neural network. When extracting features using traditional convolutional neural networks, due to the locality of the convolution operation, only local features can be extracted, and then instance segmentation is achieved based on local features, which will lead to inaccuracy in the instance segmentation results to a certain extent. Summary of the invention
[0003] The embodiments of the present application provide an instance segmentation method, related devices, and storage media, which can improve the accuracy of video instance segmentation.
[0004] On the one hand, an embodiment of the present application provides an instance segmentation method, including:
[0005] Obtain N frames of video images in the target video, each frame of the video image includes one or more instances, and N is an integer greater than 1;
[0006] Obtaining image features of each frame of video image and time domain memory features corresponding to each frame of video image, where the time domain memory features are used to indicate memory image features of the video image in the time domain;
[0007] Performing time-domain shift processing on the time-domain memory feature corresponding to each frame of video image to obtain the memory shift feature corresponding to each frame of video image;
[0008] Based on the image features of each frame of video image and the corresponding memory shift features, instance segmentation is performed on N frames of video image to obtain the instances included in each frame of video image.
[0009] On the one hand, an embodiment of the present application provides an instance segmentation device, including:
[0010] An acquisition unit, used to acquire N frames of video images in a target video, each frame of the video image includes one or more instances, and N is an integer greater than 1;
[0011] The acquisition unit is further used to acquire the image features of each frame of the video image and the time domain memory features corresponding to each frame of the video image, where the time domain memory features are used to indicate the memory image features of the video image in the time domain;
[0012] A processing unit, used for performing time domain shift processing on the time domain memory feature corresponding to each frame of video image to obtain the memory shift feature corresponding to each frame of video image;
[0013] The processing unit is also used to perform instance segmentation on the N frames of video images based on the image features of each frame of video images and the corresponding memory shift features to obtain the instances included in each frame of video images.
[0014] On the one hand, an embodiment of the present application provides a computer device, which includes an input device, an output device, a processor and a computer storage medium, wherein the processor and the computer storage medium are interconnected, wherein the computer storage medium is used to store a computer program, and the processor is configured to call the computer program to execute the above-mentioned instance segmentation method.
[0015] On the one hand, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, it is used to implement the above-mentioned instance segmentation method.
[0016] On the one hand, an embodiment of the present application provides a computer program product, which includes a computer program; the computer program is stored in a computer-readable storage medium, and when the computer program is executed by a processor of a computer device, the above-mentioned instance segmentation method is executed.
[0017] In an embodiment of the present application, after acquiring N frames of video images in a target video, the image features of each frame of video images and the time-domain memory features corresponding to each frame of video images can be acquired, and the time-domain memory features are used to indicate the memory image features of the video images in the time domain; the time-domain memory features corresponding to each frame of video images are subjected to time-domain shift processing to obtain the memory shift features corresponding to each frame of video images, and based on the image features of each frame of video images and the corresponding memory shift features, the N frames of video images are subjected to instance segmentation to obtain the instances included in each frame of video images. By performing time-domain shift processing on the time-domain memory features of each frame of video images, information interaction of the time-domain memory features in each frame of video images can be achieved, thereby achieving information interaction between frames in the time domain, so that the instances included in each frame of video images can be segmented more accurately according to the memory shift features and image features corresponding to each frame of video images, thereby improving the accuracy of instance segmentation. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0019] Figure 1a It is an architecture diagram of an instance segmentation model provided in an embodiment of the present application;
[0020] Figure 1b is an architecture diagram of an instance segmentation system provided in an embodiment of the present application;
[0021] Figure 2 is a flow chart of an instance segmentation method provided in an embodiment of the present application;
[0022] Figure 3a It is a structural diagram of a frame-level time-domain feature fusion module based on memory shift in an instance segmentation model provided in an embodiment of the present application;
[0023] Figure 3b It is a schematic diagram of a flow chart of a time domain shift of a time domain memory feature provided in an embodiment of the present application;
[0024] Figure 3c is an architecture diagram of another instance segmentation model provided in an embodiment of the present application;
[0025] Figure 4 is a flowchart of another instance segmentation method provided in an embodiment of the present application;
[0026] Figure 5a is an architecture diagram of another instance segmentation model provided in an embodiment of the present application;
[0027] Figure 5b It is a schematic diagram of a process of reverse time domain shifting of a memory shift feature provided in an embodiment of the present application;
[0028] Figure 6 It is a flowchart of a model training method provided in an embodiment of the present application;
[0029] Figure 7 is a structural schematic diagram of an instance segmentation device provided in an embodiment of the present application;
[0030] Figure 8 It is a structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0031] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0032] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.
[0033] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.
[0034] Computer vision technology (CV) is a science that studies how to make machines "see". To put it more specifically, it refers to using cameras and computers to replace human eyes to identify, search and measure targets, and further perform graphic processing to make computer processing into images that are more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and map construction, and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.
[0035] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.
[0036] The instance segmentation scheme provided in the embodiments of the present application involves computer vision technology of artificial intelligence. Video instance segmentation aims to simultaneously detect, segment, and search for regions of interest (RoI) or instances of interest in a video, wherein video instance segmentation is a natural extension of instance segmentation in the video field. Compared with instance segmentation, video instance segmentation requires temporal search and pursuit of video instances in the time domain (or time domain), so it can make full use of the time domain information brought by the video modality (i.e., presented in the form of video) to perform instance segmentation on the instances in the video, thereby improving the accuracy of video instance segmentation. The general principle of the instance segmentation scheme is as follows: in order to fully interact the information between frames (such as image features), specifically, when performing instance segmentation on instances in the target video, make full use of the rich information in the time domain, first obtain the target video, and obtain N frames of video images from the target video, wherein each frame of the video image may include one or more instances, which may be people, animals, objects, etc.; each frame of the video image may correspond to a time domain information, for example, the time domain information of the target video is 15 seconds of information, and the time domain information corresponding to each frame of the video image may be the information of the 1st second, the information of the 4th second, and so on.
[0037] Then, the image features of each frame of the N frames of video images can be extracted, and the time-domain memory features corresponding to each frame of the video images can be obtained; the time-domain memory features can be used to indicate the memory image features of the video images in the time domain, that is, it can be understood that: the time-domain memory features can include part or all of the image features of the video images. Then, the time-domain memory features corresponding to each frame of the video images are time-domain shifted in the time series by using the memory shift mechanism, so as to capture the information of each frame of the video images in the time domain, and obtain the shifted memory features corresponding to each frame of the video images. Among them, the information capture here can be understood as: since the time-domain memory features are used to indicate the memory image features of the video images in the time domain, the obtained shifted memory features can include the image features of other video images in the time domain in addition to the image features of the video images themselves. Further, the computer device can perform instance segmentation on the N frames of video images based on the shifted memory features and image features corresponding to each frame of the video images, and obtain the instances included in each frame of the video images. By shifting the time series, the information flow of the time domain memory features corresponding to each frame of video image is realized in the time domain, and effective information flow is constructed in the time domain, which improves the accuracy of instance segmentation.
[0038] Among them, based on the shift memory features and image features corresponding to each frame of video images, N frames of video images are instance segmented to obtain the instances included in each frame of video images. The specific implementation method can be: based on the shift memory features and image features corresponding to each frame of video images, feature fusion is performed to obtain the image fusion features of each frame of video images, and based on the image fusion features of each frame of video images, each frame of video images is instance segmented. The image fusion features include image features in other video images. Combined with the image features in other video images, the video image can be more accurately instance segmented.
[0039] In one embodiment, the embodiment of the present application can not only complete the frame-level image feature fusion processing in the time domain, but also use the spatiotemporal query interaction mechanism to realize the construction of instance-level video information synthesis. When performing instance segmentation on the video, M instance query vectors are introduced. The M instance query vectors can be responsible for predicting the corresponding video instance in each frame of the input video image. The M instance query vectors can be used together with the shift memory features and image features corresponding to each frame of the video image to predict and segment the instances in the N frames of the video image, and finally the instances included in each frame of the video image can be obtained, where M can be an integer greater than or equal to 1. As an implementation method, these M instance query vectors will be broadcast to each frame of the video image, and then based on the image features of each frame of the video image, the corresponding shift memory features and the corresponding M instance query vectors, the N frames of the video image are instance segmented to obtain the instances included in each frame of the video image. By introducing M instance query vectors, the M instance query vectors can be responsible for predicting the corresponding video instance in each frame of the input video image, thereby constructing a completely end-to-end instance segmentation solution, avoiding the use of post-processing operations such as non-maximum suppression in the case of "one-to-many" matching, greatly simplifying the complexity of video instance segmentation, and improving the efficiency of instance segmentation. The instance query method maintains a competitive reasoning speed. Among them, end-to-end means that the entire process is a unified process and does not include additional post-processing modules.
[0040] In one embodiment, the instance segmentation scheme provided in the embodiment of the present application involves artificial intelligence machine learning technology. The above instance segmentation scheme can be implemented by a time-domain efficient instance segmentation model (or instance segmentation algorithm or Tevit), which can be as follows Figure 1aAs shown, the instance segmentation model may include a backbone network (also understood as a backbone feature extraction network Backbone) and a network prediction head. The instance segmentation model uses networks such as PVT-B1 as the basic network, and introduces a frame-level time-domain feature fusion module based on memory shift in the basic network to construct the backbone network in the instance segmentation model; and introduces a spatiotemporal query module in the network prediction head. The frame-level time-domain feature fusion module based on memory shift can be used to shift the time-domain memory features corresponding to each frame of video image in the time series to capture information in the time domain, obtain the shifted memory features corresponding to each frame of video image, and perform feature fusion processing on the image features of each frame of video image and the corresponding shifted memory features to obtain the image fusion features of each frame of video image.
[0041] The spatiotemporal interactive query module can be used to perform feature fusion analysis on the M instance query vectors corresponding to each frame of video image, and obtain M fused instance query vectors corresponding to each frame of video image. The feature fusion analysis here includes intra-frame feature fusion and inter-frame feature fusion, which makes full use of the parameter sharing of the query in the time dimension (i.e., intra-frame feature fusion) and the spatial dimension (i.e., inter-frame feature fusion), and efficiently completes the instance query in the time dimension and the spatial dimension. Then, the image fusion features of each frame of video image and the corresponding M fused instance query vectors are used to perform instance segmentation on the N frames of video images. Through this instance segmentation model, time domain information can be efficiently utilized to complete the frame-level image feature fusion processing in the time domain and construct instance-level video information synthesis, realizing an instance query-based, fully end-to-end instance segmentation model without the need to use post-processing operations such as non-maximum suppression, which greatly simplifies the complexity of the instance segmentation model.
[0042] Among them, the network prediction head can also be configured with instance tasks corresponding to instance segmentation. After being processed by the frame-level temporal feature fusion module based on memory shift and the query module of spatiotemporal interaction, the instance foreground mask in each frame of video image can be obtained, and then the instance foreground mask in each frame of video image can be processed by the network prediction head to achieve the corresponding instance task, for example, the instance foreground mask, instance bounding box, instance category, etc. of each frame of video image can be output by the network prediction head.
[0043] The above instance segmentation scheme can effectively perform various video understanding tasks such as video editing and scene analysis. Taking video editing as an example, through the instance segmentation scheme, the target instance in the video can be obtained, and the video image corresponding to the target instance can be clipped to generate a video clip about the target instance.
[0044] The instance segmentation scheme provided by the embodiment of the present application has the following beneficial effects: (1) By shifting the time series, the information flow of the time domain memory features corresponding to each frame of the video image in the time domain is realized, and an effective information flow is constructed in the time domain, thereby improving the accuracy of instance segmentation. (2) By introducing M instance query vectors, the M instance query vectors can be responsible for predicting the corresponding video instance in each input video frame, thereby constructing a post-processing-free, fully end-to-end instance query scheme, avoiding the use of post-processing operations such as non-maximum suppression in the case of "one-to-many" matching, greatly simplifying the complexity of video instance segmentation, and improving the efficiency of video instance segmentation.
[0045] Based on the above instance segmentation solution, this application provides an instance segmentation system, see Figure 1b , the instance segmentation system may include a terminal device 101 and a server 102, wherein the terminal device 101 and the server 102 may be directly or indirectly connected via wired or wireless communication, and this application does not limit this. The terminal devices involved in this application may be smart phones, tablet computers, laptops, PDAs, personal computers, smart TVs, smart watches, vehicle-mounted devices, wearable devices, etc., but are not limited thereto. The server may be an independent physical server, or a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. The number of terminal devices 101 and servers 102 is also not limited.
[0046] In one embodiment, the above instance segmentation scheme can be executed by the above terminal device 101. The terminal device 101 can provide a video selection interface, which can include multiple videos. The user can select any one of the multiple videos (i.e., the target video). Then the terminal device 101 can send a target video acquisition request to the server 102. The server 102 can return the target video to the terminal device based on the target video acquisition request. Then the terminal device 101 can perform instance segmentation on the N frames of video images in the target video based on the above instance segmentation scheme to obtain the instances included in each frame of the video image. The instances included in each frame of the video image can be saved to the server 102.
[0047] In another embodiment, the above-mentioned instance segmentation scheme can be executed by the above-mentioned server 102. The user can send an instance segmentation request for the target video to the server 102 through the terminal device 101. The server 102 can directly obtain the target video based on the instance segmentation request, and use the above-mentioned instance segmentation scheme to perform instance segmentation on the N frames of video images in the target video to obtain the instances included in each frame of the video image, and save the instances included in each frame of the video image. At the same time, the server 102 can display the instances included in each frame of the video image to the user through the terminal device 101.
[0048] Based on the above instance segmentation scheme, please refer to Figure 2 , Figure 2 A flowchart of an instance segmentation method provided in an embodiment of the present application. The method can be executed by a computer device, which can be the terminal device 101 or the server 102 in the above instance segmentation system. The instance segmentation method described in this embodiment can include the following steps S201-S204:
[0049] S201. Obtain N frames of video images in a target video, each frame of video image includes one or more instances, and N is an integer greater than 1. The target video may be a video pre-recorded by a user (i.e., an offline video), or the target video may be a real-time video (or an online video), for example, the target video may be a live video. The instances included in the target video (or the instances included in each frame of video image) may be animals, tasks, objects, etc. in the video, and this application does not limit this.
[0050] In a specific implementation, when a user wants to perform instance segmentation on a target video, the target video can be uploaded to a computer device; accordingly, the computer device can obtain the target video; or, when the video is broadcast live, the computer device can directly obtain the live video as the target video.
[0051] Among them, the N frames of video images may be part of the video images in the target video, or may be all the video images in the target video. In the actual process, the N frames of video images in the target video may be obtained according to the reasoning strategy. Among them, for online videos and offline videos, the way to obtain N frames of video images in the target video according to the reasoning strategy may be different; in one embodiment, when the target video is an offline video, the specific implementation method of obtaining N frames of video images in the target video according to the reasoning strategy may be: dividing the target video into multiple video sub-segments, and obtaining the target video sub-segments from the multiple video sub-segments, and the N frames of video images are all the video images in the target video sub-segments. In another embodiment, when the target video is an online video, the specific implementation method of obtaining N frames of video images in the target video according to the reasoning strategy may be: obtaining all video images in the target video as N frames of video images. It should be noted that the reasoning strategy can be set according to demand, and this application is not limited. In some embodiments, the specific implementation method of obtaining N frames of video images in the target video may also be: randomly sampling discrete N frames of video images from the target video.
[0052] S202, obtaining image features of each frame of video image and time domain memory features corresponding to each frame of video image, wherein the time domain memory features are used to indicate memory image features of the video image in the time domain. It can be understood that: the memory image features may include part or all of the image features of the video image. The image features of each frame of video image are feature maps of the entire video image, rather than features of instances in the video image.
[0053] In one embodiment, the computer device may first perform feature extraction on each frame of video image to obtain the initial image features of each frame of video image, and obtain the initial time-domain memory features corresponding to each frame of video image, wherein the initial image features are feature maps of the entire video image, the initial time-domain memory features may be initialized in advance, the number of initial time-domain memory features corresponding to each frame of video image may be one or more, one or more initial time-domain memory features may be the same, or one or more initial time-domain memory features may be different. The computer device may obtain the image features of each frame of video image and the time-domain memory features corresponding to each frame of video image based on the initial image features of each frame of video image and the corresponding initial time-domain memory features.
[0054] Among them, based on the initial image features of each frame of video image and the corresponding initial time-domain memory features, the specific implementation method of obtaining the image features of each frame of video image and the time-domain memory features corresponding to each frame of video image can be: feature fusion of the initial image features of each frame of video image and the corresponding initial time-domain memory features to obtain the intermediate image features of each frame of video image and the corresponding intermediate time-domain memory features, and spatial transformation of the intermediate image features of each frame of video image and the corresponding intermediate time-domain memory features, and finally the image features of each frame of video image and the time-domain memory features corresponding to each frame of video image can be obtained. Among them, the feature fusion mentioned above refers to: the initial image features of each frame of video image and the corresponding initial time-domain memory features are exchanged in the time domain, so that the initial time-domain memory features corresponding to each frame of video image can notice the initial image features of the frame of video image, thereby obtaining the time-domain memory features containing part or all of the initial image features.
[0055] In one embodiment, the instance segmentation model can be called to perform feature extraction on each frame of video image, obtain the initial image features of each frame of video image, and obtain the initial time-domain memory features corresponding to each frame of video image. The initial time-domain memory features can be the network parameters of the frame-level time-domain feature fusion module based on memory shift in the above-mentioned instance segmentation model. In some embodiments, the initial time-domain memory features corresponding to each frame of video image can be inserted into the frame-level time-domain feature fusion module based on memory shift in the instance segmentation model in the form of a plug-in and randomly initialized. The frame-level time-domain feature fusion module based on memory shift in the instance segmentation model can be as follows Figure 3a As shown, the frame-level temporal feature fusion module 301 based on memory shift may include a first multi-head attention module, a first forward network, a memory shift module, a second multi-head attention module, and a second forward network; the memory shift module can be used to perform temporal shift processing on the temporal memory features corresponding to each frame of video image. The first multi-head attention module and the second multi-head attention module can perform feature extraction within the range of the entire feature map. Under a wide range of perception, the multi-head attention module usually obtains better model effects than the traditional convolutional network. The first forward network and the second forward network can be used for spatial transformation. The input of the frame-level temporal feature fusion module 301 based on memory shift can be the initial image features of each frame of video image (i.e., the initial image features extracted by the initial feature extraction module in the instance segmentation model), and the output of the frame-level temporal feature fusion module 301 based on memory shift can be the image aggregation features of each frame of video image.
[0056] The specific implementation method of obtaining the image features of each frame of video image and the time domain memory features corresponding to each frame of video image can be: calling the first multi-head attention module in the frame-level time domain feature fusion module based on memory shift to perform feature fusion on the initial image features of each frame of video image and the corresponding initial time domain memory features, so as to obtain the intermediate image features of each frame of video image and the corresponding intermediate time domain memory features, and then calling the first forward network in the frame-level time domain feature fusion module based on memory shift to perform spatial transformation on the intermediate image features of each frame of video image and the corresponding intermediate time domain memory features, so as to obtain the image features of each frame of video image and the time domain memory features corresponding to each frame of video image. The first multi-head attention module can be MHSA (Multi-head Self Attention), which is a neural network module for sequence processing, feature aggregation or feature fusion, and the first forward network can be FFN (Feed Forward Network), a fully connected network for feature processing after the multi-head attention module, which is inserted after the multi-head attention module, and the main goal is to perform spatial transformation on image features. After the initial image features are processed by the first multi-head attention module and the first forward network, the temporal memory features corresponding to each frame of the video image can successfully capture the image features of the video image, such as Figure 3a The time-domain memory feature 31 captures the image features of the first frame of video image, that is, the time-domain memory feature 31 includes the image features of the first frame of video image; the time-domain memory feature 32 captures the image features of the second frame of video image, that is, the time-domain memory feature 32 includes the image features of the second frame of video image; the time-domain memory feature 33 captures the image features of the third frame of video image, that is, the time-domain memory feature 33 includes the image features of the third frame of video image.
[0057] S203, performing time-domain shift processing on the time-domain memory features corresponding to each frame of video image to obtain memory shift features corresponding to each frame of video image.
[0058] As can be seen from step S202, the time domain memory feature is used to indicate the memory image feature of the video image in the time domain, and the memory image feature may include part or all of the image features. In order to make full use of the rich information of each frame of video image in the time domain, the embodiment of the present application can perform time domain shift processing on the time domain memory feature corresponding to each frame of video image in the time domain to obtain the memory shift feature corresponding to each frame of video image. Through the shift in the time dimension, the information interaction between each frame of video image and each frame of video image in the time domain is realized, so that when each frame of video image is subsequently segmented as an instance, the image features of other video images can be used to segment the video image as an instance, thereby improving the accuracy of instance segmentation.
[0059] In one embodiment, the specific implementation of step S203 may be: divide the temporal memory features corresponding to N frames of video images into multiple memory feature groups, each memory feature group includes the same number of temporal memory features, wherein each memory feature group may also be referred to as a memory module. For example, there are 32 temporal memory features corresponding to N frames of video images in total, and the 32 temporal memory features may be divided into 4 memory feature groups, each memory feature group includes 8 temporal memory features; then, obtain the moving step and moving direction of each memory feature group, and move the temporal memory features in each memory feature group according to the moving step and moving direction of each memory feature group, so as to obtain the memory shift feature corresponding to each frame of video images. Among them, the shift step and moving direction may be set according to the demand, for example, the shift step may be set to 1, 2, 3, etc., and the moving direction is determined based on the time series constructed by N frames of video images, for example, the time series is constructed from top to bottom based on N frames of video images, and the moving direction may be upward or downward movement; for another example, the time series is constructed from left to right based on N frames of video images, and the moving direction may be leftward movement or rightward movement.
[0060] For example, in Figure 3b In the figure, the black cubes represent the temporal memory features, and the white cubes represent the image features of each frame of the video image ( Figure 3b 5 frames are taken as an example), and from top to bottom, the time series constructed by each frame of video image (i.e. Figure 3b T in the figure represents the time domain information in the time series, and G represents the memory feature group), which represents four memory feature groups from left to right. The first memory feature group: the moving step is 1 (i.e., S=1), and the moving direction is upward; the second memory feature group: the moving step is 1 (i.e., S=1), and the moving direction is downward; the third memory feature group: the moving step is 2 (i.e., S=2), and the moving direction is upward; the fourth memory feature group has a moving step of 2 (i.e., S=2), and the moving direction is downward. Then, the computer device moves the time domain memory features in each memory feature group according to the moving direction and moving step of each memory feature group along the time series constructed by N frames of video images, and obtains the memory shift features corresponding to each frame of video images. The schematic diagram of the time domain shift processing of the time domain memory features of each memory feature group is shown in the figure. Figure 3b As shown, the information interaction between the time domain memory features corresponding to each frame of video image in the time domain is realized.
[0061] Among them, in the embodiment of the present application, the initial time-domain memory features corresponding to N frames of video images can be directly divided into multiple initial memory feature groups, each initial memory feature group includes the same number of initial time-domain memory features, and then the first multi-head attention module and the first forward network in step S202 are called to process the image features of each frame of video image and multiple initial memory feature groups, and the above-mentioned multiple memory feature groups can also be obtained, each memory feature group includes the same number of time-domain memory features. At this time, the number of memory feature groups is the same as the number of initial memory feature groups, and the number of time-domain memory features included in the memory feature group is the same as the number of initial time-domain memory features included in the initial memory feature group.
[0062] In one embodiment, the memory shift module can be called to perform time domain shift processing on the time domain memory features corresponding to each frame of video images to obtain the memory shift features corresponding to each frame of video images. Through the time domain shift processing, the time domain memory features corresponding to any frame of video images in N frames can see the time domain memory features corresponding to the remaining video images. Figure 3a In step S202, the temporal memory feature 31 corresponding to the first frame of video image, the temporal memory feature 32 corresponding to the second frame of video image, and the temporal memory feature 33 corresponding to the third frame of video image can be obtained, and then the temporal memory feature 34 corresponding to the first frame of video image, the memory shift feature 35 corresponding to the second frame of video image, and the memory shift feature 36 corresponding to the third frame of video image can be obtained by calling the memory shift module to perform temporal shift features on the temporal memory features corresponding to the three frames of video image. The memory shift feature 34 corresponding to the first frame of video image can include the temporal memory feature 32 corresponding to the second frame of video image and the temporal memory feature 33 corresponding to the third frame of video image; the memory shift feature 35 corresponding to the second frame of video image can include the temporal memory feature 31 corresponding to the first frame of video image and the temporal memory feature 33 corresponding to the third frame of video image; the memory shift feature 36 corresponding to the third frame of video image can include the temporal memory feature 32 corresponding to the second frame of video image and the temporal memory feature 31 corresponding to the first frame of video image. It should be understood that after the time domain shift processing, the time domain memory features corresponding to each frame of video image can focus on the time domain memory features corresponding to one frame or two frames before and after or even more frames of video images in the time domain, and finally obtain the memory shift features corresponding to each frame of video image.
[0063] S204: Based on the image features of each frame of video image and the corresponding memory shift features, instance segmentation is performed on the N frames of video image to obtain instances included in each frame of video image.
[0064] In one embodiment, the computer device may first perform feature fusion based on the image features of each frame of video image and the corresponding memory shift features to obtain the image fusion features of each frame of video image. At this time, the image fusion features of any frame of video image may include information about some image features related to other video images. Then, based on the image fusion features of each frame of video image, N frames of video images are instance segmented to obtain the instances included in each frame of video image, and finally the instances in the target video can be obtained. By performing instance segmentation through image fusion features, each frame of video image can be instance segmented more accurately.
[0065] In one embodiment, the computer device may call the second multi-head attention module to perform feature fusion on the image features of each frame of video image and the corresponding memory shift features to obtain the intermediate image fusion features of each frame of video image, and call the second forward network to perform spatial transformation on the intermediate image fusion features of each frame of video image to obtain the image fusion features of each frame of video image. The second multi-head attention module may be MHSA, and the second forward network may be FFN. Figure 3a The image fusion features 37, 38 and 39 in the image fusion feature 37, 38 and 39 all include the image features of the other two frames of video images in addition to their own image features. The multi-head attention mechanism of the first multi-head attention module and the second multi-head attention module can perform feature extraction within the range of the entire feature map. In a wide range of perception, the use of the multi-head attention module usually obtains better model effects than the traditional convolutional network.
[0066] In one embodiment, based on the image fusion features of each frame of video images, N frames of video images are instance segmented to obtain the specific implementation method of the instances included in each frame of video images can be: obtain M instance query vectors, and the M instance query vectors are responsible for predicting the instances included in each frame of video images in the N frames of video images. Then, based on the image fusion features of each frame of video images and the corresponding M instance query vectors, the N frames of video images are instance segmented to obtain the instances included in each frame of video images. Among them, the M instance vectors can be pre-set according to needs. By introducing instance query vectors, the M instance query vectors are responsible for predicting the instances included in each frame of video images in N frames of video images, so as to achieve end-to-end instance segmentation, accelerate the instance reasoning speed, and further improve the efficiency of instance segmentation.
[0067] In one embodiment, Figure 3c , the instance segmentation model may further include: a spatiotemporal interactive query module 302; the spatiotemporal interactive query module 302 may include a cascade of multiple multi-head attention modules (i.e. Figure 3cThe cascaded multi-head attention modules can be used to implement instance query at the instance level for each frame of video image to determine the fused instance query vector corresponding to each frame of video image. The input of the spatiotemporal interactive query module 302 can be M instance query vectors. Figure 3c In the example, the spatiotemporal interactive query module 302 includes a plurality of third multi-head attention modules and a fourth multi-head attention module, and the module parameters of the plurality of third multi-head attention modules and the fourth multi-head attention module are the same. It can be understood that the module parameters of the various multi-head attention modules cascaded in the spatiotemporal interactive query module are the same, and the frame-level temporal feature fusion module based on memory shift and the spatiotemporal interactive query module in the instance segmentation model are subsequently called to perform efficient temporal information extraction at both the feature and instance levels.
[0068] Among them, based on the image fusion features of each frame of video image and the corresponding M instance query vectors, N frames of video images are instance segmented to obtain the instances included in each frame of video image: the M instance query vectors are broadcasted to the N frames of video image to obtain the M instance query vectors corresponding to each frame of video image, and the third multi-head attention module and the fourth multi-head attention module are respectively called to perform intra-frame and inter-frame feature fusion on the M instance query vectors corresponding to each frame of video image, and finally the M fused instance query vectors corresponding to each frame of video image are obtained, and dynamic convolution is performed based on the image fusion features of each frame of video image and the corresponding M fused instance query vectors to obtain the instances included in each frame of video image. Figure 3c As shown, after the image fusion features of each frame of video image are obtained by the frame-level time domain feature fusion module based on memory shift, the image fusion features of each frame of video image are input into the spatiotemporal interactive query module 302. The spatiotemporal interactive query module 302 is called to dynamically convolve the image fusion features of each frame of video image with the corresponding M fusion instance query vectors, and finally the instances included in the three frames of video images can be obtained, that is, the instance included in the first frame of video image is Figure 3c The person and skateboard are framed by the dotted line in the middle frame, and the second frame of the video image includes an example of Figure 3c The third frame of video image includes the person and skateboard framed by the dotted line in FIG. 3c .
[0069] In an embodiment of the present application, after acquiring N frames of video images in a target video, the image features of each frame of video images and the time-domain memory features corresponding to each frame of video images can be acquired, and the time-domain memory features are used to indicate the memory image features of the video images in the time domain; the time-domain memory features corresponding to each frame of video images are subjected to time-domain shift processing to obtain the memory shift features corresponding to each frame of video images; based on the image features of each frame of video images and the corresponding memory shift features, the N frames of video images are subjected to instance segmentation to obtain the instances included in each frame of video images. By performing time-domain shift processing on the time-domain memory features of each frame of video images, information interaction between frames in the time domain is achieved, so that the instances included in each frame of video images can be segmented more accurately according to the memory shift features and image features corresponding to each frame of video images, thereby improving the accuracy of instance segmentation.
[0070] Based on the above instance segmentation scheme, please refer to Figure 4 , Figure 4 A flowchart of another instance segmentation method provided in an embodiment of the present application. The instance segmentation method may be executed by a computer device, which may be the terminal device 101 or the server 102. The instance segmentation method described in this embodiment may include the following steps S401-S407:
[0071] S401. Obtain N frames of video images in a target video, where each frame of video image includes one or more instances, and N is an integer greater than 1.
[0072] S402, obtaining image features of each frame of video image and time-domain memory features corresponding to each frame of video image, where the time-domain memory features are used to indicate memory image features of the video image in the time domain.
[0073] S403, performing time-domain shift processing on the time-domain memory feature corresponding to each frame of video image to obtain a memory shift feature corresponding to each frame of video image.
[0074] It should be noted that the specific implementation of steps S401-S403 can refer to the specific implementation of steps S201-S203 above, which will not be repeated here.
[0075] S404: Fusing the image features of each frame of video image and the corresponding memory shift features to obtain image fusion features of each frame of video image. The number of image fusion features of each frame of video image may be L, where L may be an integer greater than or equal to 1.
[0076] In one embodiment, the computer device may perform feature fusion on the image features of each frame of video image and the corresponding memory shift features to obtain the initial image fusion features of each frame of video image, and perform spatial transformation on the initial fusion image features of each frame of video image to obtain the image fusion features of each frame of video image. In one embodiment, the image features of each frame of video image and the corresponding memory shift features may also be fused to obtain the initial memory shift fusion features corresponding to each frame of video image, and the initial memory shift fusion features corresponding to each frame of video image may be spatially transformed to obtain the memory shift fusion features corresponding to each frame of video image.
[0077] In one embodiment, the computer device can call the second multi-head attention module in the frame-level time-domain feature fusion module based on memory shift to perform feature fusion on the image features of each frame of video image and the corresponding memory shift features to obtain the initial image fusion features of each frame of video image and the initial memory shift fusion features corresponding to each frame of video image, and call the second forward network to perform spatial transformation on the initial image fusion features of each frame of video image and the corresponding initial memory shift fusion features to obtain the image fusion features of each frame of video image and the memory shift fusion features corresponding to each frame of video image.
[0078] Among them, the embodiment of the present application can only calculate the fusion features of each frame of video image once, but in order to ensure the accuracy of instance segmentation, the embodiment of the present application can also calculate the image fusion features of each frame of video image multiple times, thereby obtaining multiple image fusion features of each frame of video image, so that the most appropriate image fusion features can be selected from the multiple image fusion features to perform instance segmentation on the video image. In this case, in addition to the architecture diagram shown in 3c, the instance segmentation model mentioned in the embodiment of the present application can also be as follows Figure 5a The architecture diagram shown in Figure 5a The frame-level time-domain feature fusion module based on memory shift in the instance segmentation model shown in FIG. 1 is added with a reverse memory shift module, that is, the frame-level time-domain feature fusion module based on memory shift can also be Figure 5a 501. The reverse memory shift module can be used to perform reverse time domain shift processing on the memory shift fusion features processed by the second multi-head attention module and the second forward network. The reverse shift memory features obtained after the reverse time domain shift processing can be used to calculate the image fusion features of the next video image. In this way, the information interaction in the time domain can be emphasized, and the accuracy of instance segmentation is further improved.
[0079] In one embodiment, the number of image fusion features of each frame of video image may be L, and performing feature fusion on the image features of each frame of video image and the corresponding memory shift features to obtain the image fusion features of each frame of video image may include the following steps s11-s15:
[0080] s11. Determine candidate image fusion features and corresponding memory shift fusion features of each frame of video image based on the image features of each frame of video image and the corresponding memory shift features.
[0081] In a specific implementation, the computer device can use a second multi-head attention module to perform feature fusion on the image features and the corresponding memory shift features of each frame of video image to obtain the initial image fusion features of each frame of video image and the initial memory shift fusion features corresponding to each frame of video image, and then call the second forward network to perform spatial transformation on the initial image features and the corresponding initial memory shift fusion features of each frame of video image to obtain the image fusion features of each frame of video image and the memory shift fusion features corresponding to each frame of video image.
[0082] s12. Perform reverse time domain shift processing on the memory shift fusion features corresponding to each frame of video image to obtain reverse shift features corresponding to each frame of video image.
[0083] In one embodiment, the above-mentioned frame-level temporal feature fusion module 501 based on memory shift includes a reverse memory shift module, and the computer device can call the reverse memory shift module to perform reverse temporal shift processing on the memory shift fusion feature corresponding to each frame of video image to obtain the reverse shift feature corresponding to each frame of video image. Figure 5a The reverse shift feature 51 corresponding to the first frame of video image, the reverse shift feature 52 corresponding to the second frame of video image, and the reverse shift feature 53 corresponding to the third frame of video image are shown. It can be seen that the reverse shift feature 51, the reverse shift feature 52, and the reverse shift feature 53 may include partial image features of the three frames of video images.
[0084] In one embodiment, the reverse memory shift module is called to perform reverse time domain shift processing on the memory shift fusion features corresponding to each frame of video image, and the specific implementation method of obtaining the reverse shift features corresponding to each frame of video image can be: calling the reverse memory shift module to obtain the moving step and moving direction of each memory feature group, and performing reverse time domain shift on the memory shift fusion features corresponding to each frame of video image according to the moving step and the reverse direction of the moving direction of each memory feature group, and finally obtaining the reverse shift features corresponding to each frame of video image. For example Figure 5b ,exist Figure 5b In the figure, the black cubes represent the temporal memory features, and the white cubes represent the image features of each frame of the video image ( Figure 5b 5 frames are taken as an example), and from top to bottom, the time series constructed by each frame of video image (i.e. Figure 5bT in the figure represents the time domain in the time series, and G represents the memory feature group. From left to right, four memory feature groups are represented. The first memory feature group: the moving step is 1 (i.e., S=1), and the moving direction is upward; the second memory feature group: the moving step is 1 (i.e., S=1), and the moving direction is downward; the third memory feature group: the moving step is 2 (i.e., S=2), and the moving direction is upward; the fourth memory feature group has a moving step of 2 (i.e., S=2), and the moving direction is downward; the computer device moves the memory shift fusion feature corresponding to each frame of the video image in the opposite direction of the moving step and moving direction of the four memory feature groups, and obtains the reverse shift feature corresponding to each frame of the video image. That is, when performing reverse time domain shift, the first group: the moving step is 1 (i.e. S=1), and the moving direction is downward; the second group: the moving step is 1 (i.e. S=1), and the moving direction is upward; the third group: the moving step is 2 (i.e. S=2), and the moving direction is downward; the fourth group: the moving step is 2 (i.e. S=2), and the moving direction is upward.
[0085] It should be noted that step s12 may be executed before step s13 or after step s13, and this application does not limit this.
[0086] s13. Count the number of candidate image fusion features for each frame of video image.
[0087] In a specific implementation, the computer device may count the number of candidate image fusion features of each frame of video image, and determine whether the number of candidate image fusion features of each frame of video image is greater than L. If the number of candidate image fusion features of each frame of video image is less than L, step s14 is executed; if the number of candidate image fusion features of each frame of video image is equal to L, step s15 is executed.
[0088] s14. If the determined number is less than L, the new image features and the corresponding new time-domain memory features of each frame of video image can be obtained, and the new time-domain memory features corresponding to each frame of video image can be subjected to time-domain shift processing to obtain the new memory shift features corresponding to each frame of video image; then, feature fusion is performed based on the new image features and the corresponding new memory shift features of each frame of video image to obtain new candidate image fusion features of each frame of video image. The new image features and the corresponding new time-domain memory features of each frame of video image are determined based on the candidate image fusion features and the corresponding reverse shift features of each frame of video image.
[0089] In a specific implementation, if it is determined that the number of candidate image fusion features of each frame of video image is less than L, the candidate image fusion features of each frame of video image and the corresponding reverse shift features can be used as inputs of a frame-level time-domain feature fusion module based on memory shift, that is, the candidate image fusion features of each frame of video image are regarded as new initial image features of each frame of video image, and the reverse shift features corresponding to each frame of video image are regarded as new initial time-domain memory features corresponding to each frame of video image; then, the new initial image features of each frame of video image and the corresponding new initial time-domain memory features are subjected to feature fusion, spatial transformation and other processing to obtain new image features and corresponding new time-domain memory features of each frame of video image, and then the new time-domain memory features corresponding to each frame of video image are subjected to time-domain shift processing to obtain new memory shift features corresponding to each frame of video image, and feature fusion is performed based on the new image features of each frame of video image and the corresponding new memory shift features to obtain new candidate image fusion features of each frame of video image. Then, the number of candidate image fusion features (including new candidate image fusion features at this time) of each frame of video image is counted again until the number of candidate image fusion features of each frame of video image is equal to L, and step s15 is executed. It should be noted that when the number of candidate image fusion features of each frame of video image is not equal to L, the new candidate image fusion features of each frame of video image obtained last time will be used as the new initial image features of each frame of video image when calculating the new candidate image fusion features of each frame of video image next time, and the reverse shift features corresponding to each frame of video image obtained by the reverse time domain shift processing last time will be used as the new initial time domain memory features corresponding to each frame of video image when calculating the new candidate image fusion features of each frame of video image next time. Among them, the new initial image features of each frame of video image and the corresponding new initial time domain memory features are subjected to feature fusion and spatial transformation to obtain the new image features and the corresponding new time domain memory features of each frame of video image, and the new time domain memory features corresponding to each frame of video image are subjected to time domain shift processing to obtain the new memory shift features corresponding to each frame of video image. The specific implementation method can refer to the description of the corresponding part above, which will not be repeated here.
[0090] For example, when L is 3, N is 3; for image 1 in the 3 frames of video images, the computer device can determine the candidate image fusion feature 1 and the corresponding memory shift fusion feature 1 of image 1 based on the image features of image 1 and the corresponding memory shift features; perform reverse time domain shift processing on the memory shift fusion feature 1 corresponding to image 1 to obtain the reverse shift feature 1 corresponding to image 1, and then count the number of image fusion features of image 1 as 1; if the number is less than L, the computer device can use the candidate image fusion feature 1 of image 1 as the new initial image feature of image 1, and use the reverse shift feature 1 corresponding to image 1 as the new initial time domain memory feature corresponding to image 1, and then based on the new initial image feature of image 1 and the corresponding new initial time domain memory feature, obtain the new image feature and the corresponding new time domain memory feature of image 1, and process according to steps S202-S204 to obtain the candidate image fusion feature 2 of image 1. (i.e., new candidate image fusion features) and the corresponding memory shift fusion feature 2, and reverse time-domain shift the memory shift fusion feature 2 corresponding to image 1 to obtain the reverse shift feature 2 corresponding to image 1; the number of candidate image fusion features of image 1 is counted again as 2, which is less than 3. At this time, the candidate image fusion feature 2 of image 1 is used as the new initial image feature of image 2, and the reverse shift feature 2 corresponding to image 1 is used as the new initial time-domain memory feature corresponding to image 1, and another candidate image fusion feature 3 of image 1 is obtained in the above manner. At this time, the number of candidate image fusion features of image 1 is counted as 3, which is equal to L. The three candidate image fusion features of image 1 (i.e., candidate image fusion feature 1, candidate image fusion feature 2, and candidate image fusion feature 3) are all used as image fusion features of image 1. The L image fusion features of the remaining two frames of video images can refer to the specific implementation method of determining the L image fusion features of image 1, which will not be repeated here.
[0091] It should be noted that the above description is only exemplary with L being 3. In actual practice, L can be 4, 5, etc., and the embodiments of the present application do not limit this. The L image fusion features for any frame of video image can be determined by referring to the implementation method of the L image fusion features of the above-mentioned image 1. L here can also be understood as: the number of times the frame-level time domain feature fusion module based on memory shift needs to be called.
[0092] s15. If the number is equal to L, the L candidate image fusion features of each frame of video image are determined as the L image fusion features of each frame of video image.
[0093] In one embodiment, when the number of image fusion features of each frame of video image is one (i.e., L=1), the computer device can directly perform feature fusion on the image features of each frame of video image and the corresponding memory shift features to obtain the image fusion features of each frame of video image, and use the image fusion features of each frame of video image to perform instance segmentation on each frame of video image. That is to say, it is sufficient to go through steps S402-S404 once, and at this time, it is also unnecessary to perform time domain reverse shift on the memory shift fusion features corresponding to each frame of video image. When the number of image fusion features of each frame of video image is multiple (i.e., L is greater than 1), the L candidate image fusion features of each frame of video image are determined as the L image fusion features of each frame of video image.
[0094] S405, obtaining M instance query vectors; M can be set according to the requirements, for example, M can be 3, 10, 100, etc. The M instance query vectors can be preset or randomly initialized. The M instance query vectors can be used to perform instance prediction on the instances included in the input N frames of video images. The M instance query vectors can be understood as parameters of the spatiotemporal interactive query module in the instance segmentation model.
[0095] S406 : broadcast the M instance query vectors to the N frames of video images to obtain M instance query vectors corresponding to each frame of video image.
[0096] Among them, for the M instance query vectors obtained in step S405, the M instance query vectors can be broadcast to N frames of video images to associate the M instance query vectors with each frame of video images respectively, that is, when performing instance segmentation on each frame of video images, the M instance query vectors will be used.
[0097] S407 . Based on the image fusion features of each frame of video image and the corresponding M instance query vectors, instance segmentation is performed on the N frames of video image to obtain instances included in each frame of video image.
[0098] In one embodiment, the computer device can perform feature fusion analysis on the M instance query vectors corresponding to each frame of video image to obtain M fused instance query vectors corresponding to each frame of video image; then, based on the image fusion features of each frame of video image and the corresponding M fused instance query vectors, perform instance segmentation on the N frames of video images to obtain the instances included in each frame of video image.
[0099] Among them, performing feature fusion analysis on the M instance query vectors corresponding to each frame of video image to obtain M fused instance query vectors corresponding to each frame of video image can include: performing intra-frame feature fusion on the M instance query vectors corresponding to each frame of video image to obtain M intra-frame instance query vectors corresponding to each frame of video image; performing inter-frame feature fusion on the M intra-frame instance query vectors corresponding to N frames of video images to obtain M fused instance query vectors corresponding to each frame of video image.
[0100] In one embodiment, Figure 5a The instance segmentation model shown, the spatiotemporal interactive query module 502 in the instance segmentation model may include multiple third multi-head attention modules (i.e., intra-frame attention modules) and fourth multi-head attention modules (i.e., inter-frame attention modules), and the third multi-head attention module and the fourth multi-head attention module may be MASH; the third multi-head attention module is used for intra-frame feature fusion, which can be understood as the third multi-head attention module can make the M instance query vectors corresponding to each frame of the video image pay attention to each other's instance information; the fourth multi-head attention module is used for inter-frame feature fusion, which can be understood as the fourth multi-head attention module can make the instance query vector between frames pay attention to the instance information of its own instance in other frames, and the instance-level features in the time domain can be integrated through the intra-frame and inter-frame multi-head attention machines. Among them, the module parameters in the multiple third multi-head attention modules and the fourth multi-head attention modules are consistent, such as the weight sharing in the third multi-head attention module and the fourth multi-head attention module, which can improve the parameter utilization of the instance segmentation model. The number of the third multi-head attention modules can be determined according to the number of video images in the input video. For example, in Figure 5a, if three frames of video images are input, there will be three multi-head attention modules.
[0101] Performing intra-frame feature fusion on the M instance query vectors corresponding to each frame of video image to obtain the M intra-frame instance query vectors corresponding to each frame of video image may include: the computer device may call N multi-head attention modules to perform intra-frame feature fusion on the M instance query vectors corresponding to the N frames of video image to obtain the M intra-frame instance query vectors corresponding to each frame of video image, and one multi-head attention module processes the M instance query vectors corresponding to one frame of video image. For example, in Figure 5a In the example, M is 3, and the computer device can call a third multi-head attention module to perform intra-frame feature fusion on the three instance query vectors corresponding to the video image 1 to obtain the three intra-frame instance query vectors corresponding to the video image 1, that is, Figure 5a As shown in 55 in ; the computer device can call a third multi-head attention module to perform intra-frame feature fusion on the three instance query vectors corresponding to the video image 2 to obtain the three intra-frame instance query vectors corresponding to the video image 2, that is, Figure 5aAs shown in 56 in the figure; then the computer device can call a third multi-head attention module to perform intra-frame feature fusion on the three instance query vectors corresponding to the video image 3 to obtain the three intra-frame instance query vectors corresponding to the video image 3, that is, Figure 5a As shown in 57.
[0102] In one embodiment, performing inter-frame feature fusion on M intra-frame instance query vectors corresponding to N frames of video images to obtain M fused instance query vectors corresponding to each frame of video images may include: obtaining the i-th intra-frame instance query vector among the M intra-frame instance query vectors corresponding to each frame of video images, where i is greater than or equal to 1 and i is less than or equal to M; performing inter-frame feature fusion on the i-th intra-frame instance query vector corresponding to each frame of video images based on the i-th intra-frame instance query vector corresponding to the N frames of video images to obtain the i-th fused instance query vector corresponding to each frame of video images. In one embodiment, the computer device may call the fourth multi-head attention module to perform inter-frame feature fusion on the i-th intra-frame instance query vector corresponding to each frame of video images based on the i-th intra-frame instance query vector corresponding to the N frames of video images to obtain the i-th fused instance query vector corresponding to each frame of video images.
[0103] For example, Figure 5a , M = 3, the computer device can obtain the first intra-frame instance query vector corresponding to the video image 1 (i.e. Figure 5a The intra-frame instance query vector 58 in the video image 2, the first intra-frame instance query vector corresponding to the video image 2 (i.e. Figure 5a The intra-frame instance query vector 59 in the video image 3 corresponds to the first intra-frame instance query vector (i.e. Figure 5a Then, the fourth multi-head attention module is called to perform inter-frame feature fusion on the first intra-frame instance query vector corresponding to video image 1, the first intra-frame instance query vector corresponding to video image 2, and the first intra-frame instance query vector corresponding to video image 3 to obtain the first fused instance query vector corresponding to video image 1 (i.e. Figure 5a The fused instance query vector 61 in the video image 2 and the first fused instance query vector corresponding to the video image 2 (i.e. Figure 5a The fused instance query vector 62 in the video image 3 and the first fused instance query vector corresponding to the video image 3 (i.e. Figure 5aThe computer device may then obtain the second intra-frame instance query vector corresponding to video image 1, the second intra-frame instance query vector corresponding to video image 2, and the second intra-frame instance query vector corresponding to video image 3, and obtain the third intra-frame instance query vector corresponding to video image 1, the third intra-frame instance query vector corresponding to video image 2, and the third intra-frame instance query vector corresponding to video image 3, and perform inter-frame feature fusion on the first intra-frame instance query vector of each frame of video image by calling the fourth multi-head attention module, so as to finally obtain the second fused instance query vector corresponding to video image 1, the second fused instance query vector corresponding to video image 2, and the second fused instance query vector corresponding to video image 3; and obtain the third fused instance query vector corresponding to video image 1, the third fused instance query vector corresponding to video image 2, and the third fused instance query vector corresponding to video image 3.
[0104] In one embodiment, when there are L image fusion features per frame of video image, instance segmentation is performed on N frames of video image based on the image fusion features of each frame of video image and the corresponding M fusion instance query vectors to obtain the instance included in each frame of video image. The specific implementation method may be: select the target image fusion feature corresponding to each frame of video image from the L image fusion features of each frame of video image, and perform instance segmentation on each frame of video image based on the target image fusion feature corresponding to each frame of video image and the corresponding M fusion instance query vectors to obtain the instance included in each frame of video image. Next, the target video image is taken as an example for explanation, and the target video image is any frame in the N frames of video image:
[0105] Based on the image fusion feature of the target video image and the corresponding M fusion instance query vectors, the target video image is segmented into instances, and obtaining the instances included in the target video image may include: selecting the target image fusion feature from the L image fusion features of the target video image; dynamically convolving the target image fusion feature with the M fusion instance query vectors corresponding to the target video image to obtain the instances included in the target video image. In a specific implementation, the M fusion instance query vectors corresponding to the target video image may be used as a convolution kernel, and the convolution kernel is convolved with the target image fusion feature to obtain a binary mask map of the target video image, and the instances included in the target video image can be distinguished from the binary mask map.
[0106] Among them, when L is 1, the specific implementation method of selecting the target image fusion feature from the L image fusion features of the target video image may be: directly determining the image fusion feature of the target video image as the target image fusion feature. When L is greater than or equal to 2, the specific implementation method of the computer device selecting the target image fusion feature from the L image fusion features of the target video image is: determining the instance size in the target video image, and selecting the target image fusion feature from the L image fusion features of the target video image based on the instance size in the target video image. Specifically, a correspondence between the instance size in the video image and the image fusion feature can be established, and the target image fusion feature can be selected from the L image fusion features based on the instance size in the target video image and the correspondence. In some embodiments, when the instance size in the target video image is larger, the image fusion feature with a larger pixel can be selected, and this application is not limited to this.
[0107] In one embodiment, based on the image fusion feature of the target video image and the corresponding M fusion instance query vectors, the target video image is segmented into instances, and the instance included in the target video image may include: selecting the target image fusion feature from the L image fusion features of the target video image, where the target video image is any frame in the N frames of video images; performing S dynamic convolutions based on the target image fusion feature and the M fusion instance query vectors corresponding to the target video image, to obtain the instance included in the target video image. When S is 1, that is, the target image fusion feature is directly dynamically convolved with the M fusion instance query vectors corresponding to the target video image, the instance included in the target video image can be obtained.
[0108] When S is greater than 1, the computer device can perform S dynamic convolutions based on the target image fusion features and the M fusion instance query vectors corresponding to the target video image, and can obtain M updated fusion instance query vectors corresponding to the target video image after the S dynamic convolutions and the candidate instances included in the target video image; wherein, after one dynamic convolution process, M updated fusion instance query vectors and the candidate instances included in the target video image can be obtained; when performing S dynamic convolutions, the M fusion instance query vectors used in each dynamic convolution can capture the information of the image fusion features in the target video image, and obtain M updated fusion instance query vectors, so the M fusion instance query vectors used in the next dynamic convolution are all obtained by processing the M updated fusion instance query vectors corresponding to the previous time. That is, the M updated fused instance query vectors obtained by the y-th dynamic convolution are obtained by dynamically convolving the fused instance query vector of the y-1th time with the fusion features of the target image, and the fused instance query vector of the y-1th time is obtained by performing feature fusion analysis on the M updated fused instance query vectors obtained by the y-1th dynamic convolution; y is greater than 1, and y is less than or equal to S; the candidate instance included in the target video image obtained by the S-th dynamic convolution is determined as the instance included in the target video image. Among them, the M updated fused instance query vectors are obtained by capturing the image features in the target video image using the M fused instance query vectors, that is, when performing dynamic convolution, the M fused instance query vectors can capture the image features in the target video image, so that when the M updated fused instance query vectors are used to determine the M fused instance query vectors next time, after inter-frame feature fusion and intra-frame feature fusion, and using the parameters of the spatial dimension and the temporal dimension, the instance query in the two dimensions of the temporal dimension and the spatial dimension can be efficiently completed. At the same time, when determining the M fusion instance query vectors corresponding to each frame of video image, the parameters are reused without introducing additional new parameters, thereby achieving efficient instance-level time-domain feature synthesis.
[0109] Among them, the y-1th fused instance query vector is obtained by performing feature fusion analysis on the M updated fused instance query vectors obtained by the y-1th dynamic convolution, which can be: calling the third multi-head attention module corresponding to the target video image to perform intra-frame feature fusion on the M updated fused instance query vectors obtained by the y-1th dynamic convolution, and obtain the new M intra-frame instance query vectors corresponding to the target video image, that is, each frame of the video image can finally obtain the corresponding new M intra-frame instance query vector. Then call the fourth multi-head attention module to perform inter-frame feature fusion on the new M intra-frame instance query vectors corresponding to the N frames of video images, and obtain the y-1th fused instance query vector corresponding to the target video image.
[0110] For example, when S is 3, N is 2, and the two frames of video images are video image 1 and video image 2 respectively; the computer device can perform a first dynamic convolution based on the target image fusion feature of video image 1 and the M fusion instance query vectors corresponding to video image 1 to obtain M updated fusion instance query vectors of video image 1 after the first dynamic convolution and the candidate instances included in video image 1; at the same time, the target image fusion feature of video image 2 and the M fusion instance query vectors corresponding to video image 2 can be subjected to a first dynamic convolution to obtain M updated fusion instance query vectors of video image 2 after the first dynamic convolution and the candidate instances included in video image 2; then the third multi-head attention module corresponding to video image 1 is called to perform intra-frame feature fusion on the M updated fusion instance query vectors of video image 1 after the first dynamic convolution to obtain M intra-frame instance query vectors of video image 1 after the first dynamic convolution; at the same time, the third multi-head attention module corresponding to video image 2 is called to perform intra-frame feature fusion on the M updated fusion instance query vectors of video image 2 after the first dynamic convolution to obtain video image 2 after the first dynamic convolution. The corresponding M intra-frame instance query vectors are then inter-frame feature fused for the M intra-frame instance vectors corresponding to the N frames of video image 1 after the first dynamic convolution and the M intra-frame instance vectors corresponding to the video image 2 after the first dynamic convolution, to obtain the M fused instance query vectors corresponding to the video image 1 after the first dynamic convolution and the M fused instance query vectors corresponding to the video image 2. The M fused instance query vectors corresponding to the video image 1 after the first dynamic convolution and the target image fusion features of the video image 1 are subjected to a second dynamic convolution to obtain the M updated fused instance query vectors corresponding to the video image 1 after the second dynamic convolution and the candidate instances included in the video image 1. Then, the third multi-head attention module corresponding to video image 1 is called to perform intra-frame feature fusion on the M updated fused instance query vectors corresponding to video image 1 after the second dynamic convolution, and obtain the M intra-frame instance query vectors corresponding to video image 1 after the second dynamic convolution; and the fourth multi-head attention module is called to perform inter-frame feature fusion on the M intra-frame instance query vectors corresponding to video image 1 after the second dynamic convolution and the M intra-frame instance query vectors corresponding to video image 2 after the second dynamic convolution, and obtain the M fused instance query vectors corresponding to video image 1 and the M fused instance query vectors corresponding to video image 2. The M fused instance query vectors corresponding to video image 1 after the second dynamic convolution and the target image fusion features of video image 1 are subjected to a third dynamic convolution to obtain the M updated fused instance query vectors corresponding to video image 1 after the third dynamic convolution and the candidate instances included in video image 1; finally, the candidate instances included in video image 1 obtained after the third dynamic convolution are used as the instances included in video image 1, and the candidate instances included in video image 2 obtained after the third dynamic convolution are used as the instances included in video image 2.
[0111] It should be noted that the above only gives S=3 as an example. In the embodiment of the present application, S can be set according to the requirements, for example, S can be 3 times, 6 times, etc. It should be understood that when S dynamic convolutions are performed, the M updated instance query vectors obtained by the previous dynamic convolution will be used to determine the M fused instance query vectors next time, that is, as Figure 5a The input of the spatiotemporal interactive query module 302 in the example above means that the spatiotemporal interactive query module can be called S times in the end.
[0112] In an embodiment of the present application, after obtaining N frames of video images in the target video, the computer device, on the one hand, obtains image features of each frame of video image and time-domain memory features corresponding to each frame of video image, and performs time-domain shift processing on the time-domain memory features corresponding to each frame of video image to obtain memory shift features corresponding to each frame of video image; performs feature fusion on the image features of each frame of video image and the corresponding memory shift features to obtain image fusion features of each frame of video image, which can realize information interaction of each frame of video image in the time domain and can effectively improve the subsequent video segmentation of each frame of video image; on the other hand, the computer device can obtain M instance query vectors, and broadcast the M instance query vectors to the N frames of video images to obtain M instance query vectors corresponding to each frame of video image, and perform instance segmentation on the N frames of video images based on the image fusion features of each frame of video image and the corresponding M instance query vectors to obtain the instances included in each frame of video image. M query vectors can be responsible for predicting the instance prediction of each frame of video image. Instance segmentation is performed on each frame of video image based on the instance query vector and the fused image feature set. There is no need to rely on post-processing methods such as non-maximum suppression. The instances included in each frame of video image are directly output end-to-end, achieving fast, efficient and accurate video instance segmentation.
[0113] The instances included in each frame of the video image can be obtained by calling an instance segmentation model to perform instance segmentation on each frame of the video image. The instance segmentation model can include a backbone network and a network prediction head. The instance segmentation model uses networks such as PVT-B1 as the basic network, and introduces a frame-level time domain feature fusion module based on memory shift in the basic network to construct the backbone network in the instance segmentation model, and introduces a spatiotemporal query module in the network prediction head. The network parameters of the target basic network are initialized by weights pre-trained on the MS-COCO instance segmentation dataset, and the dataset used is the COCO Instance Seg dataset. Based on the instance segmentation model, this application also provides a model training method, see Figure 6 The model training method may be executed by a computer device, which may be the terminal device 101 or the server 102. The model training method described in this embodiment may include the following steps S601-S605:
[0114] S601. Obtain R frames of sample video images sampled from a sample video and an instance annotation corresponding to each frame of the sample video image, where R is an integer greater than 1.
[0115] The sample video may be a pre-recorded video, or the sample data may be a sample video from three large-scale video segmentation baseline datasets: YouTube-VIS, OVIS, and YouTube-VIS-2021. The instance annotation may be one or more of an instance category, a bounding box, an instance foreground mask, and an instance embedding vector.
[0116] In a specific implementation, a computer device can randomly sample R consecutive frames of sample video images from a sample video. For example, R can be set as required, and R can be 5, 10, 11, and so on. In the embodiment of the present application, the single-stage instance segmentation network QueryIns is improved and extended to the field of video instance segmentation. Each frame of the video image will be scaled to a certain size first; for example, each frame of the video image will be scaled to an image with a short side at (320, 800) pixels, and the long side will be scaled to 1333. This application example is not limited to this. Later in training, the instance segmentation model QueryVIS (or Tevit, QueryVIS: Query Video Instance Segmentation) of the embodiment of the present application uses adaptive moment estimation weight decay (Adaptive moment estimation weight decay, AdamW) as an optimizer for training on 8 graphics cards, and the number of samples used for one gradient descent is 80. The QueryInst proposed in the embodiment of the present application performed a total of 12 complete iterations on the video instance segmentation dataset, with the initial learning rate set to 0.000025, and the learning rate was divided by 10 after the 8th and 11th complete iterations, respectively.
[0117] S602, calling the initial instance segmentation model to obtain sample image features of each frame of sample video image and sample time domain memory features corresponding to each frame of sample video image, where the sample time domain memory features are used to indicate sample memory image features of the sample video image in the time domain.
[0118] In a specific implementation, the computer device may call the initial feature extraction module included in the initial segmentation model to extract the initial image features of each frame of the sample video image, and obtain the sample temporal memory features corresponding to each frame of the sample video image. The sample temporal memory features may be understood as model parameters in the initial segmentation model. In some embodiments, the sample temporal memory features may be an initial frame-level temporal feature fusion module based on memory shift inserted into the initial instance segmentation model by generating a plug-in. Then, the first multi-head attention module in the initial frame-level temporal feature fusion module based on memory shift is called to perform feature fusion on the initial image features of each frame of the sample video image and the corresponding sample temporal memory features, and obtain the intermediate sample image features of each frame of the sample video image and the intermediate sample temporal memory features corresponding to each frame of the sample video image. Then, the computer device may call the first forward network in the frame-level temporal feature fusion module based on memory shift to perform spatial transformation on the intermediate sample image features of each frame of the sample video image and the intermediate sample temporal memory features corresponding to each frame of the sample video image, and obtain the sample image features of each frame of the sample video image and the sample temporal memory features corresponding to each frame of the sample video image.
[0119] S603 , performing time domain shift processing on the sample time domain memory feature corresponding to each frame of the sample video image to obtain the sample memory shift feature corresponding to each frame of the sample video image.
[0120] In one embodiment, the initial memory shift module in the initial memory shift-based frame-level time domain feature fusion module can be called to perform time domain shift processing on the sample time domain memory features corresponding to each frame of sample video image to obtain the sample memory shift features corresponding to each frame of sample video image.
[0121] S604: Based on the sample image features of each frame of the sample video image and the corresponding sample memory shift features, instance segmentation is performed on the R frame sample video image to obtain the predicted instances included in each frame of the sample video image.
[0122] In a specific implementation, the computer device can call the second multi-head attention module in the initial frame-level time-domain feature fusion module based on memory shift to perform feature fusion on the sample image features of each frame of sample video image and the corresponding sample memory shift features to obtain the intermediate sample image fusion features of each frame of sample video image, and then call the second forward network in the initial frame-level time-domain feature fusion module based on memory shift to perform spatial transformation on the intermediate sample image fusion features of each frame of sample video image to obtain the sample image fusion features of each frame of sample video image; and then perform instance segmentation on each frame of sample video image based on the sample image fusion features of each frame of sample video image.
[0123] In one embodiment, in the model initialization stage, the embodiment of the present application can randomly initialize M sample instance query vectors by randomly initializing instances. For example, the value of M is 50, 100, 200, etc., which is not limited in the embodiment of the present application. M randomly initialized sample instance query vectors are responsible for predicting video instances in the input video segment (i.e., R frame sample video images). For the input video image, M instance query vectors are broadcast to each frame of sample video image, and then the instance level features in the time domain are integrated through the multi-head attention mechanism within and between frames. Specifically, M sample instance query vectors can be obtained, and the M sample instance query vectors can be broadcast to each frame of sample video image to obtain M sample instance query vectors corresponding to each frame of sample video image; then the third multi-head attention module corresponding to each frame of sample video image in the initial spatiotemporal interactive query module is called to perform intra-frame feature fusion on the M sample instance query vectors of each frame of sample video image to obtain M intra-frame sample instance query vectors corresponding to each frame of sample video image; then the fourth multi-head attention module is called to perform inter-frame feature fusion on the M intra-frame sample instance query vectors corresponding to each frame of sample video image to obtain the sample image fusion feature of each frame of sample video image; then based on the sample image fusion feature of each frame of sample video image and the corresponding M fused sample instance query vectors, instance segmentation is performed on the R frames of sample video images to obtain the predicted instances included in each frame of sample video image.
[0124] In one embodiment, in the process of calling the second multi-head attention module in the initial frame-level temporal feature fusion module based on memory shift to perform feature fusion on the sample image features of each frame of sample video image and the corresponding sample memory shift features to obtain the intermediate sample image fusion features of each frame of sample video image, calling the second multi-head attention module in the initial frame-level temporal feature fusion module based on memory shift to perform feature fusion on the sample image features of each frame of sample video image and the corresponding sample memory shift features, and also obtain the intermediate sample memory shift features corresponding to each frame of sample video image, and using the second forward network in the initial frame-level temporal feature fusion module based on memory shift to perform spatial transformation on the intermediate sample memory shift features corresponding to each frame of sample video image, and obtain the sample memory shift fusion features corresponding to each frame of sample video image. Wherein, the number of sample image fusion features can be L, and L is an integer greater than or equal to 1.
[0125] The specific implementation method of how to determine the L sample image fusion features of each frame of sample video image can refer to the specific implementation method of how to determine the L image fusion features of each frame of sample video image, which will not be repeated here.
[0126] In one embodiment, when the initial memory shift-based frame-level temporal feature fusion module is called to obtain L sample fusion image features of each frame of sample video image, based on the sample image fusion features of each frame of sample video image and the corresponding M fusion sample instance query vectors, the R frame video image is instance segmented to obtain the predicted instance included in each frame of sample video image, which may include: selecting the target sample image fusion feature from the L sample image fusion features of the target sample video image, the target sample video image is any frame in the R frame sample video image; then calling the initial spatiotemporal query interaction module, based on the target sample image fusion feature and the M sample fusion instance query vectors corresponding to the target sample video image, to obtain M updated sample fusion instance query vectors corresponding to the target sample video image after S dynamic convolutions and the sample candidate instances included in the target sample video image; after one dynamic convolution process, the M updated sample fusion instance query vectors and the sample candidate instances included in the target sample video image are obtained; the M updated sample fusion instance query vectors obtained by the yth dynamic convolution are based on the y-1th The y-1th sample fusion instance query vector is obtained by dynamically convolving the fusion features of the target sample image, and the y-1th sample fusion instance query vector is obtained by performing feature fusion analysis on the M updated sample fusion instance query vectors obtained by the y-1th dynamic convolution; y is greater than 1, and y is less than or equal to S; the sample candidate instances included in each frame of the sample video image obtained after each dynamic convolution are all used as the predicted instances included in each frame of the sample video image.
[0127] S605: Based on the predicted instances and corresponding instance annotations included in each frame of the sample video image, the model parameters of the initial instance segmentation model are updated to obtain the instance segmentation model. The model parameters may include network parameters in the initial memory shift-based frame-level temporal feature fusion module, network parameters in the initial spatiotemporal interactive query module, M instance query vectors, and the like.
[0128] It can be seen from step S604 that the sample candidate instances included in each frame of sample video image obtained after each dynamic convolution are all used as predicted instances included in each frame of sample video image. When calculating the model loss value, it is necessary to refer to the loss value between the predicted instance included in each frame of sample video image obtained after each dynamic convolution and the corresponding instance annotation. The computer device can calculate the loss between the predicted instance included in each frame of sample video image obtained after each dynamic convolution and the corresponding instance annotation, and obtain the loss value corresponding to each frame of sample video image after each dynamic convolution; based on the loss value corresponding to each frame of sample video image obtained by S dynamic convolutions, determine the model loss value, and update the network parameters of the initial instance segmentation model based on the model loss value to obtain the instance segmentation model.
[0129] In the embodiment of the present application, the instance segmentation model (TeViT) adopts a "one-to-one" connection method between sampling samples (i.e., sample video images) and true values (i.e., labeled instances). Specifically, for several existing instance query vectors, when performing sample connection and loss function calculation, in order to achieve the purpose of one instance query vector being responsible for one true value, the loss between the sample candidate instance and the corresponding instance annotation included in each frame of the sample video image obtained after each dynamic convolution is calculated, and the specific implementation method for obtaining the loss value corresponding to each frame of the sample video image after each dynamic convolution can be: in the training stage, after the instance query vector performs bounding box prediction, the loss value between the instance query vector and the true value (labeled instance) is calculated one by one, and a two-dimensional loss matrix between the prediction of the instance query vector (predicted instance) and the bounding box of the true value (i.e., labeled instance) is obtained. Afterwards, a bipartite graph matching algorithm such as Hungarian is used to perform a one-to-one match between the instance query vector and the true value bounding box in the two-dimensional loss matrix, so as to obtain the matching result between the instance query and the true value bounding box, that is, a two-dimensional loss matrix can be generated after each dynamic convolution, and a bipartite graph matching algorithm such as Hungarian is used to obtain the matching result between the instance query (i.e., candidate instance) and the true value bounding box, and finally the loss between the sample candidate instance and the corresponding instance annotation included in each frame of the sample video image obtained after each dynamic convolution can be obtained. Since the instance query vector matched to the true value can be responsible for predicting the instance category, bounding box, instance mask and instance embedding vector of the true value, the loss function is the instance-level loss between the instance query vector prediction (i.e., predicted instance) and its corresponding true value (i.e., instance annotation). In particular, for instance query vectors that are not matched to the true value, the corresponding loss function only contains the classification loss of negative samples whose true value is a negative sample.
[0130] In an embodiment of the present application, R frames of sample video images sampled from a sample video and instance annotations corresponding to each frame of sample video images are obtained, where R is an integer greater than 1; the initial instance segmentation model is called to obtain sample image features of each frame of sample video images and sample temporal memory features corresponding to each frame of sample video images, the sample temporal memory features being used to indicate sample memory image features of the sample video images in the time domain; the sample temporal memory features corresponding to each frame of sample video images are subjected to temporal shift processing to obtain sample memory shift features corresponding to each frame of sample video images; based on the sample image features of each frame of sample video images and the corresponding sample memory shift features, the R frames of sample video images are subjected to instance segmentation to obtain the predicted instances included in each frame of sample video images; based on the predicted instances included in each frame of sample video images and the corresponding instance annotations, the initial instance segmentation model is updated to obtain the instance segmentation model. By training the initial instance segmentation model to obtain the instance segmentation model, an end-to-end instance segmentation model can be obtained, thereby improving the accuracy of the instance segmentation model.
[0131] Using the instance segmentation method proposed in the embodiment of the present application, the instance segmentation method can realize the temporal memory shift of the temporal memory features corresponding to each frame of the video image and use M instance query vectors to interact with information within and between frames (that is, to realize information interaction in the time domain and space), so as to robustly and accurately detect, segment, and find instances in the video. On the three large-scale video strength segmentation baseline datasets of YouTube-VIS, OVIS and YouTube-VIS-2021, this application has achieved a considerable accuracy improvement in mAP (Mean Average Precision, an evaluation indicator of the video instance segmentation algorithm). Compared with other current advanced methods, this instance segmentation method (TeViT) has achieved a large lead in both speed and accuracy. That is, on the three large-scale video strength segmentation baseline datasets of YouTube-VIS, OVIS and YouTube-VIS-2021, this method achieved the highest accuracy while maintaining a competitive inference speed. The following three tables are the verification results verified on the three large-scale video strength segmentation baseline datasets of YouTube-VIS, OVIS and YouTube-VIS-2021:
[0132] (1) The accuracy on the public dataset YouTube-VIS validation set is shown in Table 1, compared with the benchmark reference method
[0133] Table 1
[0134]
[0135]
[0136] (2) On the public dataset YouTube-VIS-2021, this method also achieved the current best performance, as shown in Table 2:
[0137] Table 2
[0138]
[0139]
[0140] (3) On the public dataset OVIS, this method also achieves the best performance currently, as shown in Table 3:
[0141] Table 3
[0142]
[0143] Based on the description of the above-mentioned instance segmentation method embodiment, the present application embodiment further discloses an instance segmentation device, which may be a computer program (including program code) running on the above-mentioned computer device. The instance segmentation device may execute Figure 2 , Figure 4 and Figure 6 See the method shown in Figure 7 , the instance segmentation device may include the following units:
[0144] An acquisition unit 701 is used to acquire N frames of video images in a target video, each frame of the video image includes one or more instances, and N is an integer greater than 1;
[0145] The acquisition unit 701 is further used to acquire the image features of each frame of the video image and the time domain memory features corresponding to each frame of the video image, wherein the time domain memory features are used to indicate the memory image features of the video image in the time domain;
[0146] The processing unit 702 is used to perform time-domain shift processing on the time-domain memory feature corresponding to each frame of the video image to obtain the memory shift feature corresponding to each frame of the video image;
[0147] The processing unit 702 is further configured to perform instance segmentation on the N frames of video images based on the image features of each frame of video images and the corresponding memory shift features, so as to obtain the instances included in each frame of video images.
[0148] In one embodiment, when the processing unit 702 performs time-domain shift processing on the time-domain memory feature corresponding to each frame of the video image to obtain the memory shift feature corresponding to each frame of the video image, it can be specifically used to:
[0149] Dividing the time-domain memory features corresponding to the N frames of video images into a plurality of memory feature groups, each memory feature group including the same number of time-domain memory features;
[0150] Obtaining the moving step length and moving direction of each memory feature group;
[0151] The time domain memory features in each memory feature group are moved according to the moving step length and moving direction of each memory feature group to obtain the memory shift features corresponding to each frame of the video image.
[0152] In one embodiment, when the processing unit 702 performs instance segmentation on the N frames of video images based on the image features of each frame of video images and the corresponding memory shift features to obtain the instances included in each frame of video images, it can be specifically used to:
[0153] Performing feature fusion on the image features of each frame of the video image and the corresponding memory shift features to obtain the image fusion features of each frame of the video image;
[0154] Based on the image fusion features of each frame of the video image, instance segmentation is performed on the N frames of video image to obtain instances included in each frame of the video image.
[0155] In one embodiment, the number of image fusion features of each frame of video image is L, where L is an integer greater than or equal to 1. When the processing unit 702 performs feature fusion on the image features of each frame of video image and the corresponding memory shift features to obtain the image fusion features of each frame of video image, the processing unit 702 may be specifically used to:
[0156] Determining candidate image fusion features and corresponding memory shift fusion features of each frame of video image based on the image features of each frame of video image and the corresponding memory shift features;
[0157] Performing reverse time-domain shift processing on the memory shift fusion feature corresponding to each frame of the video image to obtain a reverse shift feature corresponding to each frame of the video image;
[0158] Counting the number of candidate image fusion features of each frame of video image;
[0159] If the number is less than L, then obtain new image features and corresponding new time-domain memory features of each frame of video image, and perform time-domain shift processing on the new time-domain memory features corresponding to each frame of video image to obtain new memory shift features corresponding to each frame of video image; perform feature fusion based on the new image features and corresponding new memory shift features of each frame of video image to obtain new candidate image fusion features of each frame of video image, wherein the new image features and corresponding new time-domain memory features of each frame of video image are determined based on the candidate image fusion features and corresponding reverse shift features of each frame of video image;
[0160] If the number is equal to L, the L candidate image fusion features of each frame of video image are determined as the L image fusion features of each frame of video image.
[0161] In one embodiment, when the processing unit 702 performs instance segmentation on the N frames of video images based on the image fusion features of each frame of video images to obtain the instances included in each frame of video images, it can be specifically used to:
[0162] Obtain M instance query vectors;
[0163] Broadcasting the M instance query vectors to N frames of video images to obtain M instance query vectors corresponding to each frame of video image;
[0164] Based on the image fusion features of each frame of the video image and the corresponding M instance query vectors, instance segmentation is performed on the N frames of the video image to obtain the instances included in each frame of the video image.
[0165] In one embodiment, when the processing unit 702 performs instance segmentation on the N frames of video images based on the image fusion features of each frame of video images and the corresponding M instance query vectors to obtain the instances included in each frame of video images, it can be specifically used to:
[0166] Performing feature fusion analysis on the M instance query vectors corresponding to each frame of the video image to obtain the M fused instance query vectors corresponding to each frame of the video image;
[0167] Based on the image fusion features of each frame of the video image and the corresponding M fusion instance query vectors, instance segmentation is performed on the N frames of video images to obtain instances included in each frame of the video image.
[0168] In one embodiment, when the processing unit 702 performs feature fusion analysis on the M instance query vectors corresponding to each frame of the video image to obtain the M fused instance query vectors corresponding to each frame of the video image, it can be specifically used to:
[0169] Performing intra-frame feature fusion on the M instance query vectors corresponding to each frame of the video image to obtain the M intra-frame instance query vectors corresponding to each frame of the video image;
[0170] Inter-frame feature fusion is performed on the M intra-frame instance query vectors corresponding to the N frames of video images to obtain M fused instance query vectors corresponding to each frame of video image.
[0171] In one embodiment, when the processing unit 702 performs inter-frame feature fusion on the M intra-frame instance query vectors corresponding to the N frames of video images to obtain the M fused instance query vectors corresponding to each frame of video images, it can be specifically used to:
[0172] Obtaining an i-th intra-frame instance query vector among the M intra-frame instance query vectors corresponding to each frame of the video image, where i is greater than or equal to 1 and i is less than or equal to M;
[0173] Based on the i-th intra-frame instance query vector corresponding to the N frames of video images, inter-frame feature fusion is performed on the i-th intra-frame instance query vector corresponding to each frame of video image to obtain the i-th fused instance query vector corresponding to each frame of video image.
[0174] In one embodiment, when the number of image fusion features of each frame of video image is L, the processing unit 702 performs instance segmentation on the N frames of video image based on the image fusion features of each frame of video image and the corresponding M fusion instance query vectors to obtain the instances included in each frame of video image, which can be specifically used to:
[0175] Selecting a target image fusion feature from L image fusion features of a target video image, wherein the target video image is any frame of the N frames of video images;
[0176] Dynamically convolve the target image fusion feature and the M fusion instance query vectors corresponding to the target video image to obtain the instance included in the target video image.
[0177] In one embodiment, when the number of image fusion features of each frame of video image is L, the processing unit 702 performs instance segmentation on the N frames of video image based on the image fusion features of each frame of video image and the corresponding M fusion instance query vectors to obtain the instances included in each frame of video image, which can be specifically used to:
[0178] Selecting a target image fusion feature from L image fusion features of a target video image, wherein the target video image is any frame of the N frames of video images;
[0179] Perform S dynamic convolutions based on the target image fusion feature and the M fusion instance query vectors corresponding to the target video image to obtain the M updated fusion instance query vectors corresponding to the target video image after the S dynamic convolutions and the candidate instances included in the target video image; obtain the M updated fusion instance query vectors and the candidate instances included in the target video image after one dynamic convolution process; the M updated fusion instance query vectors obtained by the y-th dynamic convolution are obtained by performing dynamic convolution based on the y-1-th fusion instance query vector and the target image fusion feature, and the y-1-th fusion instance query vector is obtained by performing feature fusion analysis on the M updated fusion instance query vectors obtained by the y-1-th dynamic convolution; y is greater than 1, and y is less than or equal to S;
[0180] The candidate instance included in the target video image obtained by the S-th dynamic convolution is determined as the instance included in the target video image.
[0181] In one embodiment, the instances included in each frame of the video image are obtained by calling an instance segmentation model to perform instance segmentation on each frame of the video image, and the acquisition unit 701 is further used to obtain R frames of sample video images sampled from the sample video and instance annotations corresponding to each frame of the sample video image, where R is an integer greater than 1;
[0182] The processing unit 702 is further used to call the initial instance segmentation model, obtain the sample image features of each frame of the sample video image and the sample time-domain memory features corresponding to each frame of the sample video image, wherein the sample time-domain memory features are used to indicate the sample memory image features of the sample video image in the time domain; perform time-domain shift processing on the sample time-domain memory features corresponding to each frame of the sample video image to obtain the sample memory shift features corresponding to each frame of the sample video image; perform instance segmentation on the R frame sample video image based on the sample image features and the corresponding sample memory shift features of each frame of the sample video image to obtain the predicted instances included in each frame of the sample video image; and update the model parameters of the initial instance segmentation model based on the predicted instances and the corresponding instance annotations included in each frame of the sample video image to obtain the instance segmentation model.
[0183] In one embodiment, when the processing unit 702 performs instance segmentation on the R frame sample video image based on the sample image features of each frame sample video image and the corresponding sample memory shift features to obtain the predicted instances included in each frame sample video image, it can be specifically used to:
[0184] Performing feature fusion on the sample image features of each frame of the sample video image and the corresponding sample memory shift features to obtain the sample image fusion features of each frame of the sample video image;
[0185] Obtain M sample instance query vectors, and broadcast the M sample instance query vectors to each frame of sample video image to obtain M sample instance query vectors corresponding to each frame of sample video image;
[0186] Performing feature fusion analysis on the M sample instance query vectors of each frame of the sample video image to obtain M fused sample instance query vectors corresponding to each frame of the sample video image;
[0187] Based on the sample image fusion features of each frame of the sample video image and the corresponding M fusion sample instance query vectors, instance segmentation is performed on the R frame video image to obtain the predicted instances included in each frame of the sample video image.
[0188] In one embodiment, when the number of the sample image fusion features is L, the processing unit 702 performs instance segmentation on the R frame video image based on the sample image fusion features of each frame of the sample video image and the corresponding M fusion sample instance query vectors to obtain the predicted instance included in each frame of the sample video image, which can be specifically used to:
[0189] Selecting a target sample image fusion feature from L sample image fusion features of a target sample video image, wherein the target sample video image is any frame of the R frame sample video images;
[0190] Based on the target sample image fusion feature and the M sample fusion instance query vectors corresponding to the target sample video image, S dynamic convolutions are performed to obtain M updated sample fusion instance query vectors corresponding to the target sample video image after the S dynamic convolutions and the sample candidate instances included in the target sample video image; after one dynamic convolution process, M updated sample fusion instance query vectors and the sample candidate instances included in the target sample video image are obtained; the M updated sample fusion instance query vectors obtained by the y-th dynamic convolution are obtained by dynamic convolution based on the y-1-th sample fusion instance query vector and the target sample image fusion feature, and the y-1-th sample fusion instance query vector is obtained by feature fusion analysis of the M updated sample fusion instance query vectors obtained by the y-1-th dynamic convolution; y is greater than 1, and y is less than or equal to S;
[0191] The sample candidate instances included in each frame of sample video image obtained after each dynamic convolution are all used as the prediction instances included in each frame of sample video image.
[0192] In one embodiment, when the processing unit 702 updates the model parameters of the initial instance segmentation model based on the predicted instance and the corresponding instance annotation included in each frame of the sample video image to obtain the instance segmentation model, it can be specifically used to:
[0193] Calculate the loss between the predicted instance and the corresponding instance annotation included in each frame of sample video image obtained after each dynamic convolution, and obtain the loss value corresponding to each frame of sample video image after each dynamic convolution;
[0194] Determine a model loss value based on the loss value corresponding to each frame of the sample video image obtained by the S dynamic convolutions;
[0195] The model parameters of the initial instance segmentation model are updated based on the model loss value to obtain an instance segmentation model.
[0196] It can be understood that the functions of the functional units of the example segmentation device of this embodiment can be based on the above method embodiment. Figure 2 , Figure 4 and Figure 6 The specific implementation process of the method in the embodiment can refer to the above method embodiment Figure 2 , Figure 4 and Figure 6 The relevant description in will not be repeated here.
[0197] Based on the description of the above-mentioned example segmentation method embodiment, the present application embodiment further discloses a computer device, see Figure 8 The computer device may at least include a processor 801, an input device 802, an output device 803, and a memory 804. The processor 801, the input device 802, the output device 803, and the memory 804 in the computer device may be connected via a bus or other means.
[0198] The memory 804 is a memory device in a computer device, which is used to store programs and data. It is understandable that the memory 804 here can include both the built-in storage medium of the computer device and the extended storage medium supported by the computer device. The memory 804 provides a storage space, which stores the operating system of the computer device. In addition, computer programs (including program codes) are also stored in the storage space. It should be noted that the computer storage medium here can be a high-speed RAM memory; optionally, it can also be at least one computer storage medium away from the aforementioned processor. The processor can be called a central processing unit (CPU), which is the core and control center of the computer device and is used to run the computer program stored in the memory 804.
[0199] In one embodiment, the processor 801 may load and execute a computer program stored in the memory 804 to implement the corresponding steps of the method in the above-mentioned example segmentation method embodiment; specifically, the processor 801 loads and executes the computer program stored in the memory 804 to:
[0200] Obtain N frames of video images in the target video, each frame of the video image includes one or more instances, and N is an integer greater than 1;
[0201] Acquire the image features of each frame of the video image and the time domain memory features corresponding to each frame of the video image, wherein the time domain memory features are used to indicate the memory image features of the video image in the time domain;
[0202] Performing time-domain shift processing on the time-domain memory feature corresponding to each frame of the video image to obtain the memory shift feature corresponding to each frame of the video image;
[0203] Based on the image features of each frame of the video image and the corresponding memory shift features, instance segmentation is performed on the N frames of video images to obtain the instances included in each frame of the video image.
[0204] In one embodiment, when performing time-domain shift processing on the time-domain memory feature corresponding to each frame of the video image to obtain the memory shift feature corresponding to each frame of the video image, the processor 801 loads and executes the computer program stored in the memory 804, specifically for:
[0205] Dividing the time-domain memory features corresponding to the N frames of video images into a plurality of memory feature groups, each memory feature group including the same number of time-domain memory features;
[0206] Obtaining the moving step length and moving direction of each memory feature group;
[0207] The time domain memory features in each memory feature group are moved according to the moving step length and moving direction of each memory feature group to obtain the memory shift features corresponding to each frame of the video image.
[0208] In one embodiment, when performing instance segmentation on the N frames of video images based on the image features of each frame of video images and the corresponding memory shift features to obtain the instances included in each frame of video images, the processor 801 loads and executes the computer program stored in the memory 804, specifically for:
[0209] Performing feature fusion on the image features of each frame of the video image and the corresponding memory shift features to obtain the image fusion features of each frame of the video image;
[0210] Based on the image fusion features of each frame of the video image, instance segmentation is performed on the N frames of video image to obtain instances included in each frame of the video image.
[0211] In one embodiment, the number of image fusion features of each frame of video image is L, where L is an integer greater than or equal to 1. When the image features of each frame of video image and the corresponding memory shift features are subjected to feature fusion to obtain the image fusion features of each frame of video image, the processor 801 loads and executes a computer program stored in the memory 804, specifically for:
[0212] Determining candidate image fusion features and corresponding memory shift fusion features of each frame of video image based on the image features of each frame of video image and the corresponding memory shift features;
[0213] Performing reverse time-domain shift processing on the memory shift fusion feature corresponding to each frame of the video image to obtain a reverse shift feature corresponding to each frame of the video image;
[0214] Counting the number of candidate image fusion features of each frame of video image;
[0215] If the number is less than L, then obtain new image features and corresponding new time-domain memory features of each frame of video image, and perform time-domain shift processing on the new time-domain memory features corresponding to each frame of video image to obtain new memory shift features corresponding to each frame of video image; perform feature fusion based on the new image features and corresponding new memory shift features of each frame of video image to obtain new candidate image fusion features of each frame of video image, wherein the new image features and corresponding new time-domain memory features of each frame of video image are determined based on the candidate image fusion features and corresponding reverse shift features of each frame of video image;
[0216] If the number is equal to L, the L candidate image fusion features of each frame of video image are determined as the L image fusion features of each frame of video image.
[0217] In one embodiment, when performing instance segmentation on the N frames of video images based on the image fusion features of each frame of video images to obtain the instances included in each frame of video images, the processor 801 loads and executes a computer program stored in the memory 804, specifically for:
[0218] Obtain M instance query vectors;
[0219] Broadcasting the M instance query vectors to N frames of video images to obtain M instance query vectors corresponding to each frame of video image;
[0220] Based on the image fusion features of each frame of the video image and the corresponding M instance query vectors, instance segmentation is performed on the N frames of the video image to obtain the instances included in each frame of the video image.
[0221] In one embodiment, when performing instance segmentation on the N frames of video images based on the image fusion features of each frame of video images and the corresponding M instance query vectors to obtain the instances included in each frame of video images, the processor 801 loads and executes the computer program stored in the memory 804, specifically for:
[0222] Performing feature fusion analysis on the M instance query vectors corresponding to each frame of the video image to obtain the M fused instance query vectors corresponding to each frame of the video image;
[0223] Based on the image fusion features of each frame of the video image and the corresponding M fusion instance query vectors, instance segmentation is performed on the N frames of video images to obtain instances included in each frame of the video image.
[0224] In one embodiment, when performing feature fusion analysis on the M instance query vectors corresponding to each frame of video image to obtain the M fused instance query vectors corresponding to each frame of video image, the processor 801 loads and executes the computer program stored in the memory 804, specifically for:
[0225] Performing intra-frame feature fusion on the M instance query vectors corresponding to each frame of the video image to obtain the M intra-frame instance query vectors corresponding to each frame of the video image;
[0226] Inter-frame feature fusion is performed on the M intra-frame instance query vectors corresponding to the N frames of video images to obtain M fused instance query vectors corresponding to each frame of video image.
[0227] In one embodiment, when performing inter-frame feature fusion on the M intra-frame instance query vectors corresponding to the N frames of video images to obtain the M fused instance query vectors corresponding to each frame of video images, the processor 801 loads and executes the computer program stored in the memory 804, specifically for:
[0228] Obtaining an i-th intra-frame instance query vector among the M intra-frame instance query vectors corresponding to each frame of the video image, where i is greater than or equal to 1 and i is less than or equal to M;
[0229] Based on the i-th intra-frame instance query vector corresponding to the N frames of video images, inter-frame feature fusion is performed on the i-th intra-frame instance query vector corresponding to each frame of video image to obtain the i-th fused instance query vector corresponding to each frame of video image.
[0230] In one embodiment, when the number of image fusion features of each frame of video image is L, when instance segmentation is performed on the N frames of video image based on the image fusion features of each frame of video image and the corresponding M fusion instance query vectors to obtain the instances included in each frame of video image, the processor 801 loads and executes the computer program stored in the memory 804, specifically for:
[0231] Selecting a target image fusion feature from L image fusion features of a target video image, wherein the target video image is any frame of the N frames of video images;
[0232] Dynamically convolve the target image fusion feature and the M fusion instance query vectors corresponding to the target video image to obtain the instance included in the target video image.
[0233] In one embodiment, when the number of image fusion features of each frame of video image is L, when instance segmentation is performed on the N frames of video image based on the image fusion features of each frame of video image and the corresponding M fusion instance query vectors to obtain the instances included in each frame of video image, the processor 801 loads and executes the computer program stored in the memory 804, specifically for:
[0234] Selecting a target image fusion feature from L image fusion features of a target video image, wherein the target video image is any frame of the N frames of video images;
[0235] Perform S dynamic convolutions based on the target image fusion feature and the M fusion instance query vectors corresponding to the target video image to obtain the M updated fusion instance query vectors corresponding to the target video image after the S dynamic convolutions and the candidate instances included in the target video image; obtain the M updated fusion instance query vectors and the candidate instances included in the target video image after one dynamic convolution process; the M updated fusion instance query vectors obtained by the y-th dynamic convolution are obtained by performing dynamic convolution based on the y-1-th fusion instance query vector and the target image fusion feature, and the y-1-th fusion instance query vector is obtained by performing feature fusion analysis on the M updated fusion instance query vectors obtained by the y-1-th dynamic convolution; y is greater than 1, and y is less than or equal to S;
[0236] The candidate instance included in the target video image obtained by the S-th dynamic convolution is determined as the instance included in the target video image.
[0237] In one embodiment, the instances included in each frame of video image are obtained by calling an instance segmentation model to perform instance segmentation on each frame of video image, and the processor 801 loads and executes a computer program stored in the memory 804, and is further used to:
[0238] Obtain R frames of sample video images sampled from the sample video and instance annotations corresponding to each frame of the sample video image, where R is an integer greater than 1;
[0239] Calling the initial instance segmentation model to obtain the sample image features of each frame of the sample video image and the sample time-domain memory features corresponding to each frame of the sample video image, wherein the sample time-domain memory features are used to indicate the sample memory image features of the sample video image in the time domain;
[0240] Performing time-domain shift processing on the sample time-domain memory features corresponding to each frame of the sample video image to obtain the sample memory shift features corresponding to each frame of the sample video image;
[0241] Based on the sample image features of each frame of the sample video image and the corresponding sample memory shift features, instance segmentation is performed on the R frame of the sample video image to obtain the predicted instances included in each frame of the sample video image;
[0242] Based on the predicted instances and corresponding instance annotations included in each frame of the sample video image, the model parameters of the initial instance segmentation model are updated to obtain an instance segmentation model.
[0243] In one embodiment, when performing instance segmentation on the R frame sample video image based on the sample image features of each frame sample video image and the corresponding sample memory shift features to obtain the predicted instances included in each frame sample video image, the processor 801 loads and executes the computer program stored in the memory 804, specifically for:
[0244] Performing feature fusion on the sample image features of each frame of the sample video image and the corresponding sample memory shift features to obtain the sample image fusion features of each frame of the sample video image;
[0245] Obtain M sample instance query vectors, and broadcast the M sample instance query vectors to each frame of sample video image to obtain M sample instance query vectors corresponding to each frame of sample video image;
[0246] Performing feature fusion analysis on the M sample instance query vectors of each frame of the sample video image to obtain M fused sample instance query vectors corresponding to each frame of the sample video image;
[0247] Based on the sample image fusion features of each frame of the sample video image and the corresponding M fusion sample instance query vectors, instance segmentation is performed on the R frame video image to obtain the predicted instances included in each frame of the sample video image.
[0248] In one embodiment, when the number of the sample image fusion features is L, when instance segmentation is performed on the R frame video image based on the sample image fusion features of each frame of the sample video image and the corresponding M fusion sample instance query vectors to obtain the predicted instance included in each frame of the sample video image, the processor 801 loads and executes the computer program stored in the memory 804, specifically for:
[0249] Selecting a target sample image fusion feature from L sample image fusion features of a target sample video image, wherein the target sample video image is any frame of the R frame sample video images;
[0250] Based on the target sample image fusion feature and the M sample fusion instance query vectors corresponding to the target sample video image, S dynamic convolutions are performed to obtain M updated sample fusion instance query vectors corresponding to the target sample video image after the S dynamic convolutions and the sample candidate instances included in the target sample video image; after one dynamic convolution process, M updated sample fusion instance query vectors and the sample candidate instances included in the target sample video image are obtained; the M updated sample fusion instance query vectors obtained by the y-th dynamic convolution are obtained by dynamic convolution based on the y-1-th sample fusion instance query vector and the target sample image fusion feature, and the y-1-th sample fusion instance query vector is obtained by feature fusion analysis of the M updated sample fusion instance query vectors obtained by the y-1-th dynamic convolution; y is greater than 1, and y is less than or equal to S;
[0251] The sample candidate instances included in each frame of sample video image obtained after each dynamic convolution are all used as the prediction instances included in each frame of sample video image.
[0252] In one embodiment, when updating the model parameters of the initial instance segmentation model based on the predicted instance and the corresponding instance annotation included in each frame of the sample video image to obtain the instance segmentation model, the processor 801 loads and executes the computer program stored in the memory 804, specifically for:
[0253] Calculate the loss between the predicted instance and the corresponding instance annotation included in each frame of sample video image obtained after each dynamic convolution, and obtain the loss value corresponding to each frame of sample video image after each dynamic convolution;
[0254] Determine a model loss value based on the loss value corresponding to each frame of the sample video image obtained by the S dynamic convolutions;
[0255] The model parameters of the initial instance segmentation model are updated based on the model loss value to obtain an instance segmentation model.
[0256] It should be understood that in the embodiment of the present application, the processor 801 may be a central processing unit (CPU), and the processor 801 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.
[0257] In an embodiment of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the steps performed in all the above embodiments can be executed.
[0258] An embodiment of the present application also provides a computer program product, which includes computer instructions. The computer instructions are stored in a computer-readable storage medium. When the computer instructions are executed by a processor of a computer device, the methods in all the above embodiments are executed.
[0259] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.
[0260] What is disclosed above is only a preferred embodiment of the present application, and it certainly cannot be used to limit the scope of rights of the present application. Ordinary technicians in this field can understand that all or part of the processes of the above embodiment and equivalent changes made according to the claims of the present application are still within the scope of the invention.
Claims
1. An instance segmentation method, characterized in that: include: Obtain N frames of video images in the target video, each frame of the video image includes one or more instances, and N is an integer greater than 1; Acquire the image features of each frame of the video image and the time domain memory features corresponding to each frame of the video image, wherein the time domain memory features are used to indicate the memory image features of the video image in the time domain; Performing a time-domain shift processing on the time-domain memory features corresponding to each frame of the video image to obtain the memory shift features corresponding to each frame of the video image, including: dividing the time-domain memory features corresponding to the N frames of video images into a plurality of memory feature groups, each memory feature group including the same number of time-domain memory features; obtaining a moving step length and a moving direction of each memory feature group; moving the time-domain memory features in each memory feature group according to the moving step length and the moving direction of each memory feature group to obtain the memory shift features corresponding to each frame of the video image; Based on the image features of each frame of the video image and the corresponding memory shift features, instance segmentation is performed on the N frames of video images to obtain the instances included in each frame of the video image.
2. The method according to claim 1, characterized in that The performing instance segmentation on the N frames of video images based on the image features of each frame of video images and the corresponding memory shift features to obtain the instances included in each frame of video images comprises: Performing feature fusion on the image features of each frame of the video image and the corresponding memory shift features to obtain the image fusion features of each frame of the video image; Based on the image fusion features of each frame of the video image, instance segmentation is performed on the N frames of video image to obtain instances included in each frame of the video image.
3. The method according to claim 2, characterized in that The number of image fusion features of each frame of video image is L, where L is an integer greater than or equal to 1, and the image features of each frame of video image and the corresponding memory shift features are subjected to feature fusion to obtain the image fusion features of each frame of video image, including: Determining candidate image fusion features and corresponding memory shift fusion features of each frame of video image based on the image features of each frame of video image and the corresponding memory shift features; Performing reverse time-domain shift processing on the memory shift fusion feature corresponding to each frame of the video image to obtain a reverse shift feature corresponding to each frame of the video image; Counting the number of candidate image fusion features of each frame of video image; If the number is less than L, then obtain new image features and corresponding new time-domain memory features of each frame of video image, and perform time-domain shift processing on the new time-domain memory features corresponding to each frame of video image to obtain new memory shift features corresponding to each frame of video image; perform feature fusion based on the new image features and corresponding new memory shift features of each frame of video image to obtain new candidate image fusion features of each frame of video image, wherein the new image features and corresponding new time-domain memory features of each frame of video image are determined based on the candidate image fusion features and corresponding reverse shift features of each frame of video image; If the number is equal to L, the L candidate image fusion features of each frame of video image are determined as the L image fusion features of each frame of video image.
4. The method according to claim 2, characterized in that The performing instance segmentation on the N frames of video images based on the image fusion features of each frame of video images to obtain the instances included in each frame of video images includes: Obtain M instance query vectors; Broadcasting the M instance query vectors to N frames of video images to obtain M instance query vectors corresponding to each frame of video image; Based on the image fusion features of each frame of the video image and the corresponding M instance query vectors, instance segmentation is performed on the N frames of the video image to obtain the instances included in each frame of the video image.
5. The method according to claim 4, characterized in that The performing instance segmentation on the N frames of video images based on the image fusion features of each frame of video images and the corresponding M instance query vectors to obtain the instances included in each frame of video images includes: Performing feature fusion analysis on the M instance query vectors corresponding to each frame of the video image to obtain the M fused instance query vectors corresponding to each frame of the video image; Based on the image fusion features of each frame of the video image and the corresponding M fusion instance query vectors, instance segmentation is performed on the N frames of video images to obtain instances included in each frame of the video image.
6. The method according to claim 5, characterized in that The performing feature fusion analysis on the M instance query vectors corresponding to each frame of the video image to obtain the M fusion instance query vectors corresponding to each frame of the video image includes: Performing intra-frame feature fusion on the M instance query vectors corresponding to each frame of the video image to obtain the M intra-frame instance query vectors corresponding to each frame of the video image; Inter-frame feature fusion is performed on the M intra-frame instance query vectors corresponding to the N frames of video images to obtain M fused instance query vectors corresponding to each frame of video image.
7. The method according to claim 6, characterized in that The performing inter-frame feature fusion on the M intra-frame instance query vectors corresponding to the N frames of video images to obtain the M fused instance query vectors corresponding to each frame of video image includes: Obtaining an i-th intra-frame instance query vector among the M intra-frame instance query vectors corresponding to each frame of the video image, where i is greater than or equal to 1 and i is less than or equal to M; Based on the i-th intra-frame instance query vector corresponding to the N frames of video images, inter-frame feature fusion is performed on the i-th intra-frame instance query vector corresponding to each frame of video image to obtain the i-th fused instance query vector corresponding to each frame of video image.
8. The method according to claim 5, characterized in that When the number of image fusion features of each frame of video image is L, performing instance segmentation on the N frames of video image based on the image fusion features of each frame of video image and the corresponding M fusion instance query vectors to obtain the instances included in each frame of video image includes: Selecting a target image fusion feature from L image fusion features of a target video image, wherein the target video image is any frame of the N frames of video images; Dynamically convolve the target image fusion feature and the M fusion instance query vectors corresponding to the target video image to obtain the instance included in the target video image.
9. The method according to claim 5, characterized in that When the number of image fusion features of each frame of video image is L, performing instance segmentation on the N frames of video image based on the image fusion features of each frame of video image and the corresponding M fusion instance query vectors to obtain the instances included in each frame of video image includes: Selecting a target image fusion feature from L image fusion features of a target video image, wherein the target video image is any frame of the N frames of video images; Perform S dynamic convolutions based on the target image fusion feature and the M fusion instance query vectors corresponding to the target video image to obtain the M updated fusion instance query vectors corresponding to the target video image after the S dynamic convolutions and the candidate instances included in the target video image; obtain the M updated fusion instance query vectors and the candidate instances included in the target video image after one dynamic convolution process; the M updated fusion instance query vectors obtained by the y-th dynamic convolution are obtained by performing dynamic convolution based on the y-1-th fusion instance query vector and the target image fusion feature, and the y-1-th fusion instance query vector is obtained by performing feature fusion analysis on the M updated fusion instance query vectors obtained by the y-1-th dynamic convolution; y is greater than 1, and y is less than or equal to S; The candidate instance included in the target video image obtained by the S-th dynamic convolution is determined as the instance included in the target video image.
10. The method according to claim 1, characterized in that The instances included in each frame of the video image are obtained by calling the instance segmentation model to perform instance segmentation on each frame of the video image, and the method further includes: Obtain R frames of sample video images sampled from the sample video and instance annotations corresponding to each frame of the sample video image, where R is an integer greater than 1; Calling the initial instance segmentation model to obtain the sample image features of each frame of the sample video image and the sample time-domain memory features corresponding to each frame of the sample video image, wherein the sample time-domain memory features are used to indicate the sample memory image features of the sample video image in the time domain; Performing time-domain shift processing on the sample time-domain memory features corresponding to each frame of the sample video image to obtain the sample memory shift features corresponding to each frame of the sample video image; Based on the sample image features of each frame of the sample video image and the corresponding sample memory shift features, instance segmentation is performed on the R frame of the sample video image to obtain the predicted instances included in each frame of the sample video image; Based on the predicted instances and corresponding instance annotations included in each frame of the sample video image, the model parameters of the initial instance segmentation model are updated to obtain an instance segmentation model.
11. An instance segmentation device, characterized in that: include: An acquisition unit, used to acquire N frames of video images in a target video, each frame of the video image includes one or more instances, and N is an integer greater than 1; The acquisition unit is further used to acquire the image features of each frame of the video image and the time domain memory features corresponding to each frame of the video image, wherein the time domain memory features are used to indicate the memory image features of the video image in the time domain; A processing unit is used to perform time-domain shift processing on the time-domain memory features corresponding to each frame of video image to obtain the memory shift features corresponding to each frame of video image, including: dividing the time-domain memory features corresponding to the N frames of video image into a plurality of memory feature groups, each memory feature group including the same number of time-domain memory features; obtaining a moving step length and a moving direction of each memory feature group; and moving the time-domain memory features in each memory feature group according to the moving step length and the moving direction of each memory feature group to obtain the memory shift features corresponding to each frame of video image; The processing unit is further used to perform instance segmentation on the N frames of video images based on the image features of each frame of video images and the corresponding memory shift features to obtain the instances included in each frame of video images.
12. A computer device, characterized in that: include: Memory for storing computer programs; A processor is configured to call the computer program in the memory to execute the instance segmentation method according to any one of claims 1 to 10.
13. A computer-readable storage medium, characterized in that: The computer storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the instance segmentation method according to any one of claims 1 to 10 is executed.
14. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the computer program is used to load and execute the instance segmentation method according to any one of claims 1 to 10.
