Video image processing method, device, electronic device and storage medium

Abstract

The present disclosure relates to a video image processing method, apparatus, electronic device, and storage medium. The method includes: obtaining a mask of a first image frame and a second image frame of a video, wherein the position of the second image frame in the video is located after the position of the first image frame in the video; obtaining a first multi-scale feature of the first image frame; the first multi-scale feature of the first image frame is obtained by pre-processing; generating first query information and first mask information based on the first multi-scale feature and the mask of the first image frame; generating second query information and second mask information based on the second image frame; and determining a mask of the second image frame based on the first query information, the second query information, the first mask information, and the second mask information. The present application eliminates the need to determine the multi-scale features of the first image frame a second time, and only requires the mask of the first image frame as input for feature extraction, thereby saving the cost of loading a mask recognition model on the device for image recognition.

Description

Technical Field

[0001] The present disclosure relates to the field of Internet technology, and in particular to a video image processing method, device, electronic device, and storage medium. Background Art

[0002] Video Object Segmentation (VOS) is a fundamental capability for video scene understanding and video editing. This technology has broad application prospects in areas such as short video intelligent editing, special effects production, and short video creation. VOS technology refers to the process of predicting the pixel-level segmentation mask of a target object in subsequent frames given a target object mask in the initial frame of a video sequence. With the development of deep learning, deep neural networks have been applied to VOS. The high-level semantic features extracted from deep networks can more accurately distinguish target objects and backgrounds in complex scenes, thereby greatly improving the effect of target segmentation.

[0003] However, although deep neural networks bring more accurate target segmentation methods, the complex structure of deep neural networks usually requires more memory overhead for devices loading the network, which increases the burden on the devices. Summary of the Invention

[0004] The present disclosure provides a video image processing method, device, electronic device, and storage medium. The technical solutions of the present disclosure are as follows:

[0005] According to a first aspect of an embodiment of the present disclosure, a video image processing method is provided, including:

[0006] Obtaining a mask and a second image frame of a first image frame of a video, wherein a position of the second image frame in the video is located after a position of the first image frame in the video;

[0007] Acquire a first multi-scale feature of a first image frame; the first multi-scale feature of the first image frame is obtained by pre-processing;

[0008] generating first query information and first mask information based on the first multi-scale features and the mask of the first image frame;

[0009] generating second query information and second mask information based on the second image frame;

[0010] A mask for the second image frame is determined according to the first query information, the second query information, the first mask information, and the second mask information.

[0011] In some possible embodiments, the first multi-scale feature of the first image frame is pre-processed by a query encoder, and a main network of the query encoder includes a plurality of modules connected sequentially;

[0012] Before obtaining the first multi-scale features of the first image frame, the method further includes:

[0013] Extracting feature information from the first image frame based on the multiple modules to obtain first feature information corresponding to each module in the multiple modules;

[0014] Determine a first multi-scale feature of the first image frame based on the first feature information corresponding to each module;

[0015] The first multi-scale features of the first image frame are stored in the memory area.

[0016] In some possible embodiments, generating first query information and first mask information based on the first multi-scale features and the mask of the first image frame includes:

[0017] The first convolutional layer in the sequentially connected plurality of convolutional layers included in the main network of the memory encoder is used as the current convolutional layer; the mask of the first image frame is used as the current input information; the number of the plurality of convolutional layers is the same as the number of the plurality of modules included in the main network of the query encoder, and the number of the plurality of convolutional layers is one-to-one corresponding to the number of the modules;

[0018] Input the current input information into the current convolution layer to obtain the current feature information to be fused;

[0019] Performing a sum operation on the current feature information to be fused and the first feature information output by the module corresponding to the current convolutional layer to obtain fused feature information;

[0020] Use the next convolution layer of the current convolution layer to update the current convolution layer, and use the fused feature information to update the current input information; repeat the following steps: input the current input information into the current convolution layer to obtain the current feature information to be fused; add the current feature information to be fused and the first feature information output by the module corresponding to the current convolution layer to obtain the fused feature information;

[0021] Until the fusion feature information corresponding to the last convolution layer in multiple convolution layers is obtained;

[0022] Based on the fused feature information corresponding to the last convolutional layer, first query information and first mask information are generated.

[0023] In some possible embodiments, the query encoder further includes a query convolution layer and a mask convolution layer located after the main network of the query encoder; the query convolution layer and the mask convolution layer are connected in parallel;

[0024] Generating second query information and second mask information based on the second image frame includes:

[0025] Extracting feature information from the second image frame based on the multiple modules to obtain second feature information corresponding to each module in the multiple modules;

[0026] Extracting second feature information corresponding to a last module among the multiple modules based on the query convolution layer to obtain second query information of the second image frame;

[0027] Based on the mask convolution layer, second feature information corresponding to the last module in the multiple modules is extracted to obtain second mask information of the second image frame.

[0028] In some possible embodiments, the method further includes:

[0029] Determining second feature information of the second image frame corresponding to each module in the plurality of modules as a first multi-scale feature of the second image frame;

[0030] storing the first multi-scale features of the second image frame in a memory area; and applying the first multi-scale features of the second image frame to determine a mask of the third image frame;

[0031] The position of the third image frame in the video is located after the position of the second image frame in the video.

[0032] In some possible embodiments, determining the mask of the second image frame according to the first query information, the second query information, the first mask information, and the second mask information includes:

[0033] determining frame similarity information based on the first query information and the second query information;

[0034] determining spatiotemporal memory information based on the frame similarity information and the first mask information;

[0035] determining frame splicing information based on the spatiotemporal memory information and the second mask information;

[0036] The frame splicing information is input into the decoder to obtain the mask of the second image frame.

[0037] In some possible embodiments, obtaining a mask of a first image frame and a second image frame of a video, and obtaining a first multi-scale feature of the first image frame includes:

[0038] Acquire multiple masks and second image frames corresponding one to one to multiple first image frames of the video;

[0039] Acquire a first multi-scale feature of each first image frame in a plurality of first image frames;

[0040] Generating first query information and first mask information based on first multi-scale features and a mask of a first image frame includes:

[0041] determining query information and mask information for each first image frame based on the mask and the first multi-scale features of each first image frame;

[0042] First query information and first mask information corresponding to the plurality of first image frames are determined based on the fusion of query information and mask information of each first image frame in the plurality of first image frames.

[0043] According to a second aspect of an embodiment of the present disclosure, there is provided a video image processing apparatus, including:

[0044] A first acquisition module is configured to acquire a mask and a second image frame of a first image frame of a video, wherein a position of the second image frame in the video is located after a position of the first image frame in the video;

[0045] A second acquisition module is configured to acquire a first multi-scale feature of the first image frame; the first multi-scale feature of the first image frame is obtained by pre-processing;

[0046] A first generating module is configured to generate first query information and first mask information based on first multi-scale features and masks of the first image frame;

[0047] A second generating module is configured to generate second query information and second mask information based on the second image frame;

[0048] The mask determination module is configured to determine a mask of the second image frame according to the first query information, the second query information, the first mask information, and the second mask information.

[0049] In some possible embodiments, the first multi-scale feature of the first image frame is pre-processed by a query encoder, and a main network of the query encoder includes a plurality of modules connected sequentially;

[0050] Before obtaining the first multi-scale feature of the first image frame, the method further includes a first multi-scale feature processing module configured to execute:

[0051] Extracting feature information from the first image frame based on the multiple modules to obtain first feature information corresponding to each module in the multiple modules;

[0052] Determine a first multi-scale feature of the first image frame based on the first feature information corresponding to each module;

[0053] The first multi-scale features of the first image frame are stored in the memory area.

[0054] In some possible embodiments, the first generating module is configured to execute:

[0055] The first convolutional layer in the sequentially connected plurality of convolutional layers included in the main network of the memory encoder is used as the current convolutional layer; the mask of the first image frame is used as the current input information; the number of the plurality of convolutional layers is the same as the number of the plurality of modules included in the main network of the query encoder, and the number of the plurality of convolutional layers is one-to-one corresponding to the number of the modules;

[0056] Input the current input information into the current convolution layer to obtain the current feature information to be fused;

[0057] Performing a sum operation on the current feature information to be fused and the first feature information output by the module corresponding to the current convolutional layer to obtain fused feature information;

[0058] Use the next convolution layer of the current convolution layer to update the current convolution layer, and use the fused feature information to update the current input information; repeat the following steps: input the current input information into the current convolution layer to obtain the current feature information to be fused; add the current feature information to be fused and the first feature information output by the module corresponding to the current convolution layer to obtain the fused feature information;

[0059] Until the fusion feature information corresponding to the last convolution layer in multiple convolution layers is obtained;

[0060] Based on the fused feature information corresponding to the last convolutional layer, first query information and first mask information are generated.

[0061] In some possible embodiments, the query encoder further includes a query convolution layer and a mask convolution layer located after the main network of the query encoder; the query convolution layer and the mask convolution layer are connected in parallel;

[0062] The second generation module is configured to execute:

[0063] Extracting feature information from the second image frame based on the multiple modules to obtain second feature information corresponding to each module in the multiple modules;

[0064] Extracting second feature information corresponding to a last module among the multiple modules based on the query convolution layer to obtain second query information of the second image frame;

[0065] Based on the mask convolution layer, second feature information corresponding to the last module in the multiple modules is extracted to obtain second mask information of the second image frame.

[0066] In some possible embodiments, the apparatus further includes a second multi-scale feature processing module configured to execute:

[0067] Determining second feature information of the second image frame corresponding to each module in the plurality of modules as a first multi-scale feature of the second image frame;

[0068] storing the first multi-scale features of the second image frame in a memory area; and applying the first multi-scale features of the second image frame to determine a mask of the third image frame;

[0069] The position of the third image frame in the video is located after the position of the second image frame in the video.

[0070] In some possible embodiments, the mask generation module is configured to execute:

[0071] determining frame similarity information based on the first query information and the second query information;

[0072] determining spatiotemporal memory information based on the frame similarity information and the first mask information;

[0073] determining frame splicing information based on the spatiotemporal memory information and the second mask information;

[0074] The frame splicing information is input into the decoder to obtain the mask of the second image frame.

[0075] In some possible embodiments, the first acquisition module is configured to execute:

[0076] Acquire multiple masks and second image frames corresponding one to one to multiple first image frames of the video;

[0077] Acquire a first multi-scale feature of each first image frame in a plurality of first image frames;

[0078] Generating first query information and first mask information based on first multi-scale features and a mask of a first image frame includes:

[0079] determining query information and mask information for each first image frame based on the mask and the first multi-scale features of each first image frame;

[0080] First query information and first mask information corresponding to the plurality of first image frames are determined based on the fusion of query information and mask information of each first image frame in the plurality of first image frames.

[0081] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute instructions to implement a method as described in any one of the first aspects above.

[0082] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute any one of the methods in the first aspect of the embodiment of the present disclosure.

[0083] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, which includes a computer program, the computer program is stored in a readable storage medium, and at least one processor of a computer device reads and executes the computer program from the readable storage medium, so that the computer device performs any one of the methods in the first aspect of the embodiment of the present disclosure.

[0084] The technical solutions provided by the embodiments of the present disclosure bring at least the following beneficial effects:

[0085] Obtain a mask of a first image frame and a second image frame of a video, wherein the position of the second image frame in the video is located after the position of the first image frame in the video; obtain a first multi-scale feature of the first image frame; the first multi-scale feature of the first image frame is obtained by pre-processing; based on the first multi-scale feature and the mask of the first image frame, generate first query information and first mask information; based on the second image frame, generate second query information and second mask information; determine the mask of the second image frame based on the first query information, the second query information, the first mask information, and the second mask information. This application does not require a second determination of the multi-scale features of the first image frame, and only requires the mask of the first image frame as input for feature extraction, thereby saving the cost of loading a mask recognition model on the device for image recognition.

[0086] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0087] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0088] Figure 1 is a schematic diagram showing an application environment according to an exemplary embodiment;

[0089] Figure 2 is a flowchart of a video image processing method according to an exemplary embodiment;

[0090] Figure 3 is a schematic diagram showing an image and its corresponding mask according to an exemplary embodiment;

[0091] Figure 4 is a flow chart showing a method for applying a mask recognition model according to an exemplary embodiment;

[0092] Figure 5 is a schematic diagram of a query encoder according to an exemplary embodiment;

[0093] Figure 6 is a flowchart of a method for determining a first multi-scale feature of a first image frame according to an exemplary embodiment;

[0094] Figure 7is a schematic diagram of a memory encoder according to an exemplary embodiment;

[0095] Figure 8 is a flowchart illustrating a method for determining first query information and first mask information of a first image frame according to an exemplary embodiment;

[0096] Figure 9 is a flowchart illustrating a method for determining second query information and second mask information of a second image frame according to an exemplary embodiment;

[0097] Figure 10 is a flowchart of a method for determining a mask of a second image frame according to an exemplary embodiment;

[0098] Figure 11 is a flowchart of obtaining a mask of a first image frame according to an exemplary embodiment;

[0099] Figure 12 This is a flow chart showing a method of determining a mask corresponding to an object according to an exemplary embodiment;

[0100] Figure 13 is a schematic diagram showing a first image frame according to an exemplary embodiment;

[0101] Figure 14 is a schematic diagram of a mask of a first image frame according to an exemplary embodiment. Figure 1 ;

[0102] Figure 15 is a schematic diagram of a mask of a first image frame according to an exemplary embodiment. Figure 2 ;

[0103] Figure 16 is a schematic diagram of a mask of a first image frame according to an exemplary embodiment. Figure 3 ;

[0104] Figure 17 is a block diagram of a video image processing device according to an exemplary embodiment;

[0105] Figure 18 The present invention is a block diagram of an electronic device for video image processing or video object recognition according to an exemplary embodiment. DETAILED DESCRIPTION

[0106] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0107] It should be noted that the terms "first," "second," and the like in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar first objects and are not necessarily used to describe a specific order or precedence. It should be understood that the numbers used in this manner are interchangeable where appropriate so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with certain aspects of the present disclosure as detailed in the appended claims.

[0108] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0109] See also Figure 1 , Figure 1 is a schematic diagram showing an application environment of a video image processing method according to an exemplary embodiment. Figure 1 As shown, the application environment may include a server 01 and a client 02 .

[0110] In some possible embodiments, server 01 may include an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud audio recognition model training, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The operating system running on the server may include, but is not limited to, Android, iOS, Linux, Windows, Unix, etc.

[0111] In some possible embodiments, the client 02 may include, but is not limited to, smartphones, desktop computers, tablet computers, laptop computers, smart speakers, digital assistants, augmented reality (AR) / virtual reality (VR) devices, smart wearable devices, and the like. It may also be software running on the client, such as an application or applet. Optionally, the operating system running on the client may include, but is not limited to, Android, iOS, Linux, Windows, Unix, and the like.

[0112] In some possible embodiments, the server 01 or the client 02 may obtain a mask of a first image frame and a second image frame of a video, where the position of the second image frame in the video is located after the position of the first image frame in the video; obtain a first multi-scale feature of the first image frame; the first multi-scale feature of the first image frame is obtained by pre-processing; based on the first multi-scale feature and the mask of the first image frame, generate first query information and first mask information; based on the second image frame, generate second query information and second mask information; determine the mask of the second image frame according to the first query information, the second query information, the first mask information, and the second mask information.

[0113] In some possible embodiments, the client 02 may be connected to the server 01 via a wired link or a wireless link.

[0114] In one exemplary embodiment, the client, server, and corresponding database can all be node devices in a blockchain system, capable of sharing acquired and generated information with other node devices in the blockchain system, enabling information sharing among multiple node devices. Multiple node devices in a blockchain system can be configured with the same blockchain, which is composed of multiple blocks, with adjacent blocks having an associated relationship. This allows any tampering of data in any block to be detected by the next block, thereby preventing tampering of the blockchain data and ensuring the security and reliability of the blockchain data.

[0115] Figure 2 is a flow chart of a video image processing method according to an exemplary embodiment. Figure 2 As shown, the video image processing method can be applied to a server or other node devices, such as a client. The following is an example of a server. The method includes the following steps:

[0116] In step S201, a mask of a first image frame and a second image frame of a video are obtained, where the second image frame is located after the first image frame in the video. In the embodiment of the present application, the above-mentioned video can be a video received by the server from a video capture device and needs to be processed.

[0117] In an embodiment of the present application, the server may obtain a mask of a first image frame and a second image frame of a video, wherein the second image frame is located after the first image frame in the video.

[0118] In some possible embodiments, the number of first image frames and the number of second image frames can both be one. For example, the first image frame can be the second image frame in the video, and the second image frame can be the fifth image frame in the video, that is, the first image frame and the second image frame can be two discontinuous image frames in the video. Alternatively, the first image frame can be the fourth image frame in the video, and the second image frame can be the fifth image frame in the video, that is, the first image frame and the second image frame can be two continuous image frames in the video. The purpose of using two continuous image frames in the present application is to continuously propagate the subsequent image frame using the mask of the previous image frame, which can solve the problem of large changes in objects in the image frames.

[0119] In other possible embodiments, there may be multiple first image frames and one second image frame. For example, the first image frame may be the second and fourth image frames in a video, and the second image frame may be the fifth image frame in the video. The multiple first image frames, one of which is the previous image frame of the second image frame, have the following effect: using the second image frame and the previous image frame (the fourth image frame in this example) not only addresses the issue of large object variations in the image frames but also increases the robustness of the model.

[0120] When there are multiple first image frames, the server may obtain multiple masks and second image frames that correspond one to one to the multiple first image frames in the video, wherein the position of the second image frame in the video is located after the position of each of the multiple first image frames in the video.

[0121] In the embodiments of the present application, masking is an image operation used to partially or completely hide an object or element in an image. Applying a mask to an object in an image has the effect of painting the image object onto a background, thereby completely or partially obscuring a portion or portions of the object in the image, while the portion of the image within the mask remains unchanged.

[0122] Figure 3 is a schematic diagram showing an image and its corresponding mask according to an exemplary embodiment. Figure 3The upper figure in the figure shows a normal image, and the lower figure shows the mask of the normal image.

[0123] In step S203 , a first multi-scale feature of the first image frame is obtained; the first multi-scale feature of the first image frame is obtained through pre-processing.

[0124] In an optional embodiment, the first multi-scale feature of the first image frame is pre-processed by the query encoder. Specifically, the first multi-scale feature of the first image frame may be pre-processed by the query encoder at a historical moment.

[0125] Optionally, the query encoder is part of the mask recognition model. In some possible embodiments, the mask recognition model is a model based on machine learning. Machine Learning (ML) is a multi-disciplinary interdisciplinary subject involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their own performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning generally include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0126] In the embodiment of the present application, the first multi-scale feature of the first image frame is a set of features including multiple scales. Optionally, the server can obtain the first multi-scale feature of the first image frame from a memory area corresponding to the mask recognition model.

[0127] Figure 4 FIG. 1 is a flowchart of a method for applying a mask recognition model according to an exemplary embodiment. The mask recognition model includes a memory encoder, a query encoder, and a decoder. The server can store the first image frame ( Figure 4 The mask of the image frame 2 in the video shown in FIG and the first multi-scale feature are input into the memory encoder to obtain the first query information Key and the first mask information Value, and the second image frame ( Figure 4 The image frame shown in 5) is input into the query encoder to obtain the second query information Key and the second mask information Value, and the first query information, the first mask information, the second query information and the second mask information are input into the decoder to obtain the mask of the second image frame.

[0128] Optionally, the embodiment of the present application may determine a first multi-scale feature of the first image frame. In the embodiment of the present application, the first multi-scale feature of the first image frame may be pre-processed by a main network in a query encoder in a mask recognition model at a historical moment.

[0129] Optionally, the first multi-scale features of the first image frame are a set of features at multiple scales. The number of modules in the main network is the same as the number of features at multiple scales. For example, if the first multi-scale features include features at four scales corresponding to the first image frame, the number of modules in the main network can be set to four.

[0130] Figure 5 is a schematic diagram of a query encoder according to an exemplary embodiment. Figure 5 As shown in FIG, the query encoder includes a main network and two parallel convolutional layers located after the main network. The main network of the query encoder includes multiple modules connected in sequence, such as Figure 5 The first module, the second module, the third module and the fourth module are shown. Optionally, each of the multiple modules may include at least one convolutional layer.

[0131] Figure 6 This is a flowchart of a method for determining a first multi-scale feature of a first image frame according to an exemplary embodiment, including:

[0132] In step S601 , feature information of a first image frame is extracted based on multiple modules to obtain first feature information corresponding to each module in the multiple modules.

[0133] The following will be combined Figure 5 How to obtain the first multi-scale features of the first image frame is described.

[0134] In the implementation of this application, the server can input the first image frame into the first module in the main network to obtain the features of the first scale, that is, the first feature information corresponding to the first module.

[0135] Optionally, the first image frame can be a three-primary color RGB image X∈R H×W×3 The resolution of the first image frame is H×W, where H is the height of the resolution of the first image frame, W is the width of the resolution of the first image frame, and the number of channels of the first image frame is 3. For example, the resolution of the first image frame may be 224*224, or the resolution of the first image frame may be 640*360. The server may resize the first image frame to obtain a first image frame with a resolution of 224*224.

[0136] In the embodiment of the present application, the first module may include a convolution layer, and the server may input the first image frame into the convolution layer to obtain a feature of the first scale with a resolution of 1 / 2 and an output channel of 64. Among them, c1=64.

[0137] In the embodiment of the present application, the second module may include multiple convolutional layers. The server may input the features of the first scale into the second module to obtain the features of the second scale with a resolution of 1 / 4 and an output channel of 256, that is, the first feature information corresponding to the second module. Among them, c2=256.

[0138] In the embodiment of the present application, the third module may include multiple convolutional layers. The server may input the second scale feature into the third module to obtain the third scale feature with a resolution of 1 / 8 and an output channel of 512, that is, the first feature information corresponding to the third module. Among them, c3=512.

[0139] In the embodiment of the present application, the fourth module may include multiple convolutional layers. The server may input the features of the third scale into the fourth module to obtain the features of the fourth scale with a resolution of 1 / 16 and an output channel of 1024, that is, the first feature information corresponding to the fourth module. Among them, c4=1024.

[0140] In step S603 , a first multi-scale feature of the first image frame is determined based on the first feature information corresponding to each module.

[0141] Optionally, the server may determine the first scale feature, the second scale feature, the third scale feature, and the fourth scale feature as first multi-scale features of the first image frame, so as to input them into the memory encoder when determining the mask of the second image frame.

[0142] In step S605, the first multi-scale features of the first image frame are stored in a memory area. In this embodiment of the present application, the first multi-scale features of the first image frame are pre-processed by a query encoder in the mask recognition model at a historical moment and stored in a corresponding memory area of ​​the mask recognition model for use in determining the mask of the second image frame at the current moment.

[0143] Optionally, the first image frame may be multiple RGB images. For example, if the multiple first image frames are the second image frame and the fourth image frame in a video, the server may determine the first multi-scale features of the second image frame and the first multi-scale features of the fourth image frame based on the first multi-scale features obtained above.

[0144] Specifically, the server may input the second image frame into the first module to obtain features at a first scale corresponding to the second image frame, input the features at the first scale corresponding to the second image frame into the second module to obtain features at a second scale corresponding to the second image frame, input the features at the second scale corresponding to the second image frame into the third module to obtain features at a third scale corresponding to the second image frame, and input the features at the third scale corresponding to the second image frame into the fourth module to obtain features at a fourth scale corresponding to the second image frame. The server may then determine the features at the first scale corresponding to the second image frame, the features at the second scale corresponding to the second image frame, the features at the third scale corresponding to the second image frame, and the features at the fourth scale corresponding to the second image frame as first multi-scale features of the second image frame.

[0145] The server may input the fourth image frame into the first module to obtain features at a first scale corresponding to the fourth image frame, input the features at the first scale corresponding to the fourth image frame into the second module to obtain features at a second scale corresponding to the fourth image frame, input the features at the second scale corresponding to the fourth image frame into the third module to obtain features at a third scale corresponding to the fourth image frame, and input the features at the third scale corresponding to the fourth image frame into the fourth module to obtain features at a fourth scale corresponding to the fourth image frame. The server may then determine the features at the first scale corresponding to the fourth image frame, the features at the second scale corresponding to the fourth image frame, the features at the third scale corresponding to the fourth image frame, and the features at the fourth scale corresponding to the fourth image frame as first multi-scale features of the fourth image frame.

[0146] However, deep learning networks can suffer from vanishing and exploding gradients due to their depth. Optionally, a batch normalization (BN) layer can be added after each module in the encoder's main network. Furthermore, to ensure that each layer in each module of the encoder's main network undergoes a nonlinear transformation at output, an activation function can be added after each layer.

[0147] Optionally, the network structure of the main network of the query encoder can refer to the network structure of Resnet50, which will not be repeated here.

[0148] In step S205 , first query information and first mask information are generated based on the first multi-scale features and the mask of the first image frame.

[0149] In an embodiment of the present application, the server can obtain the first multi-scale feature of the first image frame from the memory area corresponding to the mask recognition model, and determine the first query information and first mask information of the first image frame based on the memory encoder in the mask recognition model, the mask of the first image frame and the first multi-scale feature.

[0150] Figure 7 is a schematic diagram of a memory encoder according to an exemplary embodiment. Figure 7 As shown, the memory encoder includes a main network and a convolutional layer located after the main network, and two convolutional layers connected in parallel. The main network of the memory encoder includes multiple convolutional layers connected in sequence, such as Figure 7 The first convolutional layer, the second convolutional layer, the third convolutional layer, and the fourth convolutional layer are shown. The number of the multiple convolutional layers is the same as the number of the multiple modules included in the main network of the query encoder, and they correspond one to one.

[0151] Figure 8 This is a flowchart of a method for determining first query information and first mask information of a first image frame according to an exemplary embodiment, including:

[0152] In step S801, the first convolutional layer in the sequentially connected multiple convolutional layers included in the main network of the memory encoder is used as the current convolutional layer; the mask of the first image frame is used as the current input information; the number of the multiple convolutional layers is the same as the number of the multiple modules included in the main network of the query encoder, and they correspond one to one.

[0153] The following combination Figure 7 How to determine the first query information and the first mask information of the first image frame is described.

[0154] According to the content in step S801, the server can use the first convolution layer of the multiple convolution layers connected sequentially in the main network of the memory encoder as the current convolution layer, and use the mask of the first image frame as the current input information. Figure 5 and Figure 7 It can be seen that the number of multiple convolutional layers is the same as the number of multiple modules contained in the main network of the query encoder, and they correspond one to one.

[0155] In step S803, the current input information is input into the current convolution layer to obtain the current feature information to be fused.

[0156] Optionally, the convolution kernel of the first convolution layer is 7*7, and the step size is 2. The server can input the mask of the first image frame into the first convolution layer to obtain the current feature information to be fused with a resolution of 1 / 2 and an output channel of 64. The current feature information to be fused here is the current feature information to be fused output by the first convolution layer.

[0157] In step S805, a sum operation is performed on the current feature information to be fused and the first feature information output by the module corresponding to the current convolutional layer to obtain fused feature information.

[0158] Optionally, the module corresponding to the current convolutional layer refers to the first module in the main network of the query encoder corresponding to the first convolutional layer. The first feature information output by the first module is a feature of the first scale.

[0159] Optionally, the server may perform a bit-by-bit addition operation on the current feature information to be fused output by the first convolutional layer and the features of the first scale to obtain fused feature information. The fused feature information at this time may be referred to as first fused feature information.

[0160] In step S807, the next convolution layer of the current convolution layer is used to update the current convolution layer, and the current input information is updated using the fused feature information; the following steps are repeated: the current input information is input into the current convolution layer to obtain the current feature information to be fused; the current feature information to be fused and the first feature information output by the module corresponding to the current convolution layer are summed to obtain the fused feature information.

[0161] In the embodiment of the present application, the server can add the next convolutional layer of the first convolutional layer, i.e. Figure 7 The second convolutional layer in is treated as the current convolutional layer, and the first fused feature information is updated as the current input information. Optionally, the convolution kernel of the second convolutional layer is 3*3 and the stride is 2.

[0162] Optionally, the server may input the first fused feature information into the second convolutional layer to obtain current feature information to be fused with a resolution of 1 / 4 and an output channel of 256. Here, the current feature information to be fused is the current feature information to be fused output by the second convolutional layer. Subsequently, the server may perform a bit-by-bit addition operation on the current feature information to be fused and the features at the second scale output by the second module corresponding to the second convolutional layer to obtain fused feature information. This fused feature information may be referred to as second fused feature information.

[0163] The server can transfer the next convolutional layer of the second convolutional layer, i.e. Figure 7 The third convolutional layer in is treated as the current convolutional layer, and the second fused feature information is updated as the current output information. Optionally, the convolution kernel of the third convolutional layer is 3*3 and the stride is 2.

[0164] Optionally, the server may input the second fused feature information into the third convolutional layer to obtain current feature information to be fused with a resolution of 1 / 8 and an output channel of 512. Here, the current feature information to be fused is the current feature information to be fused output by the third convolutional layer. Subsequently, the server may perform a bit-by-bit addition operation on the current feature information to be fused and the features at the third scale output by the third module corresponding to the third convolutional layer to obtain fused feature information. This fused feature information may be referred to as third fused feature information.

[0165] The server can transfer the next convolution layer of the third convolution layer, i.e. Figure 7 The fourth convolutional layer in is treated as the current convolutional layer, and the third fused feature information is updated as the current output information. Optionally, the convolution kernel of the fourth convolutional layer is 3*3 and the stride is 2.

[0166] Optionally, the server may input the third fused feature information into the fourth convolutional layer to obtain current feature information to be fused with a resolution of 1 / 16 and an output channel of 1024. Here, the current feature information to be fused is the current feature information to be fused output by the fourth convolutional layer. Subsequently, the server may perform a bit-by-bit addition operation on the current feature information to be fused and the features at the fourth scale output by the fourth module corresponding to the fourth convolutional layer to obtain fused feature information. This fused feature information may be referred to as fourth fused feature information.

[0167] In step S809, the fusion feature information corresponding to the last convolutional layer among the multiple convolutional layers is obtained.

[0168] Since the fused feature information corresponding to the last convolutional layer among the multiple convolutional layers above, that is, the fourth convolutional layer, has been determined, the server can stop the loop operation and process the fused feature information corresponding to the fourth convolutional layer.

[0169] In step S811, first query information and first mask information are generated based on the fused feature information corresponding to the last convolutional layer.

[0170] Specifically, the convolution kernel of the convolution layer after the main network is 3*3, and the stride is 1. The convolution kernel of each convolution layer in the two parallel-connected convolution layers is 3*3. In this way, the server can input the fourth fusion feature information into the convolution layer after the main network, and input the information output by the convolution layer after the main network into the two parallel-connected convolution layers respectively. The left convolution layer of the two parallel-connected convolution layers can output the first query information with a resolution of 1 / 16 and an output channel of 128, and the right convolution layer can output the first mask information with a resolution of 1 / 16 and an output channel of 128.

[0171] In this embodiment of the present application, the first query information indicates that sampling information to be obtained from the first image frame is determined based on the similarity between the first image frame and the second image frame. In other words, the first query information is used to evaluate the similarity between the second image frame and the first image frame and determine which features in the first image frame should be sampled. Thus, the first query information encodes visual semantics for achieving robustness to changes in the target's appearance.

[0172] In this embodiment of the present application, the first mask information includes appearance information, foreground information, and background information of the first image frame. In other words, the first mask information is used to generate a detailed mask result of the second image frame, including detailed target appearance information and information for distinguishing foreground and background.

[0173] In an embodiment of the present application, when there are multiple first image frames, the server may determine the first query information and first mask information for each first image frame according to steps S801-S811, and determine the comprehensive first query information and comprehensive first mask information corresponding to the multiple first image frames based on the first query information and first mask information of each first image frame. Optionally, the server may jointly add the first query information of each first image frame to obtain the comprehensive first query information corresponding to the multiple first image frames, and jointly add the first mask information of each first image frame to obtain the comprehensive first mask information corresponding to the multiple first image frames.

[0174] Optionally, to ensure that each layer of the memory encoder undergoes a nonlinear transformation at output, an activation function PReLu can be added after each layer. Specifically, this activation function can be added after the first, second, third, and fourth fused feature information, and after the convolutional layer after the main network.

[0175] As can be seen from the above, since the first multi-scale features of the first image frame are historical moments, the query encoder's main network output is stored in the memory area corresponding to the mask recognition model. Therefore, when determining the first phase query information and first mask information for the first image frame, there is no need to use the memory encoder to determine the first multi-scale features of the first image frame a second time, thus reducing the device's overhead of loading the mask recognition model for image recognition.

[0176] In step S207 , second query information and second mask information are generated based on the second image frame.

[0177] In the embodiment of the present application, the server may input the second image frame into the query encoder to obtain second query information and second mask information of the second image frame.

[0178] like Figure 5 As shown in the figure, the query encoder includes a main network and two parallel convolutional layers located after the main network. The main network of the query encoder includes a first module, a second module, a third module, and a fourth module connected in sequence. The two parallel convolutional layers located after the main network can be called the query convolution layer and the mask convolution layer from left to right.

[0179] Figure 9 This is a flowchart of a method for determining second query information and second mask information of a second image frame according to an exemplary embodiment, including:

[0180] In step S901 , feature information is extracted from the second image frame based on multiple modules to obtain second feature information corresponding to each module in the multiple modules.

[0181] Optionally, the server may use the main network of the query encoder to extract feature information of the second image frame to obtain second feature information corresponding to each module in the multiple modules.

[0182] Specifically, the server may input the second image frame into the first module to obtain the second feature information output by the first module.

[0183] Optionally, the second image frame can be a three-primary-color RGB image. The second image frame has a resolution of H×W, where H is the height of the second image frame's resolution, W is the width of the second image frame's resolution, and the number of channels of the second image frame is 3. For example, the second image frame can have a resolution of 224*224 or 640*360. The server can resize the second image frame to obtain a second image frame with a resolution of 224*224.

[0184] In an embodiment of the present application, the second module may include multiple convolutional layers, and the server may input the second feature information output by the first module into the second module to obtain the second feature information output by the second module with a resolution of 1 / 4 and an output channel of 256.

[0185] In an embodiment of the present application, the third module may include multiple convolutional layers, and the server may input the second feature information output by the second module into the third module to obtain the second feature information output by the third module with a resolution of 1 / 8 and an output channel of 512.

[0186] In an embodiment of the present application, the fourth module may include multiple convolutional layers. The server may input the second feature information output by the third module into the fourth module to obtain the second feature information output by the fourth module with a resolution of 1 / 16 and an output channel of 1024. In this way, the server can obtain the second feature information corresponding to each module. The second feature information corresponding to each module may also be referred to as the first multi-scale feature of the second image frame.

[0187] In step S903, information extraction is performed on the second feature information corresponding to the last module in the multiple modules based on the query convolution layer to obtain second query information of the second image frame.

[0188] Optionally, the convolution kernel of the query convolution layer is 3*3. In this way, the server can input the second feature information corresponding to the fourth module into the query convolution layer, and output the second query information with a resolution of 1 / 16 and an output channel of 128.

[0189] In this embodiment of the present application, the second query information indicates the sampling information to be obtained from the second image frame based on the similarity between the first and second image frames. In other words, the second query information is used to evaluate the similarity between the second and first image frames and determine which features in the second image frame should be sampled. Thus, the second query information encodes visual semantics for robustness to changes in the target's appearance.

[0190] In step S905 , the second feature information corresponding to the last module in the multiple modules is extracted based on the mask convolution layer to obtain second mask information of the second image frame.

[0191] Optionally, the convolution kernel of the mask convolution layer is 3*3. In this way, the server can input the second feature information corresponding to the fourth module into the query convolution layer, and output the second mask information with a resolution of 1 / 16 and an output channel of 128.

[0192] In the embodiment of the present application, the second mask information is used to generate a fine mask result of the second image frame, and includes detailed appearance information of the second image frame, which is used for decoding and generating a detailed mask.

[0193] In an embodiment of the present application, the server can determine the second feature information of the second image frame corresponding to each module in the multiple modules as the first multi-scale feature of the second image frame, and store the first multi-scale feature of the second image frame in the memory area. Optionally, the first multi-scale feature of the second image frame acts on the determination of the mask of the third image frame, and the position of the third image frame in the video is located after the position of the second image frame in the video. From the above content, it can be seen that the first image frame output by the query encoder pre-processed at the historical moment (for example Figure 4 The first multi-scale features (features of the first scale, features of the second scale, features of the third scale, and features of the fourth scale) of the image frame 2 in the video shown can be used as information input to the memory encoder at the current moment to determine the second image frame (for example Figure 4 The mask of the image frame 5) in the video shown in FIG.

[0194] Optionally, when the server needs to determine the mask of the third image frame based on the first multi-scale feature of the second image frame, the second image frame (e.g. Figure 4 The image frame 5 in the video shown in FIG is determined as the standby image frame at a future moment, that is, the first image frame at a future moment, to determine the third image frame, that is, the second image frame at a future moment (for example Figure 4 The server can use the mask of the image frame 10 in the video shown in FIG. 10 to store the first image frame at a future moment (eg Figure 4 The second feature information output by the first module corresponding to the image frame 5 in the video shown in FIG is determined as the first image frame at a future moment (eg Figure 4 The first scale feature of the image frame 5 in the video shown in the figure is determined, the second feature information output by the second module is determined as the second scale feature of the first image frame at the future moment, the second feature information output by the third module is determined as the third scale feature of the first image frame at the future moment, and the second feature information output by the fourth module is determined as the fourth scale feature of the first image frame at the future moment.

[0195] Then, the server may determine the first scale feature, the second scale feature, the third scale feature, and the fourth scale feature of the first image frame at the future moment as the first multi-scale feature of the first image frame at the future moment, and store the first multi-scale feature of the first image frame at the future moment in the memory area for determining the second image frame at the future moment (for example, Figure 4 The mask of the image frame 10) in the video shown in FIG.

[0196] In this way, the server may not need to use a memory encoder to calculate the first image frame at a future time (e.g. Figure 4 5) in the video shown in FIG, performing a second determination of the first multi-scale features, thereby saving the device the overhead of loading the mask recognition model for image recognition. Of course, the aforementioned future moment can be considered the second mask determination in the entire loop of video mask determination.

[0197] In step S209 , a mask of the second image frame is determined according to the first query information, the second query information, the first mask information, and the second mask information.

[0198] Optionally, the server may determine the mask of the second image frame based on the decoder, the first query information, the first mask information, the second query information, and the second mask information in the mask recognition model. That is, the decoder decodes the first query information, the second query information, the first mask information, and the second mask information to determine the mask of the second image frame.

[0199] Figure 10 This is a flowchart of a method for determining a mask of a second image frame according to an exemplary embodiment, including:

[0200] In step S1001 , frame similarity information is determined based on first query information and second query information.

[0201] Optionally, the server can perform an inner product operation on the first query information and the second similarity rating information to obtain frame similarity information. The frame similarity value information is equivalent to a spatiotemporal attention mechanism that assigns weights to the appearance information of different events and regions.

[0202] In step S1002 , spatiotemporal memory information is determined based on the frame similarity information and the first mask information.

[0203] Optionally, the server may multiply the frame similarity information and the first mask information to obtain spatiotemporal memory information.

[0204] In step S1003 , frame splicing information is determined based on the spatiotemporal memory information and the second mask information.

[0205] Optionally, the server may splice the spatiotemporal memory information and the second mask information bit by bit to obtain frame splicing information.

[0206] In step S1004, the frame splicing information is input into the decoder to obtain a mask of the second image frame.

[0207] Optionally, the server may input the frame splicing information into a decoder in the mask recognition model to obtain a mask of the second image frame.

[0208] In summary, the server may not need to use a memory encoder to encode the first image frame (eg Figure 4 The image frame 5 in the video shown in the figure performs a second determination of the first multi-scale features, and only the mask of the first image frame needs to be used as the input for feature extraction, which saves memory space and saves the cost of loading the mask recognition model for image recognition on the device.

[0209] This embodiment of the present application also includes acquiring a mask of the first image frame.

[0210] Figure 11 FIG. 1 is a flowchart showing a mask acquisition process of a first image frame according to an exemplary embodiment. Figure 11 As shown, the method includes the following steps:

[0211] In step S1101 , a first image frame is acquired.

[0212] In the embodiment of the present application, the server can obtain the first image frame. For example, the first image frame can be Figure 4 In the video shown in the image frame 1, image frame 2, image frame 3 or image frame 4, image frame 5 can be the second image frame. When the mask of image frame 5 is determined by the mask recognition model, it can continue to propagate to the subsequent image frames of the video without the need to use Figure 11 An embodiment of the invention determines the mask.

[0213] In the embodiment of the present application, when there are multiple first image frames, the server may obtain N first image frames, and each of the N first image frames may be recorded as: x i , i=1,2,3...N。

[0214] In step S1103 , a mask corresponding to the object in the first image frame is determined.

[0215] In this embodiment of the present application, if there is only one first image frame, the server can determine the mask corresponding to the object in the first image frame. If there are multiple first image frames, the server can determine the mask image corresponding to the object in each first image frame. The following uses a single first image frame as an example to illustrate how to determine the mask of the object in the first image frame.

[0216] In some optional embodiments, the server can directly obtain the mask corresponding to the first image frame. Specifically, before the server obtains the mask, the objects in the first image frame can be manually annotated. If there are multiple objects in the first image frame, the different objects can be distinguished and annotated to obtain masks corresponding to the different objects. Although the mask obtained by this embodiment is highly accurate, it requires a large amount of manpower and time.

[0217] In some optional embodiments, the server may identify the preset object in the first image frame and determine a mask of the preset object. Specifically, the server may call an instance segmentation algorithm to identify the preset object in the first image frame and determine a mask of the preset object.

[0218] The process of the instance segmentation algorithm based on the first image frame is as follows: the first image frame is preprocessed, and the preprocessed first image frame is input into a trained neural network to obtain the feature map corresponding to the first image frame. Subsequently, the server can set a predetermined number of regions of interest (ROI) for each point in the feature map to obtain multiple candidate regions of interest. These candidate regions of interest are sent to the region proposal network (RPN) for binary classification (classification of foreground and background) and bounding-box (BB) regression to filter out a part of the candidate regions of interest. Subsequently, the ROI alignment operation can be performed on the remaining regions of interest, that is, the pixels of the first image frame and the feature map are matched, and then the feature map is matched with the fixed features. Finally, the remaining regions of interest are classified, BB regressed and mask generated. In this way, the server can obtain the mask of the preset object in the first image frame.

[0219] However, while instance segmentation algorithms can automatically segment different objects and generate masks for them, the types of objects they can segment are limited, typically to common categories like people, cars, trees, dogs, and cats. Therefore, the server only obtains masks for pre-set objects (e.g., people, cars, trees, dogs, cats, etc.) in the first image frame.

[0220] In some other possible embodiments, in order to prevent the server from only being able to obtain the mask of the preset object, Figure 12 FIG. 1 is a flowchart showing a method for determining a mask corresponding to an object according to an exemplary embodiment. Figure 12 Shown, including:

[0221] In step S1201 , an object in a first image frame is identified, and pixels of the object in the first image frame are determined.

[0222] Figure 13 is a schematic diagram showing a first image frame according to an exemplary embodiment. Figure 14-16 FIG. 1 is a schematic diagram of a mask of a first image frame according to an exemplary embodiment. In the embodiment of the present application, the server may perform masking on the first image frame, such as Figure 13 The foreground object in the image frame is identified, and the pixels of the foreground object (the object shown in 1301 and 1302) in the first image frame are determined.

[0223] In step S1203 , the first image frame is binarized (0 and 255) based on the pixels of the object to obtain a mask corresponding to the first image frame.

[0224] Optionally, the server may perform binarization processing on the first image frame based on the pixels of the foreground objects (the objects shown in 1301 and 1302), such as Figure 14 As shown, the foreground object and the background are distinguished to obtain masks 1401 and 1402 corresponding to the first image frame.

[0225] Among them, the mask can be recorded as y i , i=1,2,3...N。

[0226] In step S1205 , connected region segmentation processing is performed on the mask corresponding to the first image frame to obtain a mask corresponding to the object in the first image frame.

[0227] Optionally, in an embodiment of the present application, the server may perform connected region segmentation processing on the mask corresponding to the first image frame based on the characteristics of the connected domain in the mask corresponding to the first image frame, wherein the pixels of the object corresponding to 1301 and the pixels of the object corresponding to 1302 are not connected, so the two objects 1301 and 1302 may be segmented to obtain Figure 15 The object 1301 shown corresponds to the object mask 1501 and Figure 16 The object mask 1601 corresponding to the object two 1302 is shown.

[0228] If the first image frame x i Containing m objects, the mask corresponding to the object in the first image frame can be recorded as:

[0229]

[0230] That is, if there is one object in a first image frame, the mask corresponding to the first image frame is the mask corresponding to the object. If there are multiple objects in a first image frame, the mask corresponding to the first image frame can be divided into multiple masks corresponding to multiple objects.

[0231] Figure 17 This is a block diagram of a video image processing device according to an exemplary embodiment. The device has the function of implementing the data processing method in the above method embodiment. The function can be implemented by hardware or by hardware executing corresponding software. Figure 14 The device includes a first acquisition module 1701, a second acquisition module 1702, a first generation module 1703, a second generation module 1704 and a mask determination module 1705.

[0232] A first acquisition module 1701 is configured to acquire a mask and a second image frame of a first image frame of a video, where the second image frame is located after the first image frame in the video;

[0233] The second acquisition module 1702 is configured to acquire a first multi-scale feature of the first image frame; the first multi-scale feature of the first image frame is obtained by pre-processing;

[0234] A first generating module 1703 is configured to generate first query information and first mask information based on the first multi-scale features and the mask of the first image frame;

[0235] The second generating module 1704 is configured to generate second query information and second mask information based on the second image frame;

[0236] The mask determination module 1705 is configured to determine a mask of the second image frame according to the first query information, the second query information, the first mask information, and the second mask information.

[0237] In some possible embodiments, the first multi-scale feature of the first image frame is pre-processed by a query encoder, and a main network of the query encoder includes a plurality of modules connected sequentially;

[0238] Before obtaining the first multi-scale feature of the first image frame, the method further includes a first multi-scale feature processing module configured to execute:

[0239] Extracting feature information from the first image frame based on the multiple modules to obtain first feature information corresponding to each module in the multiple modules;

[0240] Determine a first multi-scale feature of the first image frame based on the first feature information corresponding to each module;

[0241] The first multi-scale features of the first image frame are stored in the memory area.

[0242] In some possible embodiments, the first generating module is configured to execute:

[0243] The first convolutional layer in the sequentially connected plurality of convolutional layers included in the main network of the memory encoder is used as the current convolutional layer; the mask of the first image frame is used as the current input information; the number of the plurality of convolutional layers is the same as the number of the plurality of modules included in the main network of the query encoder, and the number of the plurality of convolutional layers is one-to-one corresponding to the number of the modules;

[0244] Input the current input information into the current convolution layer to obtain the current feature information to be fused;

[0245] Performing a sum operation on the current feature information to be fused and the first feature information output by the module corresponding to the current convolutional layer to obtain fused feature information;

[0246] Use the next convolution layer of the current convolution layer to update the current convolution layer, and use the fused feature information to update the current input information; repeat the following steps: input the current input information into the current convolution layer to obtain the current feature information to be fused; add the current feature information to be fused and the first feature information output by the module corresponding to the current convolution layer to obtain the fused feature information;

[0247] Until the fusion feature information corresponding to the last convolution layer in multiple convolution layers is obtained;

[0248] Based on the fused feature information corresponding to the last convolutional layer, first query information and first mask information are generated.

[0249] In some possible embodiments, the query encoder further includes a query convolution layer and a mask convolution layer located after the main network of the query encoder; the query convolution layer and the mask convolution layer are connected in parallel;

[0250] The second generation module is configured to execute:

[0251] Extracting feature information from the second image frame based on the multiple modules to obtain second feature information corresponding to each module in the multiple modules;

[0252] Extracting second feature information corresponding to a last module among the multiple modules based on the query convolution layer to obtain second query information of the second image frame;

[0253] Based on the mask convolution layer, second feature information corresponding to the last module in the multiple modules is extracted to obtain second mask information of the second image frame.

[0254] In some possible embodiments, the apparatus further includes a second multi-scale feature processing module configured to execute:

[0255] Determining second feature information of the second image frame corresponding to each module in the plurality of modules as a first multi-scale feature of the second image frame;

[0256] storing the first multi-scale features of the second image frame in a memory area; and applying the first multi-scale features of the second image frame to determine a mask of the third image frame;

[0257] The position of the third image frame in the video is located after the position of the second image frame in the video.

[0258] In some possible embodiments, the mask generation module is configured to execute:

[0259] determining frame similarity information based on the first query information and the second query information;

[0260] determining spatiotemporal memory information based on the frame similarity information and the first mask information;

[0261] determining frame splicing information based on the spatiotemporal memory information and the second mask information;

[0262] The frame splicing information is input into the decoder to obtain the mask of the second image frame.

[0263] In some possible embodiments, the first acquisition module is configured to execute:

[0264] Acquire multiple masks and second image frames corresponding one to one to multiple first image frames of the video;

[0265] Acquire a first multi-scale feature of each first image frame in a plurality of first image frames;

[0266] Generating first query information and first mask information based on first multi-scale features and a mask of a first image frame includes:

[0267] determining query information and mask information for each first image frame based on the mask and the first multi-scale features of each first image frame;

[0268] First query information and first mask information corresponding to the plurality of first image frames are determined based on the fusion of query information and mask information of each first image frame in the plurality of first image frames.

[0269] It should be noted that the apparatus provided in the above embodiments, when implementing its functions, is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0270] Figure 18 FIG3 is a block diagram of an apparatus 3000 for video image processing according to an exemplary embodiment. For example, the apparatus 3000 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0271] Reference Figure 18 , the device 3000 may include one or more of the following components: a processing component 3002 , a memory 3004 , a power component 3006 , a multimedia component 3008 , an audio component 3010 , an input / output (I / O) interface 3012 , a sensor component 3014 , and a communication component 3016 .

[0272] The processing component 3002 generally controls the overall operation of the device 3000, such as operations associated with display, phone calls, data communications, camera operation, and recording operations. The processing component 3002 may include one or more processors 3020 to execute instructions to perform all or part of the steps of the above-described method. In addition, the processing component 3002 may include one or more modules to facilitate interaction between the processing component 3002 and other components. For example, the processing component 3002 may include a multimedia module to facilitate interaction between the multimedia component 3008 and the processing component 3002.

[0273] The memory 3004 is configured to store various types of data to support operations on the device 3000. Examples of such data include instructions for any application or method operating on the device 3000, contact data, phone book data, messages, pictures, videos, etc. The memory 3004 can be implemented by any type of volatile or non-volatile storage device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0274] The power supply component 3006 provides power to the various components of the device 3000. The power supply component 3006 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device 3000.

[0275] The multimedia component 3008 includes a screen that provides an output interface between the device 3000 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, slides, and gestures on the touch panel. The touch sensor can not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 3008 includes a front camera and / or a rear camera. When the device 3000 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera can receive external multimedia data. Each front camera and rear camera can be a fixed optical lens system or have a focal length and optical zoom capability.

[0276] The audio component 3010 is configured to output and / or input audio signals. For example, the audio component 3010 includes a microphone (MIC) that is configured to receive external audio signals when the device 3000 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal can be further stored in the memory 3004 or transmitted via the communication component 3016. In some embodiments, the audio component 3010 also includes a speaker for outputting audio signals.

[0277] I / O interface 3012 provides an interface between processing component 3002 and peripheral interface modules, such as a keyboard, click wheel, buttons, etc. These buttons may include but are not limited to: a home button, volume buttons, a start button, and a lock button.

[0278] The sensor assembly 3014 includes one or more sensors for providing various aspects of the status assessment of the device 3000. For example, the sensor assembly 3014 can detect the open / closed state of the device 3000, the relative positioning of components, such as the display and keypad of the device 3000. The sensor assembly 3014 can also detect changes in the position of the device 3000 or a component of the device 3000, the presence or absence of user contact with the device 3000, the orientation or acceleration / deceleration of the device 3000, and changes in the temperature of the device 3000. The sensor assembly 3014 can include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 3014 can also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 3014 can also include an accelerometer, a gyroscope, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0279] The communication component 3016 is configured to facilitate wired or wireless communication between the device 3000 and other devices. The device 3000 can access a wireless network based on a communication standard, such as WiFi, an operator network (such as 2G, 3G, 4G or 5G), or a combination thereof. In an exemplary embodiment, the communication component 3016 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 3016 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0280] In an exemplary embodiment, the apparatus 3000 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above-described method.

[0281] An embodiment of the present invention also provides a computer-readable storage medium, which can be set in an electronic device to store at least one instruction or at least one program related to implementing a video image processing method. The at least one instruction or the at least one program is loaded and executed by the processor to implement the video image processing method provided by the above method embodiment.

[0282] An embodiment of the present invention also provides a computer program product, which includes a computer program, the computer program is stored in a readable storage medium, and at least one processor of a computer device reads and executes the computer program from the readable storage medium, so that the computer device performs any method of the first aspect of the embodiment of the present disclosure.

[0283] It should be noted that the order in which the embodiments of the present invention are described above is for illustrative purposes only and does not represent the superiority or inferiority of the embodiments. The above description is of specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0284] The various embodiments in this specification are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences from other embodiments. In particular, the device embodiments are generally similar to the method embodiments, so the description is relatively simple. For relevant parts, refer to the description of the method embodiments.

[0285] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or by a program to instruct the relevant hardware, and the program may be stored in a computer-readable storage medium, which may be a read-only memory, a disk, or an optical disk, etc.

[0286] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A video image processing method, characterized in that: include: Acquire a mask and a second image frame of a first image frame of a video, wherein a position of the second image frame in the video is located after a position of the first image frame in the video; Acquire first multi-scale features of the first image frame; The first multi-scale feature of the first image frame is obtained by pre-processing by a query encoder; The first convolutional layer among the plurality of sequentially connected convolutional layers included in the main network of the memory encoder is used as the current convolutional layer; and the mask of the first image frame is used as the current input information; The number of the multiple convolutional layers is the same as the number of the multiple modules of the main network in the query encoder and corresponds one to one; Inputting the current input information into the current convolutional layer to obtain current feature information to be fused; Performing a sum operation on the current feature information to be fused and the first feature information output by the module corresponding to the current convolutional layer to obtain fused feature information, where the module corresponding to the current convolutional layer is the module corresponding to the current convolutional layer in the main network of the query encoder; Updating the current convolutional layer with the next convolutional layer of the current convolutional layer, and updating the current input information with the fused feature information; Repeating the steps of: inputting the current input information into the current convolutional layer to obtain current feature information to be fused; and performing a sum operation on the current feature information to be fused and the first feature information output by the module corresponding to the current convolutional layer to obtain fused feature information. Until the fusion feature information corresponding to the last convolutional layer in the multiple convolutional layers is obtained; Generate first query information and first mask information based on the fused feature information corresponding to the last convolutional layer; generating second query information and second mask information based on the second image frame; A mask for the second image frame is determined according to the first query information, the second query information, the first mask information, and the second mask information.

2. The video image processing method according to claim 1, wherein: The main network of the query encoder includes a plurality of modules connected in sequence; Before acquiring the first multi-scale features of the first image frame, the method further includes: Extracting feature information from the first image frame based on the multiple modules to obtain first feature information corresponding to each module in the multiple modules; determining a first multi-scale feature of the first image frame based on the first feature information corresponding to each module; The first multi-scale features of the first image frame are stored in a memory area.

3. The video image processing method according to claim 2, characterized in that: The query encoder further includes a query convolutional layer and a mask convolutional layer located after the main network of the query encoder; The query convolution layer and the mask convolution layer are connected in parallel; The generating second query information and second mask information based on the second image frame includes: Extracting feature information from the second image frame based on the multiple modules to obtain second feature information corresponding to each module in the multiple modules; Extracting second feature information corresponding to a last module among the multiple modules based on the query convolution layer to obtain second query information of the second image frame; Based on the mask convolution layer, second feature information corresponding to the last module of the multiple modules is extracted to obtain second mask information of the second image frame.

4. The video image processing method according to claim 3, wherein: The method further comprises: determining second feature information of the second image frame corresponding to each module in the plurality of modules as a first multi-scale feature of the second image frame; storing the first multi-scale features of the second image frame in the memory area; and applying the first multi-scale features of the second image frame to determine a mask of the third image frame; The position of the third image frame in the video is located after the position of the second image frame in the video.

5. The video image processing method according to any one of claims 1 to 4, characterized in that: The determining the mask of the second image frame according to the first query information, the second query information, the first mask information, and the second mask information includes: determining frame similarity information based on the first query information and the second query information; determining spatiotemporal memory information based on the frame similarity information and the first mask information; determining frame splicing information based on the spatiotemporal memory information and the second mask information; The frame splicing information is input into a decoder to obtain a mask of the second image frame.

6. The video image processing method according to any one of claims 1 to 4, characterized in that: The acquiring of a mask of a first image frame and a second image frame of a video, and acquiring a first multi-scale feature of the first image frame, includes: Acquire a plurality of masks and a second image frame that correspond one-to-one to a plurality of first image frames of the video; Acquire a first multi-scale feature of each first image frame in the plurality of first image frames; The generating first query information and first mask information based on the first multi-scale features and the mask of the first image frame includes: determining query information and mask information of each first image frame based on the mask and the first multi-scale feature of each first image frame; The first query information and the first mask information corresponding to the plurality of first image frames are determined based on the fusion of the query information and the mask information of each first image frame in the plurality of first image frames.

7. A video image processing device, characterized in that: include: A first acquisition module is configured to acquire a mask and a second image frame of a first image frame of a video, wherein a position of the second image frame in the video is located after a position of the first image frame in the video; A second acquisition module is configured to acquire first multi-scale features of the first image frame; The first multi-scale feature of the first image frame is obtained by pre-processing by a query encoder; The first generation module is configured to execute the first convolutional layer among the plurality of sequentially connected convolutional layers included in the main network of the memory encoder as a current convolutional layer; and to use the mask of the first image frame as current input information; The number of the multiple convolutional layers is the same as the number of the multiple modules of the main network in the query encoder and corresponds one to one; inputting the current input information into the current convolutional layer to obtain the current feature information to be fused; Performing a sum operation on the current feature information to be fused and the first feature information output by the module corresponding to the current convolutional layer to obtain fused feature information, where the module corresponding to the current convolutional layer is the module corresponding to the current convolutional layer in the main network of the query encoder; updating the current convolutional layer with the next convolutional layer of the current convolutional layer, and updating the current input information with the fused feature information; Repeating the steps of: inputting the current input information into the current convolutional layer to obtain current feature information to be fused; performing a sum operation on the current feature information to be fused and the first feature information output by the module corresponding to the current convolutional layer to obtain fused feature information; until obtaining fused feature information corresponding to the last convolutional layer in the multiple convolutional layers; Generate first query information and first mask information based on the fused feature information corresponding to the last convolutional layer; A second generating module is configured to generate second query information and second mask information based on the second image frame; The mask determination module is configured to determine the mask of the second image frame according to the first query information, the second query information, the first mask information, and the second mask information.

8. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the video image processing method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the video image processing method according to any one of claims 1 to 6.

Images