A video segmentation model training method, device and equipment and storage medium

Abstract

The present disclosure relates to a video segmentation model training method, device and equipment, and a storage medium, and relates to the technical field of computers, and can improve the accuracy of a video segmentation model. The video segmentation model training method comprises: obtaining second video information and a plurality of first video information; determining a plurality of first encoding results based on a first encoding network in a video segmentation model to be trained; determining a plurality of first decoding results based on an auxiliary decoding network; determining a first loss function based on each first decoding result and an actual mask object of a historical video frame in corresponding first video information; determining a second loss function based on a current video frame segmentation submodel in the video segmentation model to be trained and the plurality of first encoding results; and adjusting the model parameters of the auxiliary decoding network and the video segmentation model to be trained based on the first loss function and the second loss function to obtain a target video segmentation model.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a video segmentation model training method, apparatus, device, and storage medium. Background Technology

[0002] With the rapid development of video processing technology, Video Object Segmentation (VOS) technology has broad application prospects in fields such as intelligent editing of short videos, special effects production, and short video creation.

[0003] With the development of deep learning, deep neural networks have been applied to Video Segmentation (VOS). Deep neural network models can more accurately distinguish target objects from the background in complex scenes, thus greatly improving the object segmentation effect. However, traditional deep neural network models are generally trained based on the loss function obtained from the final decoder, resulting in poor model accuracy and consequently, poor target object segmentation. Summary of the Invention

[0004] This disclosure provides a video segmentation model training method, apparatus, device, and storage medium, which can improve the accuracy of video segmentation model training.

[0005] The technical solution of this disclosure is as follows:

[0006] According to a first aspect of the present disclosure, a video segmentation model training method is provided, which can be applied to an electronic device. The method may include:

[0007] Acquire second video information and multiple first video information; the second video information includes the current video frame of the video to be processed and the reference mask object annotated for the current video frame; each first video information includes: a historical video frame of the video to be processed and an actual mask object of the historical video frame;

[0008] Based on the first encoding network in the video segmentation model to be trained, the historical video frames in each first video information are encoded to obtain multiple first encoding results that correspond one-to-one with multiple first video information.

[0009] Based on the auxiliary decoding network, each of the multiple first encoding results is decoded to obtain multiple first decoding results that correspond one-to-one with the multiple first video information.

[0010] Based on each of the multiple first decoding results and the actual mask object of the historical video frame in the first video information corresponding to each first decoding result, a first loss function is determined.

[0011] Based on the current video frame segmentation sub-model and multiple first encoding results in the video segmentation model to be trained, the current video frame is segmented to obtain the actual mask object of the current video frame. Based on the actual mask object of the current video frame and the reference mask object of the current video frame, the second loss function is determined.

[0012] The model parameters of the auxiliary decoding network and the video segmentation model to be trained are adjusted based on the first loss function and the second loss function until the preset training termination condition is met, and the video segmentation model at the end of training is used as the target video segmentation model; the target video segmentation model is used to determine the mask object of the video frame to be processed.

[0013] Optionally, based on each of the multiple first decoding results and the actual mask object of the historical video frame in the first video information corresponding to each first decoding result, a first loss function is determined, including:

[0014] Based on each first decoding result and the actual mask object of the historical video frame in the first video information corresponding to each first decoding result, determine multiple difference information that correspond one-to-one with multiple first decoding results;

[0015] Based on multiple differences and a preset loss function, the first loss function is determined.

[0016] Optionally, based on the current video frame segmentation sub-model in the video segmentation model to be trained and multiple first encoding results, video segmentation processing is performed on the current video frame to obtain the actual mask object of the current video frame, including:

[0017] Based on the second coding network in the current video frame segmentation sub-model, the current video frame is encoded to obtain the second coding result;

[0018] Based on the second encoding result and multiple first encoding results, video frame stitching features are generated;

[0019] Based on the decoding network in the current video frame segmentation sub-model, the video frame splicing features are decoded to obtain a second decoding result, which indicates the actual mask object of the current video frame.

[0020] Optionally, the first encoding result includes: a first historical feature for identifying historical video frames and a second historical feature for storing detailed information of historical video frames; the detailed information of historical video frames is used to generate a reference mask object for historical video frames; the second encoding result includes: a first current feature for identifying the current video frame and a second current feature for storing detailed information of the current video frame; the detailed information of the current video frame is used to generate an actual mask object for the current video frame;

[0021] Based on the second encoding result and multiple first encoding results, video frame stitching features are generated, including:

[0022] Based on the combination function, a first combination feature and a second combination feature are determined; the first combination feature includes: a plurality of first historical features that correspond one-to-one with a plurality of first encoding results; the second combination feature includes: a plurality of second historical features that correspond one-to-one with a plurality of first encoding results;

[0023] Determine the similarity between the first combined feature and the first current feature;

[0024] The product of similarity and the second combined feature is determined, and the product is concatenated with the second current feature to obtain the video frame concatenation feature.

[0025] Optionally, based on the first encoding network in the video segmentation model to be trained, the historical video frames in each first video information are encoded to obtain multiple first encoding results corresponding one-to-one with multiple first video information, including:

[0026] The historical video frames in each first video information are input into the coding layer in the first coding network to obtain multiple current coding results that correspond one-to-one with multiple first video information.

[0027] Each of the multiple current encoding results is input into the shared network output layer to obtain multiple first encoding results; the shared network output layer is the network output layer shared by the first encoding network and the encoding network in the current video frame segmentation sub-model; the network output layer includes: a convolutional output layer and / or a regularized output layer.

[0028] Optionally, based on the second coding network in the current video frame segmentation sub-model, the current video frame is encoded to obtain a second coding result, including:

[0029] The current video frame is input into the coding layer of the second coding network to obtain the current coding result of the current video frame;

[0030] The current encoding result is input into the shared network output layer to obtain the second encoding result; the shared network output layer is the network output layer shared by the first encoding network and the second encoding network; the network output layer includes: a convolutional output layer and / or a regularized output layer.

[0031] Optionally, the number of parameter types in the second coding network is less than the number of parameter types in the first coding network.

[0032] According to a second aspect of the present disclosure, a video segmentation model training apparatus is provided, which can be applied to an electronic device, including: an acquisition unit and a processing unit;

[0033] The acquisition unit is used to acquire second video information and multiple first video information; the second video information includes the current video frame of the video to be processed and a reference mask object annotated for the current video frame; each first video information includes: a historical video frame of the video to be processed and an actual mask object of the historical video frame;

[0034] The processing unit is used to encode the historical video frames in each first video information based on the first encoding network in the video segmentation model to be trained, so as to obtain multiple first encoding results that correspond one-to-one with multiple first video information.

[0035] The processing unit is also configured to perform decoding processing on each of the multiple first encoding results based on the auxiliary decoding network, so as to obtain multiple first decoding results that correspond one-to-one with the multiple first video information;

[0036] The processing unit is further configured to determine a first loss function based on each of the multiple first decoding results and the actual mask object of the historical video frame in the first video information corresponding to each first decoding result;

[0037] The processing unit is also used to perform video segmentation processing on the current video frame based on the current video frame segmentation sub-model in the video segmentation model to be trained and multiple first encoding results, to obtain the actual mask object of the current video frame, and to determine the second loss function based on the actual mask object of the current video frame and the reference mask object of the current video frame.

[0038] The processing unit is also used to adjust the model parameters of the auxiliary decoding network and the video segmentation model to be trained based on the first loss function and the second loss function until the preset training termination condition is met, and to use the video segmentation model at the end of training as the target video segmentation model; the target video segmentation model is used to determine the mask object of the video frame to be processed.

[0039] Optional, processing unit, specifically used for:

[0040] Based on each first decoding result and the actual mask object of the historical video frame in the first video information corresponding to each first decoding result, determine multiple difference information that correspond one-to-one with multiple first decoding results;

[0041] Based on multiple differences and a preset loss function, the first loss function is determined.

[0042] Optional, processing unit, specifically used for:

[0043] Based on the second coding network in the current video frame segmentation sub-model, the current video frame is encoded to obtain the second coding result;

[0044] Based on the second encoding result and multiple first encoding results, video frame stitching features are generated;

[0045] Based on the decoding network in the current video frame segmentation sub-model, the video frame splicing features are decoded to obtain a second decoding result, which indicates the actual mask object of the current video frame.

[0046] Optionally, the first encoding result includes: a first historical feature for identifying historical video frames and a second historical feature for storing detailed information of historical video frames; the detailed information of historical video frames is used to generate a reference mask object for historical video frames; the second encoding result includes: a first current feature for identifying the current video frame and a second current feature for storing detailed information of the current video frame; the detailed information of the current video frame is used to generate an actual mask object for the current video frame;

[0047] The processing unit is specifically used for:

[0048] Based on the combination function, a first combination feature and a second combination feature are determined; the first combination feature includes: a plurality of first historical features that correspond one-to-one with a plurality of first encoding results; the second combination feature includes: a plurality of second historical features that correspond one-to-one with a plurality of first encoding results;

[0049] Determine the similarity between the first combined feature and the first current feature;

[0050] The product of similarity and the second combined feature is determined, and the product is concatenated with the second current feature to obtain the video frame concatenation feature.

[0051] Optional, processing unit, specifically used for:

[0052] The historical video frames in each first video information are input into the coding layer in the first coding network to obtain multiple current coding results that correspond one-to-one with multiple first video information.

[0053] Each of the multiple current encoding results is input into the shared network output layer to obtain multiple first encoding results; the shared network output layer is the network output layer shared by the first encoding network and the encoding network in the current video frame segmentation sub-model; the network output layer includes: a convolutional output layer and / or a regularized output layer.

[0054] Optional, processing unit, specifically used for:

[0055] The current video frame is input into the coding layer of the second coding network to obtain the current coding result of the current video frame;

[0056] The current encoding result is input into the shared network output layer to obtain the second encoding result; the shared network output layer is the network output layer shared by the first encoding network and the second encoding network; the network output layer includes: a convolutional output layer and / or a regularized output layer.

[0057] Optionally, the number of parameter types in the second coding network is less than the number of parameter types in the first coding network.

[0058] According to a third aspect of the present disclosure, an electronic device is provided, which may include: a processor and a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement any of the optional video segmentation model training methods of the first aspect described above.

[0059] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which instructions are stored, such that when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is able to perform any of the optional video segmentation model training methods of the first aspect described above.

[0060] According to a fifth aspect of the present disclosure, a computer program product is provided, the computer program product including computer instructions that, when executed on a processor of an electronic device, cause the processor of the electronic device to perform a video segmentation model training method as described in any of the optional implementations of the first aspect.

[0061] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure.

[0062] The technical solutions provided by the embodiments of this disclosure bring at least the following beneficial effects:

[0063] Based on any of the above, in this disclosure, after acquiring second video information (including the current video frame of the video to be processed and the reference mask object labeled for the current video frame) and multiple first video information (each first video information includes: a historical video frame of the video to be processed and an actual mask object of the historical video frame), the electronic device can encode the historical video frame in each first video information based on the first encoding network in the video segmentation model to be trained, so as to obtain multiple first encoding results corresponding one-to-one with the multiple first video information.

[0064] Then, the electronic device can perform decoding processing on each of the multiple first encoding results based on the auxiliary decoding network to obtain multiple first decoding results that correspond one-to-one with the multiple first video information, and determine the first loss function based on each of the multiple first decoding results and the actual mask object of the historical video frame in the first video information corresponding to each first decoding result.

[0065] Then, the electronic device can perform video segmentation processing on the current video frame based on the current video frame segmentation sub-model in the video segmentation model to be trained and multiple first encoding results, obtain the actual mask object of the current video frame, and determine the second loss function based on the actual mask object of the current video frame and the reference mask object of the current video frame.

[0066] Subsequently, the electronic device can adjust the model parameters of the auxiliary decoding network and the video segmentation model to be trained based on the first loss function and the second loss function until the preset training termination condition is met, and use the video segmentation model at the end of training as the target video segmentation model; the target video segmentation model is used to determine the mask object of the video frame to be processed.

[0067] In this way, when training the training model, the electronic device can adjust the parameters of the training model not only through the second loss function (determined by the actual mask object of the current video frame and the reference mask object of the current video frame), but also through the first loss function (determined by each of the multiple first decoding results and the actual mask object of the historical video frame in the first video information corresponding to each first decoding result), thereby improving the accuracy of video segmentation model training.

[0068] Secondly, since the first loss function is determined by each of the multiple first decoding results and the actual mask object of the historical video frame in the first video information corresponding to each first decoding result, and each first decoding result is obtained by encoding and decoding the historical video frame in each first video information, adjusting the model parameters of the video segmentation model to be trained through the first loss function can improve the ability of the target video segmentation model to distinguish between mask objects and non-mask objects, enhance the representation ability of the target video segmentation model, and further improve the accuracy and performance of the target video segmentation model. Attached Figure Description

[0069] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0070] Figure 1This diagram illustrates the structure of a video segmentation model training system provided in an embodiment of the present disclosure.

[0071] Figure 2 A flowchart illustrating a video segmentation model training method provided in an embodiment of this disclosure is shown.

[0072] Figure 3 A flowchart illustrating another video segmentation model training method provided in this disclosure is shown.

[0073] Figure 4 A flowchart illustrating another video segmentation model training method provided in this disclosure is shown.

[0074] Figure 5 A flowchart illustrating another video segmentation model training method provided in this disclosure is shown.

[0075] Figure 6 A flowchart illustrating another video segmentation model training method provided in this disclosure is shown.

[0076] Figure 7 A flowchart illustrating another video segmentation model training method provided in this disclosure is shown.

[0077] Figure 8 A flowchart illustrating another video segmentation model training method provided in this disclosure is shown.

[0078] Figure 9 This diagram illustrates the structure of a video segmentation model training apparatus provided in an embodiment of the present disclosure.

[0079] Figure 10 A schematic diagram of the structure of a terminal provided in an embodiment of this disclosure is shown;

[0080] Figure 11 A schematic diagram of the structure of a server provided in an embodiment of this disclosure is shown. Detailed Implementation

[0081] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0082] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0083] It should also be understood that the term "comprising" indicates the presence of the described feature, whole, step, operation, element and / or component, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements and / or components.

[0084] The data disclosed herein may be data authorized by the user or fully authorized by all parties.

[0085] With the development of deep learning, deep neural networks have been applied to Video Segmentation (VOS). Deep neural network models can more accurately distinguish target objects from the background in complex scenes, thus greatly improving the object segmentation effect. However, traditional deep neural network models are generally trained based on the loss function obtained from the final decoder, resulting in poor model accuracy and consequently, poor target object segmentation.

[0086] Based on this, embodiments of this disclosure provide a video segmentation model training method. After acquiring second video information (including the current video frame of the video to be processed and a reference mask object labeled for the current video frame) and multiple first video information (each first video information includes: a historical video frame of the video to be processed and an actual mask object of the historical video frame), the electronic device can encode the historical video frames in each first video information based on the first encoding network in the video segmentation model to be trained, so as to obtain multiple first encoding results corresponding one-to-one with the multiple first video information.

[0087] Then, the electronic device can perform decoding processing on each of the multiple first encoding results based on the auxiliary decoding network to obtain multiple first decoding results that correspond one-to-one with the multiple first video information, and determine the first loss function based on each of the multiple first decoding results and the actual mask object of the historical video frame in the first video information corresponding to each first decoding result.

[0088] Then, the electronic device can perform video segmentation processing on the current video frame based on the current video frame segmentation sub-model in the video segmentation model to be trained and multiple first encoding results, obtain the actual mask object of the current video frame, and determine the second loss function based on the actual mask object of the current video frame and the reference mask object of the current video frame.

[0089] Subsequently, the electronic device can adjust the model parameters of the auxiliary decoding network and the video segmentation model to be trained based on the first loss function and the second loss function until the preset training termination condition is met, and use the video segmentation model at the end of training as the target video segmentation model; the target video segmentation model is used to determine the mask object of the video frame to be processed.

[0090] In this way, when training the training model, the electronic device can adjust the parameters of the training model not only through the second loss function (determined by the actual mask object of the current video frame and the reference mask object of the current video frame), but also through the first loss function (determined by each of the multiple first decoding results and the actual mask object of the historical video frame in the first video information corresponding to each first decoding result), thereby improving the accuracy of video segmentation model training.

[0091] Secondly, since the first loss function is determined by each of the multiple first decoding results and the actual mask object of the historical video frame in the first video information corresponding to each first decoding result, and each first decoding result is obtained by encoding and decoding the historical video frame in each first video information, adjusting the model parameters of the video segmentation model to be trained through the first loss function can improve the ability of the target video segmentation model to distinguish between mask objects and non-mask objects, enhance the representation ability of the target video segmentation model, and further improve the accuracy and performance of the target video segmentation model.

[0092] Figure 1 This is a schematic diagram of a video segmentation model training system provided in an embodiment of the present disclosure, such as... Figure 1 As shown, the video segmentation model training system may include a server 110 and an electronic device 120. The server 110 can establish a connection with the electronic device 120 through a wired network or a wireless network.

[0093] Server 110 can be a data server for multimedia resource service platforms, used for storing and processing multimedia resources. For example, a multimedia resource service platform could be a short video application service platform, a news service platform, a live streaming service platform, a shopping service platform, a food delivery service platform, a sharing service platform, or a functional website. The multimedia resources provided by a short video application service platform can be short video works, the multimedia resources provided by a news service platform can be news information, and the multimedia resources provided by a live streaming service platform can be live broadcasts, etc., and so on. This disclosure does not limit the specific type of multimedia resource service platform.

[0094] In this disclosure, server 110 is mainly used to store sample data required for training the video segmentation model, such as the video to be processed and video frames to be processed. When server 110 receives a data acquisition request from electronic device 120, it can send the corresponding sample data to electronic device 120.

[0095] In some embodiments, server 110 may also include or be connected to a database, and sample data of the multimedia resource service platform may be stored in the database. Electronic device 120 can access the sample data in the database through server 110.

[0096] Electronic device 120 may be a server, a terminal, or other electronic devices used for training video segmentation models; this disclosure does not limit the scope of the device.

[0097] When electronic device 120 is a server, electronic device 120 and server 110 can be two independent servers or integrated into the same server. This application does not make specific limitations in this regard.

[0098] It is easy to understand that when electronic device 120 and server 110 are integrated into the same server, the communication method between electronic device 120 and server 110 is the same as the communication between internal modules of the server. In this case, the communication process between the two is the same as that between electronic device 120 and server 110 when they are independent of each other.

[0099] For ease of understanding, this application mainly uses the independent configuration of electronic device 120 and server 110 as an example for illustration.

[0100] The aforementioned server can be a single server, or it can be a server cluster consisting of multiple servers. In some embodiments, the server cluster can also be a distributed cluster. This disclosure does not limit the specific implementation method of the server.

[0101] When electronic device 120 is used as a terminal, it can be a mobile phone, tablet computer, desktop computer, laptop computer, handheld computer, notebook computer, ultra-mobile personal computer (UMPC), netbook, as well as cellular phone, personal digital assistant (PDA), augmented reality (AR) / virtual reality (VR) device, etc., which can install and use content community applications (such as Kuaishou). This disclosure does not impose any special restrictions on the specific form of the terminal. It can interact with users through one or more methods such as keyboard, touchpad, touch screen, remote control, voice interaction, or handwriting devices.

[0102] Optionally, the above Figure 1 In the video segmentation model training system shown, electronic device 120 can be connected to at least one server 110. This disclosure does not limit the number or type of server 110.

[0103] The video segmentation model training method provided in this disclosure can be applied to the aforementioned... Figure 1 Electronic device 120 in the application scenario shown.

[0104] The video segmentation model training method provided in this disclosure will be described in detail below with reference to the accompanying drawings.

[0105] like Figure 2 As shown, when a video segmentation model training method is applied to an electronic device, the video segmentation model training method may include:

[0106] S201, The electronic device acquires second video information and multiple first video information.

[0107] The second video information includes the current video frame of the video to be processed and the reference mask object labeled for the current video frame. Each piece of first video information includes: a historical video frame of the video to be processed and the actual mask object of the historical video frame.

[0108] In one feasible approach, when the electronic device acquires the second video information and multiple first video information sets, it can determine each video frame in the video to be processed. Next, the electronic device can identify any number of video frames preceding the current video frame as historical video frames and determine the mask object in each historical video frame. Then, the electronic device can store the current video frame and multiple historical video frame information in a database, and retrieve the current video frame and multiple historical video frame information from the database when model training is required.

[0109] Optionally, the database mentioned above can be an external memory bank or other types of databases, and this disclosure does not limit it.

[0110] In another possible implementation, after receiving a video segmentation model training instruction (e.g., a video segmentation model training instruction sent by a user to the electronic device), the electronic device can send a request message to a server storing information about the current video frame and multiple historical video frames, requesting the acquisition of second video information and multiple pieces of first video information. Correspondingly, upon receiving the request message, the server can, in response, retrieve the second video information and multiple pieces of first video information from a database or other storage platform, and then send the second video information and multiple pieces of first video information to the electronic device. Consequently, the electronic device can acquire the second video information and multiple pieces of first video information.

[0111] In another possible implementation, after receiving a video segmentation model training instruction (e.g., a video segmentation model training instruction sent by a user to the server), the server can proactively send second video information and multiple first video information to the electronic device, so that the electronic device can train the video segmentation model based on the second video information and multiple first video information.

[0112] The mask object is used to represent the object information of the target object in a certain video frame (such as the position, region, size, shape, etc. of the target object).

[0113] For example, the target object in video frame A has a pixel value of 1. Correspondingly, the other background pixels in video frame A have a pixel value of 0. In this way, the electronic device can identify the target object with a pixel value of 1 as the mask object for video frame A.

[0114] Specifically, during the training of a video segmentation model, the electronic device needs to acquire a large amount of training data. This training data may include the video to be processed. For ease of understanding, this embodiment uses the video to be processed as the training data for illustration.

[0115] S202. The electronic device encodes the historical video frames in each first video information based on the first encoding network in the video segmentation model to be trained, so as to obtain multiple first encoding results that correspond one-to-one with multiple first video information.

[0116] The feature form of the aforementioned multiple first encoding results can be a matrix, a vector, or other forms of encoding results, and this disclosure does not limit them.

[0117] In one possible approach, the electronic device can encode historical video frames in each first video information based on a first encoder.

[0118] Optionally, the first encoder may be an encoder in a memory encoder network or other types of encoders, which are not limited in this disclosure.

[0119] The above-mentioned query coding network is the first coding network provided in the embodiments of this application.

[0120] It should be noted that the aforementioned first encoder may include multiple encoders or may consist of only one encoder.

[0121] When there are multiple first encoders, the multiple first encoders can be the same encoder or different encoders, and this disclosure does not limit them.

[0122] In another possible implementation, the electronic device can also encode the historical video frames in each first video information through a feature extraction algorithm in the first coding network to obtain multiple first coding results that correspond one-to-one with multiple first video information.

[0123] S203. The electronic device, based on the auxiliary decoding network, decodes each of the multiple first encoding results to obtain multiple first decoding results that correspond one-to-one with the multiple first video information.

[0124] In one possible implementation, the electronic device can decode each of a plurality of first encoded results based on a first decoder.

[0125] Optionally, the first decoder may be a decoder in an auxiliary coding network or other types of decoders, and this disclosure does not limit it.

[0126] It should be noted that the aforementioned first decoder may include multiple decoders, or may consist of only one decoder.

[0127] When there are multiple first decoders, the multiple first decoders can be the same decoder or different decoders, and this disclosure does not limit them.

[0128] S204. The electronic device determines a first loss function based on each of the multiple first decoding results and the actual mask object of the historical video frame in the first video information corresponding to each first decoding result.

[0129] Optionally, the first loss function can be any loss function of a neural network model, such as the mean squared error loss function, the cross-entropy loss function, etc., and this disclosure does not limit it.

[0130] In one feasible approach, the electronic device can determine each first decoding result and the difference information of the actual mask object in the historical video frames in the first video information corresponding to each first decoding result, thereby obtaining multiple difference information. Subsequently, the electronic device can sum the multiple difference information to obtain a first loss function.

[0131] S205. The electronic device performs video segmentation processing on the current video frame based on the current video frame segmentation sub-model in the video segmentation model to be trained and multiple first encoding results, obtains the actual mask object of the current video frame, and determines the second loss function based on the actual mask object of the current video frame and the reference mask object of the current video frame.

[0132] The first loss function is used to characterize the degree of deviation between the actual mask object of the current video frame and the reference mask object of the current video frame.

[0133] Optionally, the second loss function can be any loss function of a neural network model, such as the mean squared error loss function, the cross-entropy loss function, etc., and this disclosure does not limit it.

[0134] In one possible implementation, the electronic device can determine the second loss function based on the difference information between the actual mask object of the current video frame and the reference mask object of the current video frame.

[0135] S206. The electronic device adjusts the model parameters of the auxiliary decoding network and the video segmentation model to be trained based on the first loss function and the second loss function until the preset training termination condition is met, and uses the video segmentation model at the end of training as the target video segmentation model.

[0136] The target video segmentation model is used to determine the mask object for the video frame to be processed.

[0137] Specifically, after determining the first loss function and multiple second loss functions, the electronic device can train the training model based on the first loss function and multiple second loss functions to adjust the parameters of the training model and obtain the target video segmentation model. Subsequently, after acquiring the video frame to be processed, the electronic device can directly input the video frame to be processed into the target video segmentation model to obtain the mask object of the video frame to be processed.

[0138] It should be noted that the video frame to be processed can be any video frame in the video to be processed, or any video frame in other videos; this disclosure does not limit it in this respect.

[0139] In one possible implementation, the preset training termination condition can be that the number of training iterations reaches a preset number, the output of the video segmentation model reaches a preset accuracy, or other preset training termination conditions. This application embodiment does not limit these conditions.

[0140] The technical solution provided by the above embodiments brings at least the following beneficial effects: As can be seen from S201-S206, after obtaining the second video information (including the current video frame of the video to be processed and the reference mask object labeled for the current video frame) and multiple first video information (each first video information includes: a historical video frame of the video to be processed and an actual mask object of the historical video frame), the electronic device can encode the historical video frame in each first video information based on the first encoding network in the video segmentation model to be trained, so as to obtain multiple first encoding results corresponding one-to-one with the multiple first video information.

[0141] Then, the electronic device can perform decoding processing on each of the multiple first encoding results based on the auxiliary decoding network to obtain multiple first decoding results that correspond one-to-one with the multiple first video information, and determine the first loss function based on each of the multiple first decoding results and the actual mask object of the historical video frame in the first video information corresponding to each first decoding result.

[0142] Then, the electronic device can perform video segmentation processing on the current video frame based on the current video frame segmentation sub-model in the video segmentation model to be trained and multiple first encoding results, obtain the actual mask object of the current video frame, and determine the second loss function based on the actual mask object of the current video frame and the reference mask object of the current video frame.

[0143] Subsequently, the electronic device can adjust the model parameters of the auxiliary decoding network and the video segmentation model to be trained based on the first loss function and the second loss function until the preset training termination condition is met, and use the video segmentation model at the end of training as the target video segmentation model; the target video segmentation model is used to determine the mask object of the video frame to be processed.

[0144] In this way, when training the training model, the electronic device can adjust the parameters of the training model not only through the second loss function (determined by the actual mask object of the current video frame and the reference mask object of the current video frame), but also through the first loss function (determined by each of the multiple first decoding results and the actual mask object of the historical video frame in the first video information corresponding to each first decoding result), thereby improving the accuracy of video segmentation model training.

[0145] Secondly, since the first loss function is determined by each of the multiple first decoding results and the actual mask object of the historical video frame in the first video information corresponding to each first decoding result, and each first decoding result is obtained by encoding and decoding the historical video frame in each first video information, adjusting the model parameters of the video segmentation model to be trained through the first loss function can improve the ability of the target video segmentation model to distinguish between mask objects and non-mask objects, enhance the representation ability of the target video segmentation model, and further improve the accuracy and performance of the target video segmentation model.

[0146] In one feasible way, combining Figure 2 ,like Figure 3 As shown, in S204 above, the method by which the electronic device determines the first loss function based on each of the multiple first decoding results and the actual mask object of the historical video frame in the first video information corresponding to each first decoding result specifically includes:

[0147] S301. The electronic device determines multiple difference information corresponding one-to-one with multiple first decoding results based on each first decoding result and the actual mask object of the historical video frame in the first video information corresponding to each first decoding result.

[0148] Specifically, after obtaining multiple first decoding results, since the first decoding results are the reference mask objects of historical video frames in each first video information determined by the video segmentation model to be trained, in order to improve the ability of the target video segmentation model to distinguish between mask objects and non-mask objects, the electronic device can determine multiple difference information corresponding one-to-one with the multiple first decoding results based on each first decoding result and the actual mask objects of historical video frames in the first video information corresponding to each first decoding result.

[0149] S302. The electronic device determines the first loss function based on multiple difference information and a preset loss function.

[0150] The first loss function is used to characterize the deviation of each first decoding result from the actual mask object of the historical video frame in the first video information corresponding to each first decoding result.

[0151] Specifically, after determining multiple difference information that correspond one-to-one with multiple first decoding results, the electronic device can determine the first loss function based on the multiple difference information and the preset loss function.

[0152] Optionally, the preset loss function can be a summation function or other loss functions, and this disclosure does not limit it.

[0153] In one possible implementation, each first decoding result can be obtained from a decoder in a convolutional neural network. Each output first decoding result can be derived from the output of the last layer of the neural network. Since the specific detailed features in each first decoding result are known, after obtaining each first decoding result, the electronic device can determine multiple difference information corresponding one-to-one with the multiple first decoding results based on each first decoding result and the actual mask object of the historical video frames in the first video information corresponding to each first decoding result.

[0154] In this case, the electronic device can input the expression of each first decoding result obtained by the convolutional neural network and the actual mask object of the historical video frame in the first video information corresponding to each first decoding result into the preset loss function to obtain the loss function represented by the network parameters of the convolutional neural network, i.e., the first loss function.

[0155] In another possible approach, the electronic device can calculate the similarity between multiple pieces of difference information and determine the preset function corresponding to the similarity as the first loss function.

[0156] The technical solution provided by the above embodiments brings at least the following beneficial effects: As can be seen from S301-S302, an electronic device is given a specific implementation method for determining the first loss function based on multiple difference information and a preset loss function, so that when training the training model in the future, the parameters of the training model can be adjusted through the first loss function, thereby improving the accuracy of video segmentation model training.

[0157] Secondly, since the first loss function is determined by each of the multiple first decoding results and the actual mask object of the historical video frame in the first video information corresponding to each first decoding result, and each first decoding result is obtained by encoding and decoding the historical video frame in each first video information, adjusting the model parameters of the video segmentation model to be trained through the first loss function can improve the ability of the target video segmentation model to distinguish between mask objects and non-mask objects, enhance the representation ability of the target video segmentation model, and further improve the accuracy and performance of the target video segmentation model.

[0158] In one feasible way, combining Figure 2 ,like Figure 4 As shown, in S205 above, the electronic device performs video segmentation processing on the current video frame based on the current video frame segmentation sub-model in the video segmentation model to be trained and multiple first encoding results, to obtain the actual mask object of the current video frame, including:

[0159] S401. The electronic device encodes the current video frame based on the second coding network in the current video frame segmentation sub-model to obtain the second coding result.

[0160] The second encoding network can be the second encoder in the current video frame segmentation sub-model. The second encoder is used to encode the current video frame to obtain the second encoding result.

[0161] Optionally, the second encoder may be an encoder in a query encoder network or other types of encoders, which are not limited in this disclosure.

[0162] S402. The electronic device generates video frame splicing features based on the second encoding result and multiple first encoding results.

[0163] Specifically, after obtaining the second encoding result and multiple first encoding results, in order to accurately determine the decoding result corresponding to the second encoding result, the electronic device can generate video frame splicing features based on the second encoding result and multiple first encoding results.

[0164] In this way, the electronic device determines the video frame splicing features, which is equivalent to a spatiotemporal attention mechanism. It assigns weights to the second encoding results in different times and regions, and thus can accurately determine the decoding results corresponding to the second encoding results.

[0165] In one possible implementation, the first encoding result includes: a first historical feature for identifying historical video frames and a second historical feature for storing detailed information about the historical video frames. The detailed information about the historical video frames is used to generate a reference mask object for the historical video frames. The second encoding result includes: a first current feature for identifying the current video frame and a second current feature for storing detailed information about the current video frame. The detailed information about the current video frame is used to generate the actual mask object for the current video frame.

[0166] Optionally, the first and second encoding results described above can be key-value pair information. In this case, the first current feature is used to represent the key of the current video frame, and the second current feature is used to represent the value of the current video frame. Correspondingly, the first historical feature is used to represent the key of historical video frames, and the second historical feature is used to represent the value of historical video frames.

[0167] Optionally, the key of the current video frame (i.e., the first current feature) and the key of the historical video frames (i.e., the first historical feature) can be in the form of a three-dimensional matrix, namely H×W×C. Here, H is the height of the current video frame, W is the width of the current video frame, and C is the number of channels in the current video frame. The above-mentioned concepts can be found in the description of feature maps in general techniques, and will not be repeated here.

[0168] The method for generating video frame stitching features based on a second encoding result and multiple first encoding results by an electronic device specifically includes:

[0169] S1. The electronic device determines the first combination feature and the second combination feature based on the combination function.

[0170] The first combined feature includes multiple first historical features that correspond one-to-one with multiple first coding results. The second combined feature includes multiple second historical features that correspond one-to-one with multiple first coding results.

[0171] Specifically, since the number of first encoding results is greater than the number of second encoding results, the electronic device needs to determine, based on the combination function, a first combination feature corresponding one-to-one with multiple first historical features and a second combination feature corresponding one-to-one with multiple second historical features of the multiple first encoding results.

[0172] Optionally, the above-mentioned combination function can be the concat function or other combination functions, and this disclosure does not limit it.

[0173] S2. The electronic device determines the similarity between the first combined feature and the first current feature.

[0174] Optionally, the above similarity algorithm can be the Euclidean distance algorithm or other similarity algorithms, and this disclosure does not limit it.

[0175] The similarity between the first combined feature and the first current feature can be determined based on a similarity algorithm.

[0176] Optionally, the above similarity algorithm can be the Euclidean distance algorithm or other similarity algorithms, and this disclosure does not limit it.

[0177] S3. The electronic device determines the product of similarity and the second combined feature, and concatenates the product with the second current feature to obtain the video frame concatenation feature.

[0178] The technical solution provided by the above embodiments brings at least the following beneficial effects: As can be seen from S1-S3, a specific implementation method for an electronic device to determine video frame splicing features based on the first encoding result and the second encoding result is given, so that the second loss function can be determined based on the video frame splicing features. Then, when training the training model, the parameters of the training model can be adjusted through the second loss function, thereby improving the accuracy of video segmentation model training.

[0179] S403. The electronic device decodes the video frame splicing features based on the decoding network in the current video frame segmentation sub-model to obtain a second decoding result.

[0180] The second decoding result indicates the actual mask object of the current video frame.

[0181] Optionally, the decoding network in the current video frame segmentation sub-model can be a decoder. This decoder can be a decoder in a query decoder network, or it can be other types of decoders; this disclosure does not limit this.

[0182] The technical solution provided by the above embodiments brings at least the following beneficial effects: As can be seen from S401-S403, an electronic device is given a specific implementation method for decoding video frame splicing features based on the decoding network in the current video frame segmentation sub-model to obtain a second decoding result. This allows the second loss function to be determined through the second decoding result when training the training model, and the parameters of the training model to be adjusted according to the second loss function, thereby improving the accuracy of video segmentation model training.

[0183] Combination Figure 4 ,like Figure 5 As shown, in S202 above, the method by which the electronic device encodes historical video frames in each first video information based on the first encoding network in the video segmentation model to be trained, so as to obtain multiple first encoding results corresponding one-to-one with multiple first video information, specifically includes:

[0184] S501, The electronic device inputs the historical video frames in each first video information into the coding layer in the first coding network to obtain multiple current coding results that correspond one-to-one with multiple first video information.

[0185] S502, the electronic device inputs each of the multiple current encoding results into the shared network output layer to obtain multiple first encoding results.

[0186] The shared network output layer is the network output layer shared by the first coding network and the coding network in the current video frame segmentation sub-model. The network output layer includes: a convolutional output layer and / or a regularized output layer.

[0187] The technical solution provided by the above embodiments brings at least the following beneficial effects: As can be seen from S501-S502, a specific implementation method for an electronic device to obtain multiple first encoding results corresponding one-to-one with multiple first video information is given. The electronic device can obtain multiple first encoding results corresponding one-to-one with multiple first video information based on the shared network output layer, which can not only improve the efficiency of determining the mask object, but also improve the accuracy of determining the mask object.

[0188] In one feasible way, combining Figure 5 ,like Figure 6As shown, in S401 above, the method by which the electronic device encodes the current video frame based on the second coding network in the current video frame segmentation sub-model to obtain the second coding result specifically includes:

[0189] S601. The electronic device inputs the current video frame into the coding layer of the second coding network to obtain the current coding result of the current video frame.

[0190] S602, The electronic device inputs the current encoding result to the output layer of the shared network to obtain the second encoding result.

[0191] The shared network output layer is the network output layer shared by the first encoding network and the second encoding network.

[0192] The network output layer includes a convolutional output layer and / or a regularized output layer.

[0193] The technical solution provided by the above embodiments brings at least the following beneficial effects: As can be seen from S601-S602, a specific implementation method for an electronic device to obtain a second encoding result is given. The electronic device can obtain the second encoding result based on the shared network output layer, which can not only improve the efficiency of determining the mask object, but also improve the accuracy of determining the mask object.

[0194] In one possible implementation, the number of parameter types in the second coding network is less than the number of parameter types in the first coding network.

[0195] In some embodiments, the common first and second encoding networks generally share the same basic network structure (such as ResNet50), and then each trains its own parameters.

[0196] In this scenario, for the same mask object in a video, the electronic device only needs to run the first encoding network once, but it needs to run the second encoding network multiple times (because the second encoder network needs to be run for each subsequent video frame to determine the mask object). Therefore, the common first and second encoding networks reduce the efficiency of determining the mask object.

[0197] To address the aforementioned issues, this disclosure proposes a decoupled first and second coding network, which can use different network structures to adapt to the complex and diverse needs of actual production activities.

[0198] That is, the number of parameter types in the second coding network is less than the number of parameter types in the first coding network.

[0199] Thus, by employing a second encoding network with lower computational cost (fewer parameter types reduce computational cost), the efficiency of determining the mask object can be improved by running the second encoding network multiple times. Conversely, by employing a first encoding network with higher computational cost (a larger number of parameter types increases computational cost), the accuracy of determining the mask object can be improved by running the first encoding network only once.

[0200] However, simply using different network structures may result in lower accuracy. That is, the numerical output ranges of different network structures are not the same. In this case, simply changing the network structure may reduce the segmentation accuracy of the algorithm.

[0201] To address the aforementioned issues, this disclosure further proposes a method that shares a single output unit (structure and parameters shared): a 1x1 convolution kernel, and / or batch normalization, thereby constraining the numerical range between different network structures to achieve a stable algorithmic segmentation result.

[0202] That is, the first and second encoding networks share a single network output layer. This network output layer includes a convolutional output layer and / or a regularized output layer.

[0203] For example, such as Figure 7 As shown, in the network architecture of the first encoding network, the first layer uses 64 channels, the second layer uses 256 channels, the third layer uses 512 channels, and the fourth layer uses 1024 channels.

[0204] In the network architecture of the second coding network, the first layer uses 16 channels, the second layer uses 24 channels, the third layer uses 32 channels, and the fourth layer uses 96 channels.

[0205] The first and second encoding networks share a shared network output layer (structure and parameters are shared): a 1x1 convolution kernel and / or batch normalization, so that the same features are output to the corresponding convolution kernel, and then the first encoding result (including key and value) and the second encoding result (including key and value) are obtained through the convolution kernel.

[0206] As can be seen from the above, the video segmentation model training method provided in this disclosure can improve both the efficiency and accuracy of determining the mask object by determining that the number of parameter types of the second coding network is less than the number of parameter types of the first coding network.

[0207] In one feasible way Figure 8This application illustrates a video segmentation model training method provided by an embodiment of the present application. For example... Figure 8 As shown, the training method for this video segmentation model specifically includes:

[0208] Electronic devices can store previous image frames (i.e., historical image frames) and their corresponding target masks (i.e., mask objects) in an external memory.

[0209] When predicting the target mask for the current frame, the electronic device first selects several frames (i.e., multiple historical image frames) and their mask objects from the external memory bank and inputs them into the memory encoding network (i.e., the first encoding network) to obtain the corresponding key and value depth features (i.e., the first encoding result, where the key is the first historical feature and the value is the second historical feature).

[0210] The key and value form a key-value pair. The key is used for addressing, while the value stores more detailed information used to generate the mask object.

[0211] Then, the electronic device inputs the current frame into the Query Encoder, which is the second encoding network, to obtain the deep features of the key and value of the current frame image (i.e., the second encoding information, where the key is the first current feature and the value is the second current feature).

[0212] Next, the electronic device can determine, based on the combination function, a first combination feature corresponding one-to-one with multiple first historical features and a second combination feature corresponding one-to-one with multiple second historical features of multiple first coding results.

[0213] Subsequently, the electronic device performs an inner product operation on the first combined features and the key output by the query encoding network to calculate the similarity.

[0214] The aforementioned similarity algorithm is equivalent to a spatiotemporal attention mechanism, used to assign weights to values ​​from different times and regions.

[0215] Next, the electronic device multiplies this similarity with the second combined feature, which is the result of Space-time Memory Read.

[0216] Next, the electronic device concatenates the result with the value output by the query encoding network to obtain the video frame concatenation feature, and then inputs the video frame concatenation feature into the final decoding network (Decoder) to obtain the mask object and the second loss function of the current frame.

[0217] Secondly Figure 8The training process of the video segmentation model shown also includes an auxiliary decoding network for the memory coding network. The electronic device can input each of the multiple first encoding results into its corresponding auxiliary decoding network to obtain a first loss function.

[0218] It should be noted that the aforementioned auxiliary decoding network only exists during the training of the video segmentation model. Once the target video segmentation model is trained and used to determine the mask object, the auxiliary decoding network can be removed. Therefore, when electronic devices use the target video segmentation model to determine the mask object, it enhances algorithm performance without introducing additional computational overhead.

[0219] In other words, by reconstructing and predicting the masked object of its input, the aforementioned auxiliary decoding network can be guided to improve its ability to distinguish between masked and non-masked objects, thereby further improving the segmentation accuracy of the target video segmentation model.

[0220] It is understood that, in practical implementation, the terminal / server described in the embodiments of this disclosure may include one or more hardware structures and / or software modules for implementing the aforementioned corresponding video segmentation model training method. These hardware structures and / or software modules can constitute an electronic device. Those skilled in the art should readily recognize that, based on the algorithm steps of the various examples described in conjunction with the embodiments disclosed herein, this disclosure can be implemented in hardware or a combination of hardware and computer software. Whether a function is implemented in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this disclosure.

[0221] Based on this understanding, the present disclosure also provides a video segmentation model training device that can be applied to electronic devices. Figure 9 A schematic diagram of the structure of the video segmentation model training apparatus provided in an embodiment of this disclosure is shown. Figure 9 As shown, the video segmentation model training device may include: an acquisition unit 901 and a processing unit 902;

[0222] The acquisition unit 901 is used to acquire second video information and multiple first video information; the second video information includes the current video frame of the video to be processed and a reference mask object annotated for the current video frame; each first video information includes: a historical video frame of the video to be processed and an actual mask object of the historical video frame;

[0223] The processing unit 902 is used to encode the historical video frames in each first video information based on the first coding network in the video segmentation model to be trained, so as to obtain multiple first coding results that correspond one-to-one with multiple first video information.

[0224] The processing unit 902 is further configured to perform decoding processing on each of the multiple first encoding results based on the auxiliary decoding network, so as to obtain multiple first decoding results that correspond one-to-one with the multiple first video information;

[0225] The processing unit 902 is further configured to determine a first loss function based on each of the multiple first decoding results and the actual mask object of the historical video frame in the first video information corresponding to each first decoding result;

[0226] The processing unit 902 is also used to perform video segmentation processing on the current video frame based on the current video frame segmentation sub-model and multiple first encoding results in the video segmentation model to be trained, to obtain the actual mask object of the current video frame, and to determine the second loss function based on the actual mask object of the current video frame and the reference mask object of the current video frame.

[0227] The processing unit 902 is also used to adjust the model parameters of the auxiliary decoding network and the video segmentation model to be trained based on the first loss function and the second loss function until the preset training termination condition is reached, and to use the video segmentation model at the end of training as the target video segmentation model; the target video segmentation model is used to determine the mask object of the video frame to be processed.

[0228] Optionally, the processing unit 902 is specifically used for:

[0229] Based on each first decoding result and the actual mask object of the historical video frame in the first video information corresponding to each first decoding result, determine multiple difference information that correspond one-to-one with multiple first decoding results;

[0230] Based on multiple differences and a preset loss function, the first loss function is determined.

[0231] Optionally, the processing unit 902 is specifically used for:

[0232] Based on the second coding network in the current video frame segmentation sub-model, the current video frame is encoded to obtain the second coding result;

[0233] Based on the second encoding result and multiple first encoding results, video frame stitching features are generated;

[0234] Based on the decoding network in the current video frame segmentation sub-model, the video frame splicing features are decoded to obtain a second decoding result, which indicates the actual mask object of the current video frame.

[0235] Optionally, the first encoding result includes: a first historical feature for identifying historical video frames and a second historical feature for storing detailed information of historical video frames; the detailed information of historical video frames is used to generate a reference mask object for historical video frames; the second encoding result includes: a first current feature for identifying the current video frame and a second current feature for storing detailed information of the current video frame; the detailed information of the current video frame is used to generate an actual mask object for the current video frame;

[0236] Processing unit 902 is specifically used for:

[0237] Based on the combination function, a first combination feature and a second combination feature are determined; the first combination feature includes: a plurality of first historical features that correspond one-to-one with a plurality of first encoding results; the second combination feature includes: a plurality of second historical features that correspond one-to-one with a plurality of first encoding results;

[0238] Determine the similarity between the first combined feature and the first current feature;

[0239] The product of similarity and the second combined feature is determined, and the product is concatenated with the second current feature to obtain the video frame concatenation feature.

[0240] Optionally, the processing unit 902 is specifically used for:

[0241] The historical video frames in each first video information are input into the coding layer in the first coding network to obtain multiple current coding results that correspond one-to-one with multiple first video information.

[0242] Each of the multiple current encoding results is input into the shared network output layer to obtain multiple first encoding results; the shared network output layer is the network output layer shared by the first encoding network and the encoding network in the current video frame segmentation sub-model; the network output layer includes: a convolutional output layer and / or a regularized output layer.

[0243] Optionally, the processing unit 902 is specifically used for:

[0244] The current video frame is input into the coding layer of the second coding network to obtain the current coding result of the current video frame;

[0245] The current encoding result is input into the shared network output layer to obtain the second encoding result; the shared network output layer is the network output layer shared by the first encoding network and the second encoding network; the network output layer includes: a convolutional output layer and / or a regularized output layer.

[0246] Optionally, the number of parameter types in the second coding network is less than the number of parameter types in the first coding network.

[0247] As described above, the embodiments of this disclosure can divide the electronic device into functional modules according to the above method examples. The integrated modules can be implemented in hardware or as software functional modules. Furthermore, it should be noted that the module division in these embodiments is illustrative and represents only one logical functional division; in actual implementation, other division methods may be used. For example, each function can be divided into its own functional modules, or two or more functions can be integrated into a single processing module.

[0248] The specific methods by which each module performs its operation and the beneficial effects of the video segmentation model training device in the above embodiments have been described in detail in the foregoing method embodiments, and will not be repeated here.

[0249] This disclosure also provides a terminal, which may be a user terminal such as a mobile phone or a computer. Figure 10 A schematic diagram of the structure of a terminal provided in an embodiment of this disclosure is shown. This terminal, which may be a video segmentation model training device, may include at least one processor 61, a communication bus 62, a memory 63, and at least one communication interface 64.

[0250] Processor 61 can be a central processing unit (CPU), a microprocessor unit, an ASIC, or one or more integrated circuits for controlling the execution of programs according to the present disclosure. As an example, combined with... Figure 9 The processing unit 902 in the electronic device performs the functions of... Figure 10 The processor 61 in it performs the same function.

[0251] The communication bus 62 may include a path for transmitting information between the aforementioned components.

[0252] Communication interface 64 uses any transceiver-like device for communicating with other devices or communication networks, such as servers, Ethernet, radio access networks (RAN), wireless local area networks (WLAN), etc. As an example,

[0253] Memory 63 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital versatile optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. Memory may exist independently and be connected to the processing unit via a bus. Memory may also be integrated with the processing unit.

[0254] The memory 63 stores the application code that executes the present invention, and its execution is controlled by the processor 61. The processor 61 executes the application code stored in the memory 63 to implement the functions of the method of the present invention.

[0255] In a specific implementation, as one example, processor 61 may include one or more CPUs, for example... Figure 10 CPU0 and CPU1 in the CPU.

[0256] In a specific implementation, as one example, the terminal may include multiple processors, for example... Figure 10 Processors 61 and 65 are included. Each of these processors may be a single-core (single-CPU) processor or a multi-core (multi-CPU) processor. A processor here may refer to one or more devices, circuits, and / or processing cores used to process data (e.g., computer program instructions).

[0257] In a specific implementation, as one embodiment, the terminal may further include an input device 66 and an output device 67. The input device 66 and output device 67 communicate and can accept user input in various ways. For example, the input device 66 may be a mouse, keyboard, touchscreen device, or sensing device. The output device 67 communicates with the processor 61 and can display information in various ways. For example, the output device 61 may be a liquid crystal display (LCD), a light emitting diode (LED) display device, etc.

[0258] Those skilled in the art will understand that Figure 10 The structure shown does not constitute a limitation on the terminal and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0259] This disclosure also provides a server. Figure 11 A schematic diagram of the server structure provided in this embodiment is shown. This server can be a video segmentation model training device. The server can vary significantly due to differences in configuration or performance, and may include one or more processors 71 and one or more memories 72. The memory 72 stores at least one instruction, which is loaded and executed by the processor 71 to implement the video segmentation model training method provided in the various method embodiments described above. Of course, the server may also have wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. The server may also include other components for implementing device functions, which will not be elaborated here.

[0260] This disclosure also provides a computer-readable storage medium including instructions stored thereon, which, when executed by a processor of a computer device, enable the computer to perform the video segmentation model training method provided in the embodiments described above. For example, the computer-readable storage medium may be a memory 63 including instructions, which may be executed by a processor 61 of a terminal to complete the method. As another example, the computer-readable storage medium may be a memory 72 including instructions, which may be executed by a processor 71 of a server to complete the method. Optionally, the computer-readable storage medium may be a non-transitory computer-readable storage medium, such as a ROM, RAM, CD-ROM, magnetic tape, floppy disk, or optical data storage device.

[0261] This disclosure also provides a computer program product including computer instructions that, when executed on an electronic device, cause the electronic device to perform the above-described actions. Figures 1-8 The training method for the video segmentation model shown in any of the attached figures.

[0262] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0263] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A video segmentation model training method, characterized in that, include: Obtain second video information and multiple first video information; The second video information includes the current video frame of the video to be processed and the reference mask object labeled for the current video frame; Each first video information includes: a historical video frame of the video to be processed and the actual mask object of the historical video frame; Based on the first encoding network in the video segmentation model to be trained, the historical video frames in each first video information are encoded to obtain multiple first encoding results that correspond one-to-one with the multiple first video information. Based on the auxiliary decoding network, each of the plurality of first encoding results is decoded to obtain a plurality of first decoding results that correspond one-to-one with the plurality of first video information; Based on each of the plurality of first decoding results, and the actual mask object of the historical video frame in the first video information corresponding to each of the first decoding results, a first loss function is determined; Based on the current video frame segmentation sub-model in the video segmentation model to be trained and the multiple first encoding results, the current video frame is segmented to obtain the actual mask object of the current video frame, and based on the actual mask object of the current video frame and the reference mask object of the current video frame, a second loss function is determined. The model parameters of the auxiliary decoding network and the video segmentation model to be trained are adjusted based on the first loss function and the second loss function until the preset training termination condition is met, and the video segmentation model at the end of training is used as the target video segmentation model; the target video segmentation model is used to determine the mask object of the video frame to be processed.

2. The video segmentation model training method according to claim 1, characterized in that, The step of determining a first loss function based on each of the plurality of first decoding results and the actual mask object of the historical video frame in the first video information corresponding to each first decoding result includes: Based on each of the first decoding results and the actual mask object of the historical video frame in the first video information corresponding to each of the first decoding results, a plurality of difference information corresponding one-to-one with the plurality of first decoding results is determined; Based on the multiple difference information and the preset loss function, the first loss function is determined.

3. The video segmentation model training method according to claim 1, characterized in that, The process of segmenting the current video frame based on the current video frame segmentation sub-model in the video segmentation model to be trained and the plurality of first encoding results to obtain the actual mask object of the current video frame includes: Based on the second coding network in the current video frame segmentation sub-model, the current video frame is encoded to obtain the second coding result; Based on the second encoding result and multiple first encoding results, video frame stitching features are generated; Based on the decoding network in the current video frame segmentation sub-model, the video frame splicing features are decoded to obtain a second decoding result, which indicates the actual mask object of the current video frame.

4. The video segmentation model training method according to claim 3, characterized in that, The first encoding result includes: a first historical feature for identifying the historical video frame and a second historical feature for storing detailed information of the historical video frame; the detailed information of the historical video frame is used to generate a reference mask object for the historical video frame; the second encoding result includes: a first current feature for identifying the current video frame and a second current feature for storing detailed information of the current video frame; the detailed information of the current video frame is used to generate an actual mask object for the current video frame; Based on the second encoding result and multiple first encoding results, video frame stitching features are generated, including: Based on the combination function, a first combination feature and a second combination feature are determined; the first combination feature includes a plurality of first historical features that correspond one-to-one with the plurality of first encoding results; the second combination feature includes a plurality of second historical features that correspond one-to-one with the plurality of first encoding results. Determine the similarity between the first combined feature and the first current feature; The product of the similarity and the second combined feature is determined, and the product is concatenated with the second current feature to obtain the video frame concatenation feature.

5. The video segmentation model training method according to any one of claims 1-4, characterized in that, The first encoding network in the video segmentation model to be trained encodes historical video frames in each first video information to obtain multiple first encoding results corresponding one-to-one with the multiple first video information, including: The historical video frames in each of the first video information are input into the coding layer in the first coding network to obtain multiple current coding results that correspond one-to-one with the multiple first video information; Each of the plurality of current encoding results is input into a shared network output layer to obtain the plurality of first encoding results; the shared network output layer is a network output layer shared by the first encoding network and the encoding network in the current video frame segmentation sub-model; the network output layer includes: a convolutional output layer and / or a regularized output layer.

6. The video segmentation model training method according to claim 3 or 4, characterized in that, The second encoding network in the current video frame segmentation sub-model is used to encode the current video frame to obtain a second encoding result, including: The current video frame is input into the coding layer of the second coding network to obtain the current coding result of the current video frame; The current encoding result is input into the shared network output layer to obtain the second encoding result; the shared network output layer is a network output layer shared by the first encoding network and the second encoding network; the network output layer includes: a convolutional output layer and / or a regularized output layer.

7. The video segmentation model training method according to claim 3 or 4, characterized in that, The number of parameter types in the second encoding network is less than the number of parameter types in the first encoding network.

8. A video segmentation model training device, characterized in that, include: Acquisition unit and processing unit; The acquisition unit is used to acquire second video information and multiple first video information; The second video information includes the current video frame of the video to be processed and a reference mask object labeled for the current video frame; each piece of first video information includes: a historical video frame of the video to be processed and the actual mask object of the historical video frame; The processing unit is used to encode the historical video frames in each first video information based on the first encoding network in the video segmentation model to be trained, so as to obtain multiple first encoding results that correspond one-to-one with the multiple first video information. The processing unit is further configured to perform decoding processing on each of the plurality of first encoding results based on the auxiliary decoding network, so as to obtain a plurality of first decoding results that correspond one-to-one with the plurality of first video information; The processing unit is further configured to determine a first loss function based on each of the plurality of first decoding results and the actual mask object of the historical video frame in the first video information corresponding to each of the first decoding results; The processing unit is further configured to perform video segmentation processing on the current video frame based on the current video frame segmentation sub-model in the video segmentation model to be trained and the plurality of first encoding results, to obtain the actual mask object of the current video frame, and to determine a second loss function based on the actual mask object of the current video frame and the reference mask object of the current video frame. The processing unit is further configured to adjust the model parameters of the auxiliary decoding network and the video segmentation model to be trained based on the first loss function and the second loss function until a preset training termination condition is met, and to use the video segmentation model at the end of training as the target video segmentation model; the target video segmentation model is used to determine the mask object of the video frame to be processed.

9. An electronic device, characterized in that, The electronic device includes: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the video segmentation model training method as described in any one of claims 1-7.

10. A computer-readable storage medium storing instructions thereon, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is enabled to perform the video segmentation model training method as described in any one of claims 1-7.

Images