Target segmentation method and device for video stream, electronic equipment and storage medium

Abstract

The invention relates to the technical field of image target segmentation, in particular to a target segmentation method and device for a video stream, electronic equipment and a storage medium, which are used for solving the problem of unstable target segmentation precision, and the method comprises the following steps: fusing a target frame image and a target segmentation image of a previous frame image of the target frame image to obtain a target segmentation image; inputting the fused image into a target segmentation model to obtain a target segmentation map of the target frame image; wherein the target frame image is a non-first frame image in the video stream to be displayed; the target segmentation model is obtained by training a to-be-trained segmentation model based on a plurality of sample video streams, and in the training process, a target image input into the segmentation model is a sample image in any one sample video stream in the plurality of sample video streams, or an image obtained by fusing the sample image and a sample segmentation image, and the target image input into the target segmentation model is a target image in any one sample video stream in the plurality of sample video streams. The sample segmentation image is a randomly generated noise segmentation image or a prediction segmentation image of a previous frame of sample image of the sample image; therefore, the target segmentation precision is improved.

Description

Technical Field

[0001] The present application relates to the technical field of image target segmentation, and in particular to a method, device, electronic device and storage medium for target segmentation of a video stream. Background Art

[0002] Object segmentation aims to identify the target object in the image and obtain its pixel-level segmentation information. The object segmentation model in the video stream can learn the feature representation of the target object in the video stream from a large amount of labeled data by using a deep neural network. These feature representations can be used to perform pixel-level segmentation on each frame of the video stream.

[0003] However, in actual use, due to the characteristics of video streams, pixel changes between consecutive frames or changes in the position of the target object, the segmentation results of the model become unstable, resulting in a large difference between the segmentation results of the previous and next frames, and unable to accurately perform stable pixel-level segmentation. Therefore, the segmentation accuracy of the model in the target segmentation task of the video stream is seriously reduced. Summary of the invention

[0004] The embodiments of the present application provide a method, device, electronic device and storage medium for object segmentation of a video stream, so as to improve the segmentation accuracy of a model in the object segmentation task of a video stream.

[0005] The specific technical solutions provided by the embodiments of this application are as follows:

[0006] In a first aspect, an embodiment of the present application provides a method for object segmentation of a video stream, comprising:

[0007] Fusing a target frame image with a target segmentation map of a frame image before the target frame image to obtain a fused image, wherein the target frame image is a non-first frame image in a video stream to be displayed, and the target segmentation map of the first frame image in the video stream is obtained by inputting the first frame image into a target segmentation model;

[0008] Inputting the fused image into the target segmentation model to obtain a target segmentation map of the target frame image;

[0009] In which, the target segmentation model is obtained by training the segmentation model to be trained based on multiple sample video streams. During the training process, the target image input into the segmentation model is a sample image in any one of the multiple sample video streams, or an image after the sample image and the sample segmentation map are fused. The sample segmentation map is a randomly generated noise segmentation map or a predicted segmentation map of a sample image in the previous frame of the sample image.

[0010] By adopting the above method in the embodiment of the present application, a single-frame image is used as the model input, which improves the actual running speed of the target segmentation model without bringing more running time consumption and computing consumption, thereby expanding the scope of application of the model and being widely applicable to various hardware platforms; further, in the model training process, the target image input to the segmentation model adopts a variety of forms, such as a sample image (i.e., the current frame image), an image after the sample image is fused with a randomly generated noise segmentation map, and an image after the sample image is fused with the predicted segmentation map of the previous frame sample image. In this way, in model training, the model may not rely too much on the predicted segmentation map of the previous frame image, but pay more attention to the image features of the current frame image, so that the model can make a more stable and continuous segmentation result for the video stream, thereby improving the segmentation accuracy of the model.

[0011] In a possible implementation, fusing the target frame image with a target segmentation map of a frame image before the target frame image to obtain a fused image includes:

[0012] The following operations are performed on each pixel in the target frame image respectively:

[0013] For any pixel among the pixels, taking the pixel value of any pixel in the target segmentation image of the previous frame image as a gain coefficient, and performing a multiplication operation with the channel value of any pixel in the target frame image to obtain a first value;

[0014] The channel value of any pixel is replaced by the first value to obtain the fused image.

[0015] The above method uses the pixel value of each pixel in the target segmentation map of the previous frame image as the gain coefficient. On the basis of focusing on the image feature extraction of the target frame image, it can fuse the segmentation result of the previous frame image, thereby enhancing the model's attention to the segmented area of ​​the previous frame in the target frame image, and making the target segmentation map output by the model for the target frame image more stable and continuous, so as to meet the dual requirements of segmentation accuracy and performance in the target segmentation task for the video stream.

[0016] In a possible implementation, after obtaining the target segmentation map of any frame of the image in the video stream, the method further includes:

[0017] Based on the target segmentation map of the arbitrary frame image, each pixel corresponding to the target object in the target segmentation map is enhanced, and the arbitrary frame image after the enhancement process is displayed.

[0018] The above method can obtain the specific pixel position of the target object according to the pixel value of the target segmentation map, so as to call the image quality processing algorithm in a targeted manner and process the target object to improve the image quality and user experience.

[0019] In a possible implementation, the target segmentation model is trained in the following manner:

[0020] The segmentation model is trained for multiple rounds of iterations based on the multiple sample video streams. When the total loss value of the input sample video streams is less than the loss threshold, the segmentation model is output to obtain the target segmentation model, wherein in one round of iterative training:

[0021] The following operations are performed for any one of the multiple sample video streams:

[0022] Inputting the first frame sample image in any one of the sample video streams into the segmentation module to obtain a predicted segmentation map of the first frame sample image;

[0023] For any frame of sample image except the first frame of sample image in any one of the sample video streams, based on a target image determination rule randomly selected from a plurality of preset target image determination rules and the any frame of sample image, a target sample image is determined, the target sample image is input into the segmentation model, and a predicted segmentation map corresponding to the any frame of sample image is obtained;

[0024] Based on the predicted allocation map and the true segmentation map corresponding to each sample image, obtaining a total loss value of any sample video stream;

[0025] Model parameters of the segmentation model are adjusted based on the total loss value.

[0026] In a possible implementation, determining the target sample image based on the target image determination rule randomly selected from a plurality of preset target image determination rules and the arbitrary frame sample image includes:

[0027] If the randomly selected target image determination rule is the first rule, taking the arbitrary frame of sample image as the target sample image;

[0028] If the randomly selected target image determination rule is the second rule, the predicted segmentation map of the previous frame sample image of the arbitrary frame sample image is fused with the arbitrary frame sample image to obtain the target sample image;

[0029] If the randomly selected target image determination rule is the third rule, a noise segmentation map is randomly generated, and the noise segmentation map is fused with the arbitrary frame sample image to obtain the target sample image.

[0030] The above method can add unstable segmentation results during the model training process by setting a random mechanism, so that the model can learn the characteristics of the unstable segmentation result of the previous frame image during the training process, and then the model can not rely too much on the predicted segmentation map of the previous frame, and pay more attention to the image feature extraction of the current frame image, thereby improving the robustness of the model and improving the segmentation accuracy of the model; further, during the model training process, when the predicted segmentation result of the previous frame is good, it can also promote the model to make a continuous and stable segmentation prediction result for the current frame image and the previous one, further improving the segmentation accuracy of the model.

[0031] In a possible implementation, the target sample segmentation map is a predicted segmentation map of a previous frame sample image of the arbitrary frame sample image or the noise segmentation map, and the target sample segmentation map is fused with the arbitrary frame sample image to obtain the target sample image, including:

[0032] The following operations are performed for each sample pixel in any one frame of sample image:

[0033] For any sample pixel among the sample pixels, taking the pixel value of any sample pixel in the target sample segmentation image as a gain coefficient, and performing a multiplication operation with the channel value of any sample pixel in the any frame sample image to obtain a second value;

[0034] The channel value of any sample pixel is replaced by the second value to obtain the target sample image.

[0035] In a second aspect, an embodiment of the present application provides a target segmentation device for a video stream, comprising:

[0036] A fusion module is used to fuse a target frame image with a target segmentation map of a frame image before the target frame image to obtain a fused image, wherein the target frame image is a non-first frame image in the video stream to be displayed, and the target segmentation map of the first frame image in the video stream is obtained by inputting the first frame image into a target segmentation model;

[0037] A segmentation module, used for inputting the fused image into the target segmentation model to obtain a target segmentation map of the target frame image;

[0038] In which, the target segmentation model is obtained by training the segmentation model to be trained based on multiple sample video streams. During the training process, the target image input into the segmentation model is a sample image in any one of the multiple sample video streams, or an image after the sample image and the sample segmentation map are fused. The sample segmentation map is a randomly generated noise segmentation map or a predicted segmentation map of a sample image in the previous frame of the sample image.

[0039] In a possible implementation, the fusion module is specifically used to:

[0040] The following operations are performed on each pixel in the target frame image respectively:

[0041] For any pixel among the pixels, taking the pixel value of any pixel in the target segmentation image of the previous frame image as a gain coefficient, and performing a multiplication operation with the channel value of any pixel in the target frame image to obtain a first value;

[0042] The channel value of any pixel is replaced by the first value to obtain the fused image.

[0043] In a possible implementation, after obtaining the target segmentation map of any frame of the video stream, the segmentation module is further used to:

[0044] Based on the target segmentation map of the arbitrary frame image, each pixel corresponding to the target object in the target segmentation map is enhanced, and the arbitrary frame image after the enhancement process is displayed.

[0045] In a possible implementation, the segmentation module is used to train the target segmentation model in the following manner:

[0046] The segmentation model is trained for multiple rounds of iterations based on the multiple sample video streams. When the total loss value of the input sample video streams is less than the loss threshold, the segmentation model is output to obtain the target segmentation model, wherein in one round of iterative training:

[0047] The following operations are performed for any one of the multiple sample video streams:

[0048] Inputting the first frame sample image in any one of the sample video streams into the segmentation module to obtain a predicted segmentation map of the first frame sample image;

[0049] For any frame of sample image except the first frame of sample image in any one of the sample video streams, based on a target image determination rule randomly selected from a plurality of preset target image determination rules and the any frame of sample image, a target sample image is determined, the target sample image is input into the segmentation model, and a predicted segmentation map corresponding to the any frame of sample image is obtained;

[0050] Based on the predicted allocation map and the true segmentation map corresponding to each sample image, obtaining a total loss value of any sample video stream;

[0051] Model parameters of the segmentation model are adjusted based on the total loss value.

[0052] In a possible implementation, the segmentation module is specifically used to:

[0053] If the randomly selected target image determination rule is the first rule, taking the arbitrary frame of sample image as the target sample image;

[0054] If the randomly selected target image determination rule is the second rule, the predicted segmentation map of the previous frame sample image of the arbitrary frame sample image is fused with the arbitrary frame sample image to obtain the target sample image;

[0055] If the randomly selected target image determination rule is the third rule, a noise segmentation map is randomly generated, and the noise segmentation map is fused with the arbitrary frame sample image to obtain the target sample image.

[0056] In a possible implementation, the target sample segmentation map is a predicted segmentation map of a previous frame of sample image of any frame of sample image or the noise segmentation map, and the segmentation module is used to:

[0057] The following operations are performed for each sample pixel in any one frame of sample image:

[0058] For any sample pixel among the sample pixels, taking the pixel value of any sample pixel in the target sample segmentation image as a gain coefficient, and performing a multiplication operation with the channel value of any sample pixel in the any frame sample image to obtain a second value;

[0059] The channel value of any sample pixel is replaced by the second value to obtain the target sample image.

[0060] In a third aspect, an embodiment of the present application provides an electronic device, including:

[0061] Memory, used to store computer programs or instructions;

[0062] A processor is used to execute the computer program or instructions in the memory so that any method in the first aspect described above is executed.

[0063] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium. When instructions in the storage medium are executed by a processor, the processor is enabled to execute any one of the methods described in the first aspect above.

[0064] In addition, the technical effects brought about by any one of the implementation methods in the second to fourth aspects can refer to the technical effects brought about by different implementation methods in the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 This is a schematic diagram of an application scenario in an embodiment of the present application;

[0066] Figure 2 This is another schematic diagram of an application scenario in an embodiment of the present application;

[0067] Figure 3 A schematic diagram of a flow chart of a method for object segmentation of a video stream in an embodiment of the present application;

[0068] Figure 4 This is a schematic diagram of a network architecture of a target segmentation model in an embodiment of the present application;

[0069] Figure 5 A schematic diagram of a flow chart of a method for obtaining a fused image in an embodiment of the present application;

[0070] Figure 6 A flow chart of a training method for a segmentation model in an embodiment of the present application;

[0071] Figure 7 A flowchart of a specific training method for a segmentation model in an embodiment of the present application;

[0072] Figure 8 A schematic diagram of a process for obtaining a target sample image in an embodiment of the present application;

[0073] Fig. 9 A schematic diagram of a specific training process in a model training stage in an embodiment of the present application;

[0074] Fig.10 This is a specific flow chart of a specific implementation stage in the embodiment of the present application;

[0075] Fig.11 This is a schematic diagram of the logical architecture of a device for segmenting a video stream in an embodiment of the present application;

[0076] Fig.12 It is a schematic diagram of the physical structure of the electronic device in the embodiment of the present application. DETAILED DESCRIPTION

[0077] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0078] It should be noted that the terms "first", "second", "third", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein.

[0079] The following is a brief introduction to the design concept of the embodiments of the present application.

[0080] The present application relates to the technical field of image target segmentation, and mainly to a method, device, electronic device and storage medium for target segmentation of a video stream.

[0081] At present, the target segmentation model in the video stream can learn the feature representation of the target object in the video stream from a large amount of labeled data by using a deep neural network. These feature representations can be used to perform pixel-level segmentation on each frame of the video stream. However, in actual use, due to the characteristics of the video stream, the pixel changes between consecutive frames or the position changes of the target object, the segmentation results of the model become unstable, resulting in a large difference between the segmentation results of the previous and next frames, and it is impossible to accurately perform stable pixel-level segmentation. Therefore, the segmentation accuracy of the model in the target segmentation task of the video stream is seriously reduced.

[0082] In the existing technology, in order to make the model predict the segmentation results of the target object in consecutive frames stably and continuously, most solutions use multi-frame data as model input, so that the model can learn the association between objects in multiple frames and make stable predictions. However, in this solution, due to the use of multi-frame input models, the model consumes a lot of computation and its scope of application is greatly limited.

[0083] In some schemes that use single-frame data as model input, the segmentation result of the previous frame is often used as a reference, and the segmentation result of the current frame is fused with the segmentation result of the previous frame according to algorithms such as optical flow to obtain a more stable segmentation result of the video stream. However, in this scheme, since the motion relationship between the two frames needs to be estimated, such as optical flow information, a large amount of computational consumption is also brought about. At the same time, since the segmentation result of the next frame is based on the segmentation result of the previous frame, if the segmentation result of the previous frame is wrong, it will directly affect the segmentation accuracy of the next frame.

[0084] In view of this, in order to solve the problems of unstable segmentation accuracy and high computational consumption of traditional target segmentation models in target segmentation tasks of video streams, an embodiment of the present application provides a target segmentation method for video streams. In the embodiment of the present application, a target frame image and a target segmentation map of a previous frame image of the target frame image are fused to obtain a fused image, the target frame image is a non-first frame image in the video stream to be displayed, and the target segmentation map of the first frame image in the video stream is obtained by inputting the first frame image into the target segmentation model; then, the fused image is input into the target segmentation model to obtain a target segmentation map of the target frame image; wherein the target segmentation model is obtained by training a segmentation model to be trained based on multiple sample video streams, and during the training process, the target image input into the segmentation model is a sample image in any one of the multiple sample video streams, or an image after the sample image and the sample segmentation map are fused, and the sample segmentation map is a randomly generated noise segmentation map or a predicted segmentation map of a sample image of the previous frame of the sample image.

[0085] In this way, a target segmentation method for a video stream provided in an embodiment of the present application is adopted, and a single-frame image is used as the model input, so that the actual running speed of the target segmentation model is improved, without bringing more running time consumption and computing consumption, thereby expanding the scope of application of the model; further, in the model training process, the target image input to the segmentation model adopts a variety of forms, such as a sample image (i.e., the current frame image), an image after the sample image is fused with a randomly generated noise segmentation map, and an image after the sample image is fused with the predicted segmentation map of the previous frame sample image. In this way, in model training, the model may not rely too much on the predicted segmentation map of the previous frame image, but pay more attention to the image features of the current frame image, so that the model can make a more stable and continuous segmentation result for the video stream, thereby improving the segmentation accuracy of the model.

[0086] A method for target segmentation of a video stream provided in an embodiment of the present application is suitable for an electronic device. The electronic device uses the above method to implement the segmentation of the target object of each frame image of the video stream to be displayed, and obtains a more stable and continuous segmentation result, so that when the image quality is subsequently adjusted, the position of each pixel in the area where the target object is located can be determined according to the pixel value of the target segmentation result of each frame image of the video stream, and the image quality processing algorithm can be called in a targeted manner to adjust the image quality of the target object or other areas, such as using an image enhancement algorithm to perform targeted image quality enhancement processing on the target object.

[0087] Exemplarily, the electronic device in the embodiments of the present application may be an electronic device having an image display device (e.g., a display panel); for example, it may be a smart terminal, a smart mobile terminal, a tablet computer, a laptop computer, a smart handheld device, a personal computer (PC), a computer, a smart screen, a display device, a vehicle-mounted device, various wearable devices, a personal digital assistant (PDA), etc.; among which, wearable devices include virtual reality (VR) devices, augmented reality (AR) devices, etc.

[0088] As another example, the electronic device may also be a server connected to a device with a display function (such as a display device). For example, the server may be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, and a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms; wherein the server may be connected to the device with a display function by wire or wireless means, and this application does not limit the connection method.

[0089] It is to be understood that the specific type of the above electronic device is not limited in the present application.

[0090] The following introduces possible application scenarios of an optional object segmentation method for a video stream provided in an embodiment of the present application in conjunction with the accompanying drawings.

[0091] like Figure 1 As shown, a schematic diagram of an application scenario in an embodiment of the present application is shown by taking the electronic device as a display device as an example. Figure 1 As shown, the application scenario includes a display device 10; the display device 10 executes the process of the target segmentation method of the video stream provided in the embodiment of the present application (the specific process will be introduced later), obtains the target segmentation map of each frame image of the video stream, and according to the target segmentation map of each frame image of the video stream, preferably enhances each frame image, and displays each frame image after the enhanced processing, etc.

[0092] like Figure 2 As shown, another application scenario schematic diagram in the embodiment of the present application is shown by taking the electronic device as a server as an example. Figure 2As shown, the application scenario includes a display device 10 and a server 20. The server 20 executes the process of the target segmentation method of the video stream provided in the embodiment of the present application (the specific process will be described later), obtains the target segmentation map of each frame image of the video stream, and preferably enhances each frame image according to the target segmentation map of each frame image of the video stream, and then transmits each frame image after the enhanced processing to the display device 10 through the communication network, so that the display device 10 displays each frame image after the enhanced processing, etc.; or, the server 20 directly transmits each frame image and the target segmentation map of each frame image to the display device 10 through the communication network, so that the display device 10 preferably enhances each frame image according to the target segmentation map of each frame image of the video stream, and displays each frame image after the enhanced processing, etc.

[0093] Of course, the method provided in the embodiment of the present application is not limited to Figure 1 , Figure 2 The application scenario shown can also be used in other possible application scenarios, and the image processing method is not limited to the enhancement processing method, but can also include any other form of image quality processing method, which is not limited in the embodiments of the present application.

[0094] After introducing the application scenarios of the embodiments of the present application, the preferred implementation modes of the present application are further described in detail below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application. In addition, the embodiments of the present application and the features in the embodiments may be combined with each other if there is no conflict.

[0095] See also Figure 3 As shown, the embodiment of the present application provides a method for object segmentation of a video stream, and the specific process of the method is as follows:

[0096] Step 300: Fuse the target frame image and the target segmentation map of the previous frame image of the target frame image to obtain a fused image, wherein the target frame image is a non-first frame image in the video stream to be displayed, and the target segmentation map of the first frame image in the video stream is obtained by inputting the first frame image into the target segmentation model.

[0097] In the embodiment of the present application, the target segmentation model can be a hierarchical multi-label classification neural network, and the network architecture diagram of the neural network is as follows: Figure 4 See Figure 4As shown, the network architecture includes a feature extraction subnetwork and sigmoid, wherein the feature extraction subnetwork can be any feature extraction network, such as UNet, U2Net, MobileNet, YOLO, etc.; the last layer of the feature extraction subnetwork is followed by sigmoid, and the image features extracted by the feature extraction subnetwork are output through sigmoid to form a segmentation map of the input image, and the segmentation map is preferably a binary map, 0 represents the background, and 1 represents the target object.

[0098] In an embodiment of the present application, the above-mentioned target segmentation model is obtained by training the segmentation model to be trained based on multiple sample video streams (the specific model training process is detailed in the subsequent process). During the training process, the target image input into the segmentation model is a sample image in any one of the multiple sample video streams, or an image after the sample image and the sample segmentation map are fused, wherein the sample segmentation map is a randomly generated noise segmentation map or a predicted segmentation map of the sample image of the previous frame of the sample image.

[0099] In the embodiment of the present application, during the model training process of the target segmentation model, the target image input to the segmentation model is a sample image in any one of the multiple sample video streams, or an image after the sample image and the sample segmentation map are fused, wherein the sample segmentation map is a randomly generated noise segmentation map or a predicted segmentation map of the sample image of the previous frame of the sample image, that is, the situation where the segmentation result is unstable is added during the model training process. Therefore, the target segmentation model learns the characteristics when the segmentation result of the previous frame image is unstable during the training process. The segmentation map output by the target segmentation model does not rely on the predicted segmentation map of the previous frame, but focuses more on the image feature extraction of the current frame image (i.e., the target frame image), thereby improving the segmentation accuracy of the target segmentation model, so that a more stable and continuous segmentation result can be obtained for the video stream; further, a single-frame input target segmentation model is used to perform target object segmentation processing, thereby improving the actual running speed.

[0100] After the segmentation model to be trained is trained to obtain the above target segmentation model, in the specific implementation, a display device such as a TV playing a video screen is taken as an example. After the TV is turned on, the first frame image of the received video stream is directly input into the feature extraction subnetwork in the target segmentation model, and the feature extraction subnetwork extracts image features from the first frame image. Then, the extracted image features are passed through the sigmoid in the target segmentation model to output the target segmentation map of the first frame image.

[0101] Then, when receiving each subsequent frame image of the video stream, step 300 is executed respectively to obtain the target segmentation map of each subsequent frame image; wherein each non-first frame image received is recorded as the target frame image. Figure 5As shown, the following steps are specifically performed for each pixel in the target frame image:

[0102] Step 400: For any pixel among the pixels, use the pixel value of the pixel in the target segmentation image of the previous frame image as a gain coefficient, and perform a multiplication operation with the channel value of the pixel in the target frame image to obtain a first value.

[0103] In the embodiment of the present application, the video stream to be displayed preferably adopts the YUV encoding format. Other forms of encoding formats may also be used in practical applications. The corresponding scheme can also be customized according to the design idea of ​​the present application and the specific encoding format adopted by the image, so as to achieve the method of obtaining the target sample image by using the pixel value of each sample pixel in the target sample segmentation map as the gain coefficient in the present application. It can be understood that the present application does not make any specific limitation on this.

[0104] Specifically, when executing step 400, the channel value of any pixel in the target frame image, that is, the first value of the preset channel (preferably the Y channel) of any pixel in the fused image, can be obtained by the following formula:

[0105]

[0106] Among them, (i, j) is the image coordinate of the pixel point, is the pixel value of the predicted segmentation map of the previous frame image at pixel point (i, j), is the channel value of the Y channel of the target frame image at the pixel point (i, j). When the pixel point (i, j) is the target point (that is, the pixel point represents the target object), is 1, To achieve a 2x effect, when the pixel point (i, j) is a background point (i.e., the pixel point represents the background), If it is 0, it will not be Value affects.

[0107] In this way, by taking the pixel value of each pixel in the target segmentation map of the previous frame image as the gain coefficient, the segmentation result of the previous frame image can be fused on the basis of focusing on the image feature extraction of the target frame image, thereby enhancing the model's attention to the segmented area of ​​the previous frame in the target frame image, and making the target segmentation map output by the model for the target frame image more stable and continuous, so as to meet the dual requirements of segmentation accuracy and performance in the target segmentation task for the video stream.

[0108] Step 410: Replace the channel value of the pixel with the first value to obtain a fused image.

[0109] In an embodiment of the present application, after executing step 400 to obtain the first value of a preset channel (e.g., Y channel) of any pixel in the target frame image, step 410 is executed to replace the channel value of the preset channel (e.g., Y channel) of the pixel in the target frame image with the first value.

[0110] After replacing the channel value of each pixel included in the target frame image, the aforementioned fused image is obtained.

[0111] Step 310: Input the fused image into the target segmentation model to obtain a target segmentation map of the target frame image.

[0112] In the embodiment of the present application, when executing step 310, the fused image is input into the feature extraction subnetwork of the target segmentation model, the feature extraction subnetwork extracts image features from the fused image, and obtains the target segmentation map of the target frame image through sigmoid output.

[0113] In this way, by executing steps 300 to 310, a target segmentation map of each frame of the video stream to be displayed can be obtained, so that subsequent image processing can be performed based on the target segmentation map of each frame of the image.

[0114] In some feasible embodiments, after obtaining the target segmentation map of any frame image in the video stream, each pixel corresponding to the target object in the target segmentation map can be enhanced based on the target segmentation map of the image, and the enhanced image can be displayed.

[0115] In this way, the specific pixel position of the target object can be obtained according to the pixel value of the target segmentation map, so that the image quality processing algorithm can be called in a targeted manner to process the target object to improve the image quality and user experience.

[0116] After introducing a method for object segmentation of a video stream in an embodiment of the present application, the training process of the object segmentation model in the embodiment of the present application is described. The training process in the embodiment of the present application can be applied to a server, such as Figure 2 The server 20 is shown.

[0117] In the embodiment of the present application, before training the model, first, a network architecture of the segmentation model to be trained is constructed, such as Figure 4 Then, the specific training process of the segmentation model is described below. Figure 6 As shown, the training process of the segmentation model in the embodiment of the present application is as follows:

[0118] Step 600: Acquire multiple sample video streams.

[0119] In an embodiment of the present application, in a target segmentation scenario of a video stream, each sample video stream includes multiple frames of sample images and a true segmentation map of each frame of sample image, wherein the true segmentation map may also preferably be a binary map, where 0 represents the background and 1 represents the target object.

[0120] Step 610: Perform multiple rounds of iterative training on the segmentation model based on multiple sample video streams.

[0121] In the present application, see Figure 7 As shown, in step 610, in one round of iterative training, the following steps are performed for any one of the multiple sample video streams:

[0122] Step 6101: input the first frame sample image in the sample video stream into the segmentation module to obtain a predicted segmentation map of the first frame sample image.

[0123] In an embodiment of the present application, the input for model training is sample video stream data. When the sample video stream is used for the first time to train the segmentation model, step 6101 is executed to directly input the first frame sample image in the sample video stream into the feature extraction subnetwork of the segmentation model. The feature extraction subnetwork performs image feature extraction on the first frame sample image. The extracted image features are passed through sigmoid to output a predicted segmentation map of the first frame sample image, wherein the predicted segmentation map is a binary map, where 0 represents the background and 1 represents the target object.

[0124] Step 6102: For any sample image frame except the first sample image frame in the sample video stream, determine the target sample image based on a target image determination rule randomly selected from a plurality of preset target image determination rules and the sample image.

[0125] In the embodiment of the present application, after executing step 6101, step 6102 is executed for any frame of sample image in the sample video stream except the first frame of sample image.

[0126] In order to solve the problem that the model makes an error in predicting the previous frame of the video stream, resulting in errors in the prediction of the second frame and even subsequent frames, a random mechanism is pre-set to randomly determine the target sample image in the input segmentation model to improve the robustness of the model and improve the segmentation accuracy of the model.

[0127] Specifically, the above-mentioned random mechanism includes multiple target image determination rules, which are recorded as the first rule, the second rule and the third rule; among which, the first rule is to use the sample image as the target sample image, that is, without performing a fusion operation on the sample image, the sample image is directly input into the segmentation model for model training; the second rule is to fuse the predicted segmentation map of the previous frame sample image of the sample image with the sample image to obtain the target sample image, that is, to input the target sample image obtained by fusion with the segmentation map of the previous frame sample image into the segmentation model for model training; the third rule is to fuse the randomly generated noise segmentation map with the sample image to obtain the target sample image, that is, to input the target sample image obtained by fusion with the randomly generated noise segmentation map into the segmentation model for model training.

[0128] Then, before executing step 6102, a target image determination rule is randomly selected from a plurality of preset target image determination rules, and then step 6102 is executed.

[0129] In the specific implementation, when executing step 6102, the following three situations are specifically included but not limited to:

[0130] Case 1: If the randomly selected target image determination rule is the first rule, the sample image is used as the target sample image.

[0131] Case 2: If the randomly selected target image is determined by the second rule, the predicted segmentation map of the previous frame of the sample image is fused with the sample image to obtain the target sample image.

[0132] Case three: if the randomly selected target image determination rule is the third rule, a noise segmentation map is randomly generated, and the noise segmentation map is fused with the sample image to obtain the target sample image.

[0133] In an embodiment of the present application, by setting the above-mentioned random mechanism, the situation where the segmentation result is unstable can be added during the model training process, so that the model can learn the characteristics when the segmentation result of the previous frame image is unstable during the training process, and then the model does not rely too much on the predicted segmentation map of the previous frame, and pays more attention to the image feature extraction of the current frame image, thereby improving the robustness of the model and improving the segmentation accuracy of the model; further, during the model training process, when the predicted segmentation result of the previous frame is good, it can also promote the model to make a continuous and stable segmentation prediction result for the current frame image and the previous frame, thereby further improving the segmentation accuracy of the model.

[0134] In the embodiment of the present application, assuming that the target sample segmentation map is a predicted segmentation map of a sample image of a previous frame of the sample image or the aforementioned noise segmentation map, when performing the aforementioned situation 2 or situation 3 to fuse the target sample segmentation map with the sample image to obtain the target sample image, refer to Figure 8 As shown, the following steps are specifically performed for each sample pixel in the sample image:

[0135] Step 800: For any sample pixel among the sample pixels, use the pixel value of the sample pixel in the target sample segmentation image as a gain coefficient, and perform a multiplication operation with the channel value of the sample pixel in the sample image to obtain a second value.

[0136] In the embodiment of the present application, the sample video stream preferably adopts the YUV encoding format. In practical applications, other forms of encoding formats may also be used. It is also possible to customize the corresponding scheme according to the design concept of the present application and the specific encoding format used by the image, so as to achieve the method of using the pixel value of each sample pixel in the target sample segmentation map as the gain coefficient in the present application to obtain the target sample image. It can be understood that the present application does not make any specific limitation on this.

[0137] Specifically, when executing step 800, the value of a preset channel (preferably the Y channel) of the target sample image corresponding to any sample image is calculated based on the formula used in the aforementioned step 400 (see the above content for details, which will not be repeated here), and is recorded as the second value.

[0138] Step 810: Replace the channel value of the sample pixel with the second value to obtain a target sample image.

[0139] In the embodiment of the present application, after executing step 800 to obtain the second value of a preset channel (eg, Y channel) of any sample pixel in the sample image, step 810 is executed to replace the channel value of the sample pixel in the sample image with the second value.

[0140] After replacing the channel value of each sample pixel included in the sample image, a target sample image corresponding to the aforementioned sample image is obtained.

[0141] Step 6103: Input the target sample image into the segmentation model to obtain a predicted segmentation map corresponding to the sample image.

[0142] In an embodiment of the present application, when executing step 6103, the target sample image is input into the feature extraction subnetwork of the segmentation model, the feature extraction subnetwork extracts image features from the target sample image, and outputs the extracted image features through sigmoid to obtain a predicted segmentation map of the sample image.

[0143] In this way, by executing steps 6101 to 6103, a predicted segmentation map of each frame of the sample image in the sample video stream can be obtained.

[0144] Step 6104: Based on the predicted segmentation map and the true segmentation map corresponding to each sample image, the total loss value of the sample video stream is obtained.

[0145] In an embodiment of the present application, when executing step 6104, the predicted segmentation map corresponding to each sample image is compared with the corresponding real segmentation map, and the loss value corresponding to each sample image is calculated. Then, the loss value of each sample image is added to obtain the total loss of the sample video stream.

[0146] In the embodiment of the present application, when calculating the loss value of any sample image in each sample image, a loss function may be used, and the loss function may be expressed by the following formula:

[0147]

[0148] Among them, y i is the true segmentation map of the input image of the i-th frame, x i is the predicted segmentation map of the model for the i-th frame input image, and the above loss is the BCE loss, which is the commonly used binary segmentation loss. In this way, in order to minimize the optimization function, the segmentation model will approach the direction of minimizing the segmentation error to obtain the target segmentation model.

[0149] Step 6105: Adjust the model parameters of the segmentation model based on the total loss value.

[0150] In the embodiment of the present application, after executing step 6014 to obtain the total loss value of the sample video stream, before executing step 6105, it is determined that the total loss value is greater than or equal to the loss threshold.

[0151] Step 620: When the total loss value of the input sample video stream is less than the loss threshold, the segmentation model is output to obtain a target segmentation model.

[0152] In a specific implementation, when it is determined that the total loss value of the input sample video stream is less than the loss threshold, it means that the current segmentation model meets the convergence condition, so step 620 is executed to output the model parameters of the segmentation model at this time to obtain the target segmentation model.

[0153] The following uses specific examples to further explain the model training phase and the specific implementation phase in detail.

[0154] Example 1: Model training phase.

[0155] In the present application, see Fig. 9As shown in the figure, the specific training process of the model training phase is as follows:

[0156] Step 900: construct a segmentation model and initialize the segmentation model.

[0157] In the embodiment of the present application, the segmentation model is a hierarchical multi-label classification neural network, and the network architecture of the segmentation model is still as follows Figure 4 shown.

[0158] Step 910: Perform multiple rounds of iterative training on the segmentation model based on multiple sample video streams. When the total loss value of the input sample video stream is less than the loss threshold, output the segmentation model to obtain a target segmentation model. In one round of iterative training, the following steps are performed for any one of the multiple sample video streams:

[0159] Step 9101: directly input the first frame sample image in the sample video stream into the segmentation model, perform image feature extraction on the first frame sample image based on the segmentation model, and obtain a predicted segmentation map.

[0160] Step 9102: randomly select a target image determination rule from a plurality of preset target image determination rules, and obtain a target sample image based on the randomly selected target image determination rule and the second frame sample image in the sample video stream.

[0161] In an embodiment of the present application, when executing step 9102, a target image determination rule is randomly selected from a plurality of preset target image determination rules, and when the target sample image is obtained based on the randomly selected target image determination rule and the second frame sample image in the sample video stream, the specific content related to the aforementioned model training is specifically adopted and will not be repeated here.

[0162] Step 9103: Input the target sample image into the segmentation model, extract image features of the target sample image based on the segmentation model, obtain a predicted segmentation map of the second frame sample image, and so on, until a predicted segmentation map of each frame sample image in the sample video stream is obtained.

[0163] In an embodiment of the present application, after obtaining the predicted segmentation map of the second frame sample image, a method similar to that of obtaining the predicted segmentation map of the second frame sample image can be used to obtain the predicted segmentation maps of the third frame, the fourth frame, and the last frame sample image in the sample video stream.

[0164] Step 9104: Based on the predicted segmentation map and the true segmentation map, the loss value of each frame sample image is obtained, and the sum of the loss values ​​of each frame sample image is used as the total loss value of the sample video stream.

[0165] In the embodiment of the present application, after obtaining the predicted segmentation map (i.e., predicted segmentation binary map) of all frames, the aforementioned loss function can be used to calculate the loss value of each frame of the sample image, which will not be described in detail here. It should be noted that other loss functions can also be used to calculate the above loss value, and this application does not make specific limitations.

[0166] Step 9105: Determine whether the total loss value is less than the loss threshold, if not, execute step 9106; if yes, execute step 920.

[0167] Step 9106: Repeat steps 9101 to 9105.

[0168] Step 920: Save the model parameters of the segmentation model.

[0169] In the embodiment of the present application, after executing step 920, the training is terminated and the target segmentation model is obtained.

[0170] Example 2: Specific implementation phase.

[0171] In the present application, see Fig.10 The specific process of the specific implementation stage is as follows:

[0172] Step 1000: Build a target segmentation network and load the model parameters saved after training to obtain a target segmentation model.

[0173] Step 1010: directly input the first frame image in the video stream into the target segmentation model, perform image feature extraction on the first frame image based on the target segmentation model, and obtain a target segmentation map of the first frame image.

[0174] In the embodiment of the present application, the target segmentation map is a target segmentation prediction binary map, and after executing step 1010 to obtain the target segmentation prediction binary map of the first frame image, it is retained for use in subsequent fusion operations on subsequent frame images.

[0175] Step 1020: Fuse the second frame image in the video stream with the target segmentation map of the first frame image to obtain a fused image, and input the fused image into the target segmentation model, perform image feature extraction on the fused image based on the target segmentation model to obtain the target segmentation map of the second frame image.

[0176] In the embodiment of the present application, when the second frame image is fused with the target segmentation map of the first frame image in step 1020, the fusion is performed in the above-mentioned manner, which will not be repeated here. After step 1020 is executed, the target segmentation map of the second frame image is also retained for use in the fusion operation of subsequent frame images.

[0177] Step 1030: Fuse the third frame image in the video stream with the target segmentation map of the second frame image, and input the fused image into the target segmentation model, perform image feature extraction on the fused image based on the target segmentation model to obtain the target segmentation map of the third frame image, and so on, until the target segmentation map of each frame image of the video stream is obtained.

[0178] In an embodiment of the present application, after obtaining the target segmentation map of the third frame image, the target segmentation maps of the fourth frame, the fifth frame, and until the last frame image in the video stream can be obtained in a manner similar to that used to obtain the target segmentation map of the third frame image.

[0179] In some feasible embodiments, after executing step 1030, before displaying the above-mentioned video stream, the specific pixel position of the target object can be known according to the pixel value of the target segmentation map, and the image quality processing algorithm can be called in a targeted manner to perform image quality processing on any frame of the image.

[0180] Based on the same inventive concept, refer to Fig.11 As shown, in an embodiment of the present application, a device for target segmentation of a video stream is provided, comprising:

[0181] A fusion module 1110 is used to fuse a target frame image with a target segmentation map of a frame image before the target frame image to obtain a fused image, wherein the target frame image is a non-first frame image in a video stream to be displayed, and the target segmentation map of the first frame image in the video stream is obtained by inputting the first frame image into a target segmentation model;

[0182] A segmentation module 1120, configured to input the fused image into the target segmentation model to obtain a target segmentation map of the target frame image;

[0183] In which, the target segmentation model is obtained by training the segmentation model to be trained based on multiple sample video streams. During the training process, the target image input into the segmentation model is a sample image in any one of the multiple sample video streams, or an image after the sample image and the sample segmentation map are fused. The sample segmentation map is a randomly generated noise segmentation map or a predicted segmentation map of a sample image in the previous frame of the sample image.

[0184] In a possible implementation, the fusion module 1110 is specifically configured to:

[0185] The following operations are performed on each pixel in the target frame image respectively:

[0186] For any pixel among the pixels, taking the pixel value of any pixel in the target segmentation image of the previous frame image as a gain coefficient, and performing a multiplication operation with the channel value of any pixel in the target frame image to obtain a first value;

[0187] The channel value of any pixel is replaced by the first value to obtain the fused image.

[0188] In a possible implementation, after obtaining the target segmentation map of any frame of the video stream, the segmentation module 1120 is further used to:

[0189] Based on the target segmentation map of the arbitrary frame image, each pixel corresponding to the target object in the target segmentation map is enhanced, and the arbitrary frame image after the enhancement process is displayed.

[0190] In a possible implementation, the segmentation module 1120 is used to train the target segmentation model in the following manner:

[0191] The segmentation model is trained for multiple rounds of iterations based on the multiple sample video streams. When the total loss value of the input sample video streams is less than the loss threshold, the segmentation model is output to obtain the target segmentation model, wherein in one round of iterative training:

[0192] The following operations are performed for any one of the multiple sample video streams:

[0193] Inputting the first frame sample image in any one of the sample video streams into the segmentation module to obtain a predicted segmentation map of the first frame sample image;

[0194] For any frame of sample image except the first frame of sample image in any one of the sample video streams, based on a target image determination rule randomly selected from a plurality of preset target image determination rules and the any frame of sample image, a target sample image is determined, the target sample image is input into the segmentation model, and a predicted segmentation map corresponding to the any frame of sample image is obtained;

[0195] Based on the predicted allocation map and the true segmentation map corresponding to each sample image, obtaining a total loss value of any sample video stream;

[0196] Model parameters of the segmentation model are adjusted based on the total loss value.

[0197] In a possible implementation, the segmentation module 1120 is specifically configured to:

[0198] If the randomly selected target image determination rule is the first rule, taking the arbitrary frame of sample image as the target sample image;

[0199] If the randomly selected target image determination rule is the second rule, the predicted segmentation map of the previous frame sample image of the arbitrary frame sample image is fused with the arbitrary frame sample image to obtain the target sample image;

[0200] If the randomly selected target image determination rule is the third rule, a noise segmentation map is randomly generated, and the noise segmentation map is fused with the arbitrary frame sample image to obtain the target sample image.

[0201] In a possible implementation, the target sample segmentation map is a predicted segmentation map of a previous frame of sample image of any frame of sample image or the noise segmentation map, and the segmentation module 920 is used to:

[0202] The following operations are performed for each sample pixel in any one frame of sample image:

[0203] For any sample pixel among the sample pixels, taking the pixel value of any sample pixel in the target sample segmentation image as a gain coefficient, and performing a multiplication operation with the channel value of any sample pixel in the any frame sample image to obtain a second value;

[0204] The channel value of any sample pixel is replaced by the second value to obtain the target sample image.

[0205] Based on the same inventive concept, see Fig.12 As shown, an electronic device is provided in an embodiment of the present application, including:

[0206] Memory 121, used to store computer programs or instructions;

[0207] The processor 122 is used to execute the computer program or instruction in the memory so that any one of the methods in the above embodiments is executed.

[0208] The processor 122 may include one or more central processing units (CPUs) or digital processing units, etc., for executing computer programs or instructions in the memory 121, so that any one of the methods in the above embodiments is executed;

[0209] It should be noted that the specific connection medium between the memory 121 and the processor 122 is not limited in the embodiment of the present application. Fig.12In the embodiment, the memory 121 and the processor 122 are connected via a bus 123, and the connection between other components is only for illustrative purposes and is not intended to be limiting. The bus 123 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.12 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0210] Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium, and when the instructions in the storage medium are executed by a processor, the processor is enabled to execute any one of the methods in the above-mentioned embodiments. Since the principle of solving the problem by the above-mentioned computer-readable storage medium is similar to a method for target segmentation of a video stream, the implementation of the above-mentioned computer-readable storage medium can refer to the implementation of the method, and the repeated parts will not be repeated.

[0211] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0212] The present application is described with reference to the flowchart and / or block diagram of the method, device (system), and computer program product according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the process and / or box in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the functions specified in one process or multiple processes in the flowchart and / or one box or multiple boxes in the block diagram.

[0213] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0214] These computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0215] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for object segmentation of a video stream, characterized in that: include: Fusing a target frame image with a target segmentation map of a frame image before the target frame image to obtain a fused image, wherein the target frame image is a non-first frame image in a video stream to be displayed, and the target segmentation map of the first frame image in the video stream is obtained by inputting the first frame image into a target segmentation model; Inputting the fused image into the target segmentation model to obtain a target segmentation map of the target frame image; In which, the target segmentation model is obtained by training the segmentation model to be trained based on multiple sample video streams. During the training process, the target image input into the segmentation model is a sample image in any one of the multiple sample video streams, or an image after the sample image and the sample segmentation map are fused. The sample segmentation map is a randomly generated noise segmentation map or a predicted segmentation map of a sample image in the previous frame of the sample image.

2. The method according to claim 1, characterized in that The step of fusing the target frame image with a target segmentation map of a frame image before the target frame image to obtain a fused image comprises: The following operations are performed on each pixel in the target frame image respectively: For any pixel among the pixels, taking the pixel value of any pixel in the target segmentation image of the previous frame image as a gain coefficient, and performing a multiplication operation with the channel value of any pixel in the target frame image to obtain a first value; The channel value of any pixel is replaced by the first value to obtain the fused image.

3. The method according to claim 1, characterized in that After obtaining the target segmentation map of any frame image in the video stream, the method further includes: Based on the target segmentation map of the arbitrary frame image, each pixel corresponding to the target object in the target segmentation map is enhanced, and the arbitrary frame image after the enhancement process is displayed.

4. The method according to any one of claims 1 to 3, characterized in that: The target segmentation model is obtained by training in the following way: The segmentation model is trained for multiple rounds of iterations based on the multiple sample video streams. When the total loss value of the input sample video streams is less than the loss threshold, the segmentation model is output to obtain the target segmentation model, wherein in one round of iterative training: The following operations are performed for any one of the multiple sample video streams: Inputting the first frame sample image in any one of the sample video streams into the segmentation module to obtain a predicted segmentation map of the first frame sample image; For any frame of sample image except the first frame of sample image in any one of the sample video streams, based on a target image determination rule randomly selected from a plurality of preset target image determination rules and the any frame of sample image, a target sample image is determined, the target sample image is input into the segmentation model, and a predicted segmentation map corresponding to the any frame of sample image is obtained; Based on the predicted allocation map and the true segmentation map corresponding to each sample image, obtaining a total loss value of any sample video stream; Model parameters of the segmentation model are adjusted based on the total loss value.

5. The method according to claim 4, characterized in that Determining a target sample image based on the target image determination rule randomly selected from a plurality of preset target image determination rules and the arbitrary frame sample image includes: If the randomly selected target image determination rule is the first rule, taking the arbitrary frame of sample image as the target sample image; If the randomly selected target image determination rule is the second rule, the predicted segmentation map of the previous frame sample image of the arbitrary frame sample image is fused with the arbitrary frame sample image to obtain the target sample image; If the randomly selected target image determination rule is the third rule, a noise segmentation map is randomly generated, and the noise segmentation map is fused with the arbitrary frame sample image to obtain the target sample image.

6. The method according to claim 5, characterized in that The target sample segmentation map is a predicted segmentation map of a sample image of a previous frame of the arbitrary sample image or the noise segmentation map, and the target sample segmentation map is fused with the arbitrary sample image to obtain the target sample image, including: The following operations are performed for each sample pixel in any one frame of sample image: For any sample pixel among the sample pixels, taking the pixel value of any sample pixel in the target sample segmentation image as a gain coefficient, and performing a multiplication operation with the channel value of any sample pixel in the any frame sample image to obtain a second value; The channel value of any sample pixel is replaced by the second value to obtain the target sample image.

7. A device for segmenting a video stream, characterized in that: include: A fusion module is used to fuse a target frame image with a target segmentation map of a frame image before the target frame image to obtain a fused image, wherein the target frame image is a non-first frame image in the video stream to be displayed, and the target segmentation map of the first frame image in the video stream is obtained by inputting the first frame image into a target segmentation model; A segmentation module, used for inputting the fused image into the target segmentation model to obtain a target segmentation map of the target frame image; In which, the target segmentation model is obtained by training the segmentation model to be trained based on multiple sample video streams. During the training process, the target image input into the segmentation model is a sample image in any one of the multiple sample video streams, or an image after the sample image and the sample segmentation map are fused. The sample segmentation map is a randomly generated noise segmentation map or a predicted segmentation map of a sample image in the previous frame of the sample image.

8. The device according to claim 7, characterized in that The fusion module is specifically used for: The following operations are performed on each pixel in the target frame image respectively: For any pixel among the pixels, taking the pixel value of any pixel in the target segmentation image of the previous frame image as a gain coefficient, and performing a multiplication operation with the channel value of any pixel in the target frame image to obtain a first value; The channel value of any pixel is replaced by the first value to obtain the fused image.

9. An electronic device, characterized in that: include: Memory, used to store computer programs or instructions; A processor is used to execute the computer program or instructions in the memory so that the method according to any one of claims 1 to 6 is executed.

10. A computer-readable storage medium, characterized in that: When the instructions in the storage medium are executed by a processor, the processor is enabled to execute the method according to any one of claims 1 to 6.

Images