Image processing method, device, electronic device and readable storage medium

Abstract

The embodiments of the present disclosure disclose an image processing method, device, electronic device and readable storage medium. The image processing method includes: obtaining a mask corresponding to an image frame in a video stream; generating an optical flow map matrix corresponding to the image frame according to the image frame; and generating a target mask for segmenting a target object from the video stream according to the mask and the optical flow map matrix.

Description

Technical Field

[0001] The present disclosure relates to the technical field of image processing, and in particular to an image processing method, device, electronic device and readable storage medium. Background Art

[0002] Video segmentation technology is used to segment target objects in video images and is widely used in video conferencing systems, live broadcasts, and film and television production.

[0003] The current video segmentation methods usually use a single video image to train the convolutional neural network model. When the trained model is used for reasoning, due to the lack of inter-frame information, the generated video segmentation result graph is often discontinuous in time sequence, and the visual jitter is obvious, especially at the edge of the target object. When the background is replaced, it particularly affects the viewing experience. If a video sequence is used for model training, since a large number of video sequences need to be annotated, the training data redundancy is high, which is time-consuming and costly, and the results generated by the trained model have a serious tailing effect, especially when the object moves violently. In addition, the anti-shake performance of the trained model is often not good enough, and there is still jitter in subjective vision. If a single video image is used for model training, and then the inter-frame edge jitter is eliminated through optical flow-assisted post-processing, this method generally requires a high-precision optical flow model. Since the high-precision optical flow model generates optical flow at a slow speed, it is difficult to meet the timeliness requirements of real-time applications. Summary of the invention

[0004] In order to solve the problems in the related art, the embodiments of the present disclosure provide an image processing method, an apparatus, an electronic device, and a readable storage medium.

[0005] In a first aspect, an embodiment of the present disclosure provides an image processing method, including:

[0006] Obtaining a mask corresponding to an image frame in a video stream;

[0007] According to the image frame, generating an optical flow map matrix corresponding to the image frame;

[0008] A target mask for segmenting a target object from the video stream is generated according to the mask and the optical flow map matrix.

[0009] In combination with the first aspect, in a first implementation method of the first aspect of the present disclosure, obtaining a mask corresponding to an image frame in a video stream includes: obtaining a first mask corresponding to a first frame image in the video stream and a second mask corresponding to a second frame image after the first frame image, the first mask being used to segment a target object from the first frame image, and the second mask being used to segment a target object from the second frame image.

[0010] In combination with the first implementation manner of the first aspect, in a second implementation manner of the first aspect of the present disclosure, acquiring a first mask corresponding to a first frame image in a video stream and a second mask corresponding to a second frame image after the first frame image includes:

[0011] When the first frame image is the first frame image in the video stream, the first mask is obtained by inputting the first frame image into a trained mask generation model; when the first frame image is not the first frame image in the video stream, a target mask of the first frame image obtained according to the first frame image and a frame image before the first frame image is used as the first mask;

[0012] The second mask is obtained by inputting the second frame image into a trained mask generation model.

[0013] In combination with the first aspect, in a third implementation method of the first aspect of the present disclosure, generating an optical flow map matrix corresponding to the image frame based on the image frame includes: generating an optical flow map matrix based on the first frame image and the second frame image, each element in the optical flow map matrix is ​​an optical flow vector of a corresponding unit display block in the first frame image, and the unit display block includes one or more pixels.

[0014] In combination with the first aspect, in a fourth implementation method of the first aspect of the present disclosure, generating a target mask for segmenting the target object from the video stream based on the mask and the optical flow map matrix includes: generating a target mask corresponding to the second frame image based on the first mask, the second mask and the optical flow map matrix, and the target mask is used to segment the target object from the second frame image.

[0015] In combination with the fourth implementation manner of the first aspect, in a fifth implementation manner of the first aspect of the present disclosure, generating a target mask corresponding to the second frame image according to the first mask, the second mask, and the optical flow map matrix includes:

[0016] generating a third mask according to the first mask and the optical flow map matrix;

[0017] generating a fourth mask according to the third mask, the second mask, and the optical flow map;

[0018] The target mask is generated according to the fourth mask.

[0019] In combination with the fifth implementation manner of the first aspect, in a sixth implementation manner of the first aspect of the present disclosure, generating a third mask according to the first mask and the optical flow map includes:

[0020] The first mask is pixel-remapped based on the optical flow map matrix to generate the third mask.

[0021] In combination with the fifth implementation manner of the first aspect, in a seventh implementation manner of the first aspect of the present disclosure, generating a target mask corresponding to the second frame image according to the first mask, the second mask, and the optical flow map matrix further includes:

[0022] An optical flow matrix is ​​obtained based on the optical flow map matrix, and each element of the optical flow matrix is ​​the modulus of the corresponding element in the optical flow map.

[0023] In combination with the seventh implementation manner of the first aspect, in an eighth implementation manner of the first aspect of the present disclosure, generating a fourth mask according to the third mask, the second mask, and the optical flow map includes:

[0024] Obtaining a first product obtained by performing a dot multiplication of the third mask and an inverse result of the optical flow matrix, wherein the inverse result of the optical flow matrix is ​​obtained by subtracting the optical flow matrix from a unit matrix;

[0025] Obtaining a second product obtained by performing a dot product of the second mask and the optical flow matrix;

[0026] The fourth mask is generated according to the sum of the first product and the second product.

[0027] In combination with the fifth implementation manner of the first aspect, in a ninth implementation manner of the first aspect of the present disclosure, generating the target mask according to the fourth mask includes:

[0028] The fourth mask is corrected by using the second mask and the target object edge detection result of the second frame image, and the corrected fourth mask is used as the target mask.

[0029] In combination with the ninth implementation manner of the first aspect, in a tenth implementation manner of the first aspect of the present disclosure, the step of correcting the fourth mask by using the second mask and the target object edge detection result of the second frame image includes:

[0030] Obtaining a third product obtained by performing a dot multiplication of the fourth mask and an inverse result of a target object edge detection result of the second frame image, wherein the inverse result of the target object edge detection result is obtained by subtracting the inverse result of the target object edge detection result from the unit matrix;

[0031] Obtaining a fourth product obtained by performing a dot product of the second mask and the target object edge detection result;

[0032] The target mask is generated according to the sum of the third product and the fourth product.

[0033] In combination with the tenth implementation manner of the first aspect, in an eleventh implementation manner of the first aspect of the present disclosure, generating the target mask according to the sum of the third product and the fourth product includes:

[0034] The value of each unit display block in the sum of the third product and the fourth product is compared with a corresponding preset threshold to obtain the value of the corresponding unit display block in the target mask.

[0035] In a second aspect, an image processing method is provided in an embodiment of the present disclosure, including:

[0036] receiving a request for generating a target mask for segmenting a target object from a video stream, the request including video stream identification information and target object identification information;

[0037] According to the video stream identification information, the video stream is acquired, and according to the target object identification information, a corresponding mask generation model is acquired;

[0038] Acquire a mask corresponding to an image frame in the video stream through the mask generation model;

[0039] According to the image frame, generating an optical flow map matrix corresponding to the image frame;

[0040] According to the mask and the optical flow map matrix, a target mask for segmenting the target object from the video stream is generated. In a third aspect, an image processing apparatus is provided in an embodiment of the present disclosure, comprising:

[0041] Obtaining a mask corresponding to an image frame in a video stream;

[0042] According to the image frame, generating an optical flow map matrix corresponding to the image frame;

[0043] A target mask for segmenting a target object from the video stream is generated according to the mask and the optical flow map matrix.

[0044] In combination with the third aspect, in a first implementation method of the third aspect of the present disclosure, obtaining a mask corresponding to an image frame in a video stream includes: obtaining a first mask corresponding to a first frame image in the video stream and a second mask corresponding to a second frame image after the first frame image, the first mask being used to segment a target object from the first frame image, and the second mask being used to segment a target object from the second frame image.

[0045] In combination with the first implementation manner of the third aspect, in a second implementation manner of the third aspect of the present disclosure, acquiring a first mask corresponding to a first frame image in a video stream and a second mask corresponding to a second frame image after the first frame image includes:

[0046] When the first frame image is the first frame image in the video stream, the first mask is obtained by inputting the first frame image into a trained mask generation model; when the first frame image is not the first frame image in the video stream, a target mask of the first frame image obtained according to the first frame image and a frame image before the first frame image is used as the first mask;

[0047] The second mask is obtained by inputting the second frame image into a trained mask generation model.

[0048] In combination with the third aspect, in a third implementation method of the third aspect of the present disclosure, generating an optical flow map matrix corresponding to the image frame based on the image frame includes: generating an optical flow map matrix based on the first frame image and the second frame image, each element in the optical flow map matrix is ​​an optical flow vector of a corresponding unit display block in the first frame image, and the unit display block includes one or more pixels.

[0049] In combination with the third aspect, in a fourth implementation method of the third aspect of the present disclosure, generating a target mask for segmenting the target object from the video stream based on the mask and the optical flow map matrix includes: generating a target mask corresponding to the second frame image based on the first mask, the second mask and the optical flow map matrix, and the target mask is used to segment the target object from the second frame image.

[0050] In combination with the fourth implementation manner of the third aspect, in a fifth implementation manner of the third aspect of the present disclosure, generating a target mask corresponding to the second frame image according to the first mask, the second mask, and the optical flow map matrix includes:

[0051] generating a third mask according to the first mask and the optical flow map matrix;

[0052] generating a fourth mask according to the third mask, the second mask, and the optical flow map;

[0053] The target mask is generated according to the fourth mask.

[0054] In combination with the fifth implementation manner of the third aspect, in a sixth implementation manner of the third aspect of the present disclosure, generating a third mask according to the first mask and the optical flow map includes:

[0055] The first mask is pixel-remapped based on the optical flow map matrix to generate the third mask.

[0056] In combination with the fifth implementation manner of the third aspect, in a seventh implementation manner of the third aspect of the present disclosure, generating a target mask corresponding to the second frame image according to the first mask, the second mask, and the optical flow map matrix further includes:

[0057] An optical flow matrix is ​​obtained based on the optical flow map matrix, and each element of the optical flow matrix is ​​the modulus of the corresponding element in the optical flow map.

[0058] In combination with the seventh implementation manner of the third aspect, in an eighth implementation manner of the third aspect of the present disclosure, generating a fourth mask according to the third mask, the second mask, and the optical flow map includes:

[0059] Obtaining a first product obtained by performing a dot multiplication of the third mask and an inverse result of the optical flow matrix, wherein the inverse result of the optical flow matrix is ​​obtained by subtracting the optical flow matrix from a unit matrix;

[0060] Obtaining a second product obtained by performing a dot product of the second mask and the optical flow matrix;

[0061] The fourth mask is generated according to the sum of the first product and the second product.

[0062] In combination with the fifth implementation manner of the third aspect, in a ninth implementation manner of the third aspect of the present disclosure, generating the target mask according to the fourth mask includes:

[0063] The fourth mask is corrected by using the second mask and the target object edge detection result of the second frame image, and the corrected fourth mask is used as the target mask.

[0064] In combination with the ninth implementation manner of the third aspect, in a tenth implementation manner of the third aspect of the present disclosure, the step of correcting the fourth mask by using the second mask and the target object edge detection result of the second frame image includes:

[0065] Obtaining a third product obtained by performing a dot multiplication of the fourth mask and an inverse result of a target object edge detection result of the second frame image, wherein the inverse result of the target object edge detection result is obtained by subtracting the inverse result of the target object edge detection result from the unit matrix;

[0066] Obtaining a fourth product obtained by performing a dot product of the second mask and the target object edge detection result;

[0067] The target mask is generated according to the sum of the third product and the fourth product.

[0068] In combination with the tenth implementation manner of the third aspect, in an eleventh implementation manner of the third aspect of the present disclosure, generating the target mask according to the sum of the third product and the fourth product includes:

[0069] The value of each unit display block in the sum of the third product and the fourth product is compared with a corresponding preset threshold to obtain the value of the corresponding unit display block in the target mask.

[0070] In a fourth aspect, an embodiment of the present disclosure provides an image processing device, including:

[0071] A receiving module is configured to receive a request for generating a target mask for segmenting a target object from a video stream, wherein the request includes video stream identification information and target object identification information;

[0072] A second acquisition module is configured to acquire the video stream according to the video stream identification information, and acquire a corresponding mask generation model according to the target object identification information;

[0073] A third acquisition module is configured to acquire a mask corresponding to the image frame in the video stream through the mask generation model;

[0074] A first generating module is configured to generate an optical flow map matrix corresponding to the image frame according to the image frame;

[0075] The second generating module is configured to generate a target mask for segmenting a target object from the video stream according to the mask and the optical flow map matrix.

[0076] In a fifth aspect, an embodiment of the present disclosure provides an electronic device, comprising a memory and a processor, wherein the memory is used to store one or more computer instructions, and wherein the one or more computer instructions are executed by the processor to implement a method as described in any one of the first to second aspects.

[0077] In a sixth aspect, a computer-readable storage medium is provided in an embodiment of the present disclosure, on which computer instructions are stored. When the computer instructions are executed by a processor, the methods described in the first aspect to the second aspect are implemented.

[0078] According to the technical solution provided by the embodiment of the present disclosure, an image processing method includes: obtaining a mask corresponding to an image frame in a video stream; generating an optical flow map matrix corresponding to the image frame according to the image frame; generating a target mask for segmenting a target object from the video stream according to the mask and the optical flow map matrix. According to the embodiment of the present disclosure, the target mask is generated according to the mask and the optical flow map matrix, and there is no need to use a video sequence for model training, nor a high-precision optical flow model, which reduces costs and improves timeliness. At the same time, by using the optical flow map in the process of generating the mask, the inter-frame information is introduced, which solves the problem of video mask edge jitter, and the edge of the target image video segmented according to the mask is stable, and the subjective visual experience is good.

[0079] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Other features, objectives and advantages of the present disclosure will become more apparent through the following detailed description of non-limiting embodiments in conjunction with the accompanying drawings. In the accompanying drawings:

[0081] Figure 1A A flowchart of an image processing method according to an embodiment of the present disclosure is shown.

[0082] Figure 1B A flowchart of an image processing method according to an embodiment of the present disclosure is shown.

[0083] Figure 2 The process of the image processing method according to the embodiment of the present disclosure is shown.

[0084] Figure 3A A structural block diagram of an image processing device according to an embodiment of the present disclosure is shown.

[0085] Figure 3B A structural block diagram of an image processing device according to an embodiment of the present disclosure is shown.

[0086] Figure 4 A structural block diagram of an electronic device according to an embodiment of the present disclosure is shown.

[0087] Figure 5 A schematic diagram showing the structure of a computer system suitable for implementing the method according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0088] Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily implement them. In addition, for the sake of clarity, parts not related to the description of the exemplary embodiments are omitted in the accompanying drawings.

[0089] In the present disclosure, it should be understood that terms such as "include" or "have" are intended to indicate the presence of features, numbers, steps, behaviors, components, parts, or a combination thereof disclosed in the present specification, and are not intended to exclude the possibility that one or more other features, numbers, steps, behaviors, components, parts, or a combination thereof exist or are added.

[0090] It should also be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other. The present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0091] In the present disclosure, the acquisition of user information or user data is an operation authorized and confirmed by the user, or actively selected by the user.

[0092] As mentioned above, the current video segmentation methods usually use a single video image to train the convolutional neural network model. When the trained model is used for reasoning, due to the lack of inter-frame information, the generated video segmentation result map is often discontinuous in time sequence, and the visual jitter is obvious, especially at the edge of the target object. When the background is replaced, it particularly affects the perception. If a video sequence is used for model training, since a large number of video sequences need to be annotated, the training data redundancy is high, which is time-consuming and costly, and the results generated by the trained model have a serious tailing effect, especially when the object moves violently. The tailing effect is more obvious. In addition, the anti-shake performance of the trained model is often not good enough, and there is still jitter in subjective vision. If a single video image is used for model training, and then the inter-frame edge jitter is eliminated through optical flow-assisted post-processing, this method generally requires a high-precision optical flow model. Since the high-precision optical flow model generates optical flow at a slow speed, it is difficult to meet the timeliness requirements of real-time applications.

[0093] The present disclosure provides an image processing method, including:

[0094] Obtaining a mask corresponding to an image frame in a video stream;

[0095] According to the image frame, generating an optical flow map matrix corresponding to the image frame;

[0096] A target mask for segmenting a target object from the video stream is generated according to the mask and the optical flow map matrix.

[0097] According to the embodiments of the present disclosure, a target mask is generated based on a mask and an optical flow map matrix, and there is no need to use a video sequence for model training, nor is there a need for a high-precision optical flow model, which reduces costs and improves timeliness. At the same time, by using an optical flow map in the process of generating a mask, inter-frame information is introduced, which solves the problem of jittering of the edge of the video mask. The edge of the target image video segmented by the mask is stable, and the subjective visual experience is good.

[0098] Figure 1A A flowchart of an image processing method according to an embodiment of the present disclosure is shown.

[0099] like Figure 1A As shown, the image processing method includes the following steps S101-S103:

[0100] In step S101, a mask corresponding to an image frame in a video stream is obtained;

[0101] In step S102, an optical flow map matrix corresponding to the image frame is generated according to the image frame;

[0102] In step S103, a target mask for segmenting a target object from the video stream is generated according to the mask and the optical flow map matrix.

[0103] Figure 1B A flowchart of an image processing method according to an embodiment of the present disclosure is shown.

[0104] According to an embodiment of the present disclosure, Figure 1B The illustrated method may be performed at a server.

[0105] like Figure 1B As shown, the image processing method is in reference Figure 1A In addition to steps S102 and S103 described above, the following steps S104-S106 are also included:

[0106] In step S104, a request for generating a target mask for segmenting a target object from a video stream is received, wherein the request includes video stream identification information and target object identification information;

[0107] In step S105, the video stream is acquired according to the video stream identification information, and a corresponding mask generation model is acquired according to the target object identification information;

[0108] In step S106, a mask corresponding to the image frame in the video stream is obtained through the mask generation model;

[0109] In step S102, an optical flow map matrix corresponding to the image frame is generated according to the image frame;

[0110] In step S103, a target mask for segmenting a target object from the video stream is generated according to the mask and the optical flow map matrix.

[0111] According to the embodiments of the present disclosure, a target mask is generated based on a mask and an optical flow map matrix, and there is no need to use a video sequence for model training, nor is there a need for a high-precision optical flow model, which reduces costs and improves timeliness. At the same time, by using an optical flow map in the process of generating a mask, inter-frame information is introduced, which solves the problem of jittering of the edge of the video mask. The edge of the target image video segmented by the mask is stable, and the subjective visual experience is good.

[0112] According to an embodiment of the present disclosure, the receiving of a request for generating a target mask for segmenting a target object from a video stream may be, for example, the server receiving a request for generating a target mask for segmenting a target object from a video stream from a client.

[0113] According to an embodiment of the present disclosure, the video stream is obtained according to the video stream identification information, for example, the video stream may be obtained from an internal or external storage device according to the video stream identification information, and the corresponding mask generation model is obtained according to the target object identification information, for example, a mask generation model pre-trained for segmenting the target object is selected according to the target object identification information, such as a mask generation model obtained by training a convolutional neural network with a large number of pictures without time-series relationship.

[0114] According to an embodiment of the present disclosure, the method of obtaining a mask corresponding to an image frame in a video stream includes: obtaining a first mask corresponding to a first frame image in the video stream and a second mask corresponding to a second frame image after the first frame image, wherein the first mask is used to segment a target object from the first frame image, and the second mask is used to segment a target object from the second frame image.

[0115] According to an embodiment of the present disclosure, acquiring a first mask corresponding to a first frame image in a video stream and a second mask corresponding to a second frame image after the first frame image includes:

[0116] When the first frame image is the first frame image in the video stream, the first mask is obtained by inputting the first frame image into a trained mask generation model; when the first frame image is not the first frame image in the video stream, a target mask of the first frame image obtained according to the first frame image and a frame image before the first frame image is used as the first mask;

[0117] The second mask is obtained by inputting the second frame image into a trained mask generation model.

[0118] According to an embodiment of the present disclosure, generating an optical flow map matrix corresponding to the image frame based on the image frame includes: generating an optical flow map matrix based on the first frame image and the second frame image, each element in the optical flow map matrix is ​​an optical flow vector of a corresponding unit display block in the first frame image, and the unit display block includes one or more pixels.

[0119] According to an embodiment of the present disclosure, generating a target mask for segmenting a target object from the video stream based on the mask and the optical flow map matrix includes: generating a target mask corresponding to the second frame image based on the first mask, the second mask and the optical flow map matrix, and the target mask is used to segment the target object from the second frame image.

[0120] According to an embodiment of the present disclosure, the mask is a binary or multi-valued image of the same size as the image, and the unit display blocks of the mask have a one-to-one correspondence with the unit display blocks of the image, and are used to segment the target object from the image. For example, in the mask, the value corresponding to the unit display block of the background image is 0, the value corresponding to the unit display block of the first target object is 1, and the value corresponding to the unit display block of the second target object is 2. In this way, the mask can be used to segment the target object from the video image. When the mask is operated with a matrix, the mask can be regarded as a matrix, and each element in the matrix is ​​the value of the corresponding unit display block in the mask.

[0121] According to an embodiment of the present disclosure, a neural network trained based on a single image can be used to generate a mask of an image. For example, a first frame image can be input into a neural network trained based on a single image, a first mask can be output from the neural network, and a second frame image can be input into the neural network, and a second mask can be output from the neural network.

[0122] According to an embodiment of the present disclosure, the optical flow map matrix is ​​a matrix obtained by the first frame image and the second frame image, and each element in the matrix is ​​an optical flow vector of a corresponding unit display block in the first frame image, which is used to represent the displacement of the corresponding unit display block in the first frame image from the position of the unit display block in the first frame image to the position of the unit display block in the second frame image. According to an embodiment of the present disclosure, the optical flow vector is a two-dimensional vector, the first component of which represents the displacement of the corresponding unit display block in the width direction of the image, and the second component represents the displacement of the corresponding unit display block in the height direction of the image. The displacement is represented by the number of display blocks. For example, a displacement of 0.7 in the width direction of the image represents a displacement of 0.7 unit display blocks in the width direction, and a displacement of 0.5 in the height direction of the image represents a displacement of 0.5 unit display blocks in the height direction. According to an embodiment of the present disclosure, the elements in the optical flow map matrix have a one-to-one correspondence with the unit display blocks in the first frame image. The optical flow map matrix can be generated, for example, using the optical flow generation method provided by tools such as OpenCV. OpenCV is a cross-platform computer vision and machine learning software library based on Berkeley Software Distribution (Berkeley Software Distribution), which can run on Linux, Windows, Android and Mac OS operating systems. It is lightweight and efficient, and provides interfaces for languages ​​such as Python, Ruby, and MATLAB, and implements many common algorithms in image processing and computer vision.

[0123] According to the embodiment of the present disclosure, a target mask corresponding to the second frame image is generated based on the first mask, the second mask and the optical flow map matrix, and there is no need to use a video sequence for model training, nor a high-precision optical flow model, which reduces costs and improves timeliness. At the same time, by using the optical flow map matrix in the process of generating the mask, inter-frame information is introduced, which solves the problem of jittering of the video mask edge. The edge of the target image video segmented according to the mask is stable, and the subjective visual experience is good.

[0124] Figure 2 The process of the image processing method according to the embodiment of the present disclosure is shown.

[0125] exist Figure 2 In the paper, Seg Net is a mask generation model obtained by training a convolutional neural network with a large number of pictures without time-sequential relationship. This model is directly applied to process video images, and the obtained mask has the required accuracy but the edge of the segmented target object is jittery.

[0126] like Figure 2As shown, the first frame image is input into Seg Net to obtain the first mask PM, and the second frame image is input into SegNet to obtain the second mask CM. The first mask PM is used to segment the target object from the first frame image, and the second mask CM is used to segment the target object from the second frame image.

[0127] Then, using, for example, the optical flow generation method provided by OpenCV, an optical flow map matrix O is generated from the first frame image and the second frame image, wherein the elements in the optical flow map matrix O represent the displacement of the corresponding unit display block in the first frame image from the position of the unit display block in the first frame image to the position of the unit display block in the second frame image.

[0128] The first mask PM is pixel-remapped based on the optical flow map matrix O to generate a third mask AM. Specifically, the formula for pixel remapping is:

[0129] AM(x,y)=PM(O x’ (x,y),O y’ (x,y)

[0130] Wherein, AM(x, y) represents the value of the unit display block at the xth row and the yth column in the third mask AM, x’ (x, y) represents the first component of the optical flow vector of the element in the xth row and yth column in the optical flow map matrix. y’ (x, y) represents the second component of the optical flow vector of the element in the xth row and yth column of the optical flow map matrix, PM(O x’ (x,y),O y’ (x, y)) represents the first mask PM in the O x’ (x,y) row O y’ The units in the (x,y) column show the values ​​of the block.

[0131] When O x’ (x,y) and O y’ When (x, y) is not an integer, the first mask PM can be used to x’ (x,y) row O y’ The unit display block of the (x, y) column is bilinearly interpolated with two adjacent unit display blocks in the width direction of the first mask (corresponding to the width direction of the image) and two adjacent unit display blocks in the height direction of the first mask (corresponding to the height direction of the image), and PM (O x’ (x,y),O y’ (x,y)). For example, PM(0,0), PM(0,1), PM(1,0), and PM(1,1) can be bilinearly interpolated to get PM(0.5,0.7).

[0132] Based on the optical flow matrix O, an optical flow matrix OF is obtained, and each element of the optical flow matrix OF is the modulus of the corresponding element in the optical flow matrix O. For example, assuming that the optical flow matrix O is an M*N matrix, in which the element O(x, y) is a two-dimensional vector (x', y'), then the optical flow matrix OF is also an M*N matrix, in which the element O corresponding to the element O(x, y) is m,n is (x' 2 +y' 2 ) 1 / 2 .

[0133] Then, a first product AM*(I-OF) is obtained by performing a dot multiplication of the third mask AM and the inverse result (I-OF) of the optical flow matrix OF, where the inverse result of the optical flow matrix OF is obtained by subtracting the optical flow matrix OF from the unit matrix I. A second product CM*OF is obtained by performing a dot multiplication of the second mask CM and the optical flow matrix OF. The first product AM*(I-OF) and the second product CM*OF are summed to obtain a fourth mask BM.

[0134] A target mask may be generated based on the fourth mask BM. Since the fourth mask BM is calculated by the first mask PM, the second mask CM and the optical flow matrix OF, each unit display block in the fourth mask BM may take a value other than the unit display block value of the first mask PM and the second mask CM. For example, for a video image with a target object, the unit display block of the fourth mask BM may have other values ​​besides 0 (corresponding to the background) and 1 (corresponding to the target object), such as 0.3, 0.5, 0.7, etc. At this point, the element value may be rounded to obtain a binary mask of 0 and 1 as the target mask.

[0135] According to the embodiments of the present disclosure, the jitter phenomenon of the edge of the target image obtained by segmentation using the fourth mask BM has been significantly reduced, but when the optical flow map matrix is ​​generated using a low-precision optical flow generation method, a large number of missed segmentations may be introduced. Considering that the overall segmentation accuracy of the mask of the current frame generated by the Seg Net model is high and missed segmentations rarely occur, the fourth mask BM can be combined with the second mask CM, and the stability of the fourth mask BM and the overall high-precision segmentation of the second mask CM can be combined to obtain a mask with stable edges and accurate overall segmentation.

[0136] Specifically, the fourth mask BM can be dot-multiplied with the inverse result of the edge matrix E to obtain a third product, wherein the edge matrix E is the edge detection result of the target object of the second frame image. The element value corresponding to the edge of the second frame image in the edge matrix E can be, for example, 1, and the other elements can be, for example, 0. According to an embodiment of the present disclosure, the edge matrix E can be obtained from the second mask CM. The inverse result of the edge matrix E is obtained by subtracting the edge matrix E from the unit matrix I. The second mask CM is dot-multiplied with the edge matrix E to obtain a fourth product. The third product and the fourth product are added and summed, and the value of each unit display block in the sum of the third product and the fourth product is compared with the corresponding preset threshold value to obtain the corresponding element in the target mask. Specifically, for a binary mask, each unit display block in the sum of the third product and the fourth product may have other values ​​besides 0 (corresponding to the background) and 1 (corresponding to the target object), such as 0.3, 0.5, 0.7..., and so on. At this time, the value of the unit display block may be rounded to obtain a binary mask of 0 and 1 as the target mask. For a mask of three or more values, the target mask may also be obtained by rounding each element.

[0137] Then, the current second frame image can be used as the first frame image, the target mask of the current second frame image can be used as the first mask, and the next frame image of the second frame image (for example, the third frame image) can be used as the second frame image. The above process is repeated to obtain the corresponding masks of each image frame in the video stream.

[0138] exist Figure 1A and 1B In the method shown, generating a target mask corresponding to the second frame image according to the first mask, the second mask and the optical flow map matrix includes:

[0139] generating a third mask according to the first mask and the optical flow map matrix;

[0140] generating a fourth mask according to the third mask, the second mask, and the optical flow map;

[0141] The target mask is generated according to the fourth mask.

[0142] According to an embodiment of the present disclosure, the first mask is, for example, Figure 2 The first mask PM shown, the second mask is for example Figure 2 For the second mask CM shown, the optical flow map matrix is, for example, Figure 2 The optical flow matrix O shown, the third mask is, for example, Figure 2 The third mask AM shown, the fourth mask is, for example, Figure 2The fourth mask BM shown. According to an embodiment of the present disclosure, the value of each unit display block of the fourth mask BM can be compared with the threshold value corresponding to the value interval in which the value of the unit display block is located, so as to obtain the value of the corresponding unit display block in the target mask. For example, assuming that the value of the unit display block in the fourth mask BM is Y, and the upper and lower limits of its value interval are two integers P and P+1, respectively, where P≤Y≤P+1, then Y can be compared with the threshold value P+0.5 corresponding to the value interval. If Y is less than the threshold value, the value of the unit display block corresponding to Y in the target mask is P, otherwise the value of the unit display block corresponding to Y in the target mask is P+1.

[0143] According to an embodiment of the present disclosure, generating a target mask corresponding to the second frame image according to the first mask, the second mask and the optical flow map matrix further includes:

[0144] An optical flow matrix is ​​obtained based on the optical flow map matrix, and each element of the optical flow matrix is ​​the modulus of the corresponding element in the optical flow map.

[0145] According to an embodiment of the present disclosure, the optical flow map matrix is, for example, a reference Figure 2 The optical flow matrix O described is, for example, the reference Figure 2 Describes the optical flow matrix OF.

[0146] According to an embodiment of the present disclosure, generating a third mask according to the first mask and the optical flow map includes:

[0147] The first mask is pixel-remapped based on the optical flow map matrix to generate the third mask.

[0148] According to an embodiment of the present disclosure, the third mask is, for example, a reference Figure 2 A third mask AM is described.

[0149] The step of generating a fourth mask according to the third mask, the second mask, and the optical flow map comprises:

[0150] Obtaining a first product obtained by performing a dot multiplication of the third mask and an inverse result of the optical flow matrix, wherein the inverse result of the optical flow matrix is ​​obtained by subtracting the optical flow matrix from a unit matrix;

[0151] Obtaining a second product obtained by performing a dot product of the second mask and the optical flow matrix;

[0152] The fourth mask is generated according to the sum of the first product and the second product.

[0153] According to an embodiment of the present disclosure, the inverse result of the optical flow matrix is, for example, a reference Figure 2 The first product is described as I-OF, for example, with reference to Figure 2AM*(I-OF) described, the second product is, for example, reference Figure 2 Description of CM*OF.

[0154] According to an embodiment of the present disclosure, generating the target mask according to the fourth mask includes:

[0155] The fourth mask is corrected by using the second mask and the target object edge detection result of the second frame image, and the corrected fourth mask is used as the target mask.

[0156] According to an embodiment of the present disclosure, the target object edge detection result of the second frame image is, for example, a reference Figure 2 Describes the edge matrix E.

[0157] According to an embodiment of the present disclosure, the step of correcting the fourth mask by using the second mask and the target object edge detection result of the second frame image includes:

[0158] Obtaining a third product obtained by performing a dot multiplication of the fourth mask and an inverse result of a target object edge detection result of the second frame image, wherein the inverse result of the target object edge detection result is obtained by subtracting the inverse result of the target object edge detection result from the unit matrix;

[0159] Obtaining a fourth product obtained by performing a dot product of the second mask and the target object edge detection result;

[0160] The target mask is generated according to the sum of the third product and the fourth product.

[0161] According to an embodiment of the present disclosure, the inverse result of the target object edge detection result of the second frame image is, for example, a reference Figure 2 The third product is described in IE, for example, by reference Figure 2 The fourth product of BM*(IE) described is, for example, reference Figure 2 Description of CM*E.

[0162] According to an embodiment of the present disclosure, generating the target mask according to the sum of the third product and the fourth product includes:

[0163] The value of each unit display block in the sum of the third product and the fourth product is compared with a corresponding preset threshold to obtain a corresponding element in the target mask.

[0164] According to an embodiment of the present disclosure, the value of each unit display block in the sum of the third product and the fourth product can be compared with a threshold value corresponding to the value interval in which the value of the unit display block is located, so as to obtain the value of the corresponding unit display block in the target mask. For example, assuming that the value of the unit display block in the sum of the third product and the fourth product is Z, and the upper and lower limits of its value interval are two integers Q and Q+1, respectively, where Q≤Z≤Q+1, then Z can be compared with a threshold value Q+0.5 corresponding to the value interval, and if Z is less than the threshold value, the value of the unit display block corresponding to Z in the target mask is Q, otherwise the value of the unit display block corresponding to Z in the target mask is Q+1.

[0165] According to an embodiment of the present disclosure, acquiring a first mask corresponding to a first frame image in a video stream and a second mask corresponding to a second frame image after the first frame image includes:

[0166] When the first frame image is the first frame image in the video stream, the first mask is obtained by inputting the first frame image into a trained mask generation model; when the first frame image is not the first frame image in the video stream, a target mask of the first frame image obtained according to the first frame image and a frame image before the first frame image is used as the first mask;

[0167] The second mask is obtained by inputting the second frame image into a trained mask generation model.

[0168] For example, refer to Figure 2 When the first frame image is the first frame image in the video stream, the first mask PM is obtained by inputting the first frame image into the trained mask generation model Seg Net. When the first frame image is not the first frame image in the video stream, the target mask of the first frame image obtained according to the first frame image and the frame image before the first frame image is used as the first mask PM. The second frame image is input into Seg Net to obtain the second mask CM.

[0169] Figure 3A The structural block diagram of the image processing device according to the embodiment of the present disclosure is shown. The device can be implemented as part or all of the electronic device through software, hardware or a combination of both.

[0170] like Figure 3A As shown, the image processing device 310 includes a first acquisition module 311 , a first generation module 312 , and a second generation module 313 .

[0171] The first acquisition module 311 is configured to acquire a mask corresponding to an image frame in a video stream;

[0172] The first generating module 312 is configured to generate an optical flow map matrix corresponding to the image frame according to the image frame;

[0173] The second generating module 313 is configured to generate a target mask for segmenting a target object from the video stream according to the mask and the optical flow map matrix.

[0174] Figure 3B The structural block diagram of the image processing device according to the embodiment of the present disclosure is shown. The device can be implemented as part or all of the electronic device through software, hardware or a combination of both.

[0175] like Figure 3B As shown, the image processing device 320 includes a receiving module 321 , a second acquiring module 322 , a third acquiring module 323 , a first generating module 312 , and a second generating module 313 .

[0176] The receiving module 321 is configured to receive a request for generating a target mask for segmenting a target object from a video stream, wherein the request includes video stream identification information and target object identification information;

[0177] The second acquisition module 322 is configured to acquire the video stream according to the video stream identification information, and acquire the corresponding mask generation model according to the target object identification information;

[0178] The third acquisition module 323 is configured to acquire a mask corresponding to the image frame in the video stream through the mask generation model;

[0179] The first generating module 312 is configured to generate an optical flow map matrix corresponding to the image frame according to the image frame;

[0180] The second generating module 313 is configured to generate a target mask for segmenting a target object from the video stream according to the mask and the optical flow map matrix.

[0181] According to an embodiment of the present disclosure, the method of obtaining a mask corresponding to an image frame in a video stream includes: obtaining a first mask corresponding to a first frame image in the video stream and a second mask corresponding to a second frame image after the first frame image, wherein the first mask is used to segment a target object from the first frame image, and the second mask is used to segment a target object from the second frame image.

[0182] According to an embodiment of the present disclosure, acquiring a first mask corresponding to a first frame image in a video stream and a second mask corresponding to a second frame image after the first frame image includes:

[0183] When the first frame image is the first frame image in the video stream, the first mask is obtained by inputting the first frame image into a trained mask generation model; when the first frame image is not the first frame image in the video stream, a target mask of the first frame image obtained according to the first frame image and a frame image before the first frame image is used as the first mask;

[0184] The second mask is obtained by inputting the second frame image into a trained mask generation model.

[0185] According to an embodiment of the present disclosure, generating an optical flow map matrix corresponding to the image frame based on the image frame includes: generating an optical flow map matrix based on the first frame image and the second frame image, each element in the optical flow map matrix is ​​an optical flow vector of a corresponding unit display block in the first frame image, and the unit display block includes one or more pixels.

[0186] According to an embodiment of the present disclosure, generating a target mask for segmenting a target object from the video stream based on the mask and the optical flow map matrix includes: generating a target mask corresponding to the second frame image based on the first mask, the second mask and the optical flow map matrix, and the target mask is used to segment the target object from the second frame image.

[0187] According to an embodiment of the present disclosure, generating a target mask corresponding to the second frame image according to the first mask, the second mask and the optical flow map matrix includes:

[0188] generating a third mask according to the first mask and the optical flow map matrix;

[0189] generating a fourth mask according to the third mask, the second mask, and the optical flow map;

[0190] The target mask is generated according to the fourth mask.

[0191] According to an embodiment of the present disclosure, generating a target mask corresponding to the second frame image according to the first mask, the second mask and the optical flow map matrix further includes:

[0192] An optical flow matrix is ​​obtained based on the optical flow map matrix, and each element of the optical flow matrix is ​​the modulus of the corresponding element in the optical flow map.

[0193] According to an embodiment of the present disclosure, generating a third mask according to the first mask and the optical flow map includes:

[0194] The first mask is pixel-remapped based on the optical flow map matrix to generate the third mask.

[0195] According to an embodiment of the present disclosure, generating a fourth mask according to the third mask, the second mask, and the optical flow map includes:

[0196] Obtaining a first product obtained by performing a dot multiplication of the third mask and an inverse result of the optical flow matrix, wherein the inverse result of the optical flow matrix is ​​obtained by subtracting the optical flow matrix from a unit matrix;

[0197] Obtaining a second product obtained by performing a dot product of the second mask and the optical flow matrix;

[0198] The fourth mask is generated according to the sum of the first product and the second product.

[0199] According to an embodiment of the present disclosure, generating the target mask according to the fourth mask includes:

[0200] The fourth mask is corrected by using the second mask and the target object edge detection result of the second frame image, and the corrected fourth mask is used as the target mask.

[0201] According to an embodiment of the present disclosure, the step of correcting the fourth mask by using the second mask and the target object edge detection result of the second frame image includes:

[0202] Obtaining a third product obtained by performing a dot multiplication of the fourth mask and an inverse result of a target object edge detection result of the second frame image, wherein the inverse result of the target object edge detection result is obtained by subtracting the inverse result of the target object edge detection result from the unit matrix;

[0203] Obtaining a fourth product obtained by performing a dot product of the second mask and the target object edge detection result;

[0204] The target mask is generated according to the sum of the third product and the fourth product.

[0205] According to an embodiment of the present disclosure, generating the target mask according to the sum of the third product and the fourth product includes:

[0206] The value of each unit display block in the sum of the third product and the fourth product is compared with a corresponding preset threshold to obtain a corresponding element in the target mask.

[0207] The present disclosure also discloses an electronic device, Figure 4 A structural block diagram of an electronic device according to an embodiment of the present disclosure is shown.

[0208] like Figure 4As shown, the electronic device 400 includes a memory 401 and a processor 402, wherein the memory 401 is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor 402 to implement the method according to an embodiment of the present disclosure.

[0209] According to an embodiment of the present disclosure, an image processing method includes:

[0210] Obtaining a mask corresponding to an image frame in a video stream;

[0211] According to the image frame, generating an optical flow map matrix corresponding to the image frame;

[0212] A target mask for segmenting a target object from the video stream is generated according to the mask and the optical flow map matrix.

[0213] According to an embodiment of the present disclosure, the method of obtaining a mask corresponding to an image frame in a video stream includes: obtaining a first mask corresponding to a first frame image in the video stream and a second mask corresponding to a second frame image after the first frame image, wherein the first mask is used to segment a target object from the first frame image, and the second mask is used to segment a target object from the second frame image.

[0214] According to an embodiment of the present disclosure, acquiring a first mask corresponding to a first frame image in a video stream and a second mask corresponding to a second frame image after the first frame image includes:

[0215] When the first frame image is the first frame image in the video stream, the first mask is obtained by inputting the first frame image into a trained mask generation model; when the first frame image is not the first frame image in the video stream, a target mask of the first frame image obtained according to the first frame image and a frame image before the first frame image is used as the first mask;

[0216] The second mask is obtained by inputting the second frame image into a trained mask generation model.

[0217] According to an embodiment of the present disclosure, generating an optical flow map matrix corresponding to the image frame based on the image frame includes: generating an optical flow map matrix based on the first frame image and the second frame image, each element in the optical flow map matrix is ​​an optical flow vector of a corresponding unit display block in the first frame image, and the unit display block includes one or more pixels.

[0218] According to an embodiment of the present disclosure, generating a target mask for segmenting a target object from the video stream based on the mask and the optical flow map matrix includes: generating a target mask corresponding to the second frame image based on the first mask, the second mask and the optical flow map matrix, and the target mask is used to segment the target object from the second frame image.

[0219] According to an embodiment of the present disclosure, generating a target mask corresponding to the second frame image according to the first mask, the second mask and the optical flow map matrix includes:

[0220] generating a third mask according to the first mask and the optical flow map matrix;

[0221] generating a fourth mask according to the third mask, the second mask, and the optical flow map;

[0222] The target mask is generated according to the fourth mask.

[0223] According to an embodiment of the present disclosure, generating a third mask according to the first mask and the optical flow map includes:

[0224] The first mask is pixel-remapped based on the optical flow map matrix to generate the third mask.

[0225] According to an embodiment of the present disclosure, generating a target mask corresponding to the second frame image according to the first mask, the second mask and the optical flow map matrix further includes:

[0226] An optical flow matrix is ​​obtained based on the optical flow map matrix, and each element of the optical flow matrix is ​​the modulus of the corresponding element in the optical flow map.

[0227] According to an embodiment of the present disclosure, generating a fourth mask according to the third mask, the second mask, and the optical flow map includes:

[0228] Obtaining a first product obtained by performing a dot multiplication of the third mask and an inverse result of the optical flow matrix, wherein the inverse result of the optical flow matrix is ​​obtained by subtracting the optical flow matrix from a unit matrix;

[0229] Obtaining a second product obtained by performing a dot product of the second mask and the optical flow matrix;

[0230] The fourth mask is generated according to the sum of the first product and the second product.

[0231] According to an embodiment of the present disclosure, generating the target mask according to the fourth mask includes:

[0232] The fourth mask is corrected by using the second mask and the target object edge detection result of the second frame image, and the corrected fourth mask is used as the target mask.

[0233] According to an embodiment of the present disclosure, the step of correcting the fourth mask by using the second mask and the target object edge detection result of the second frame image includes:

[0234] Obtaining a third product obtained by performing a dot multiplication of the fourth mask and an inverse result of a target object edge detection result of the second frame image, wherein the inverse result of the target object edge detection result is obtained by subtracting the inverse result of the target object edge detection result from the unit matrix;

[0235] Obtaining a fourth product obtained by performing a dot product of the second mask and the target object edge detection result;

[0236] The target mask is generated according to the sum of the third product and the fourth product.

[0237] According to an embodiment of the present disclosure, generating the target mask according to the sum of the third product and the fourth product includes:

[0238] The value of each unit display block in the sum of the third product and the fourth product is compared with a corresponding preset threshold to obtain the value of the corresponding unit display block in the target mask.

[0239] According to an embodiment of the present disclosure, an image processing method includes:

[0240] receiving a request for generating a target mask for segmenting a target object from a video stream, the request including video stream identification information and target object identification information;

[0241] According to the video stream identification information, the video stream is acquired, and according to the target object identification information, a corresponding mask generation model is acquired;

[0242] Acquire a mask corresponding to an image frame in the video stream through the mask generation model;

[0243] According to the image frame, generating an optical flow map matrix corresponding to the image frame;

[0244] A target mask for segmenting a target object from the video stream is generated according to the mask and the optical flow map matrix.

[0245] Figure 5 A schematic diagram showing the structure of a computer system suitable for implementing the method according to an embodiment of the present disclosure is shown.

[0246] like Figure 5As shown, the computer system 500 includes a processing unit 501, which can perform various processes in the above-mentioned embodiments according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage part 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the system 500 are also stored. The processing unit 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0247] The following components are connected to the I / O interface 505: an input part 506 including a keyboard, a mouse, etc.; an output part 507 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker; a storage part 508 including a hard disk, etc.; and a communication part 509 including a network interface card such as a LAN card, a modem, etc. The communication part 509 performs communication processing via a network such as the Internet. The drive 510 is also connected to the I / O interface 505 as needed. Removable media 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., are installed on the drive 510 as needed, so that the computer program read therefrom is installed into the storage part 508 as needed. Among them, the processing unit 501 can be implemented as a processing unit such as a CPU, a GPU, a TPU, an FPGA, an NPU, etc.

[0248] In particular, according to an embodiment of the present disclosure, the method described above can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program tangibly contained on a computer-readable storage medium, and the computer program includes a program code for executing the above method. In such an embodiment, the computer program can be downloaded and installed from a network through the communication part 509, and / or installed from a removable medium 511.

[0249] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0250] The units or modules involved in the embodiments described in the present disclosure may be implemented by software or programmable hardware. The units or modules described may also be set in a processor, and the names of these units or modules do not constitute limitations on the units or modules themselves in some cases.

[0251] As another aspect, the present disclosure further provides a computer-readable storage medium, which may be a computer-readable storage medium included in the electronic device or computer system in the above embodiment; or a computer-readable storage medium that exists independently and is not assembled into a device. The computer-readable storage medium stores one or more programs, and the programs are used by one or more processors to execute the method described in the present disclosure.

[0252] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present disclosure is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the inventive concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other.

Claims

1. An image processing method, comprising: Acquiring a mask corresponding to an image frame in a video stream, comprising: acquiring a first mask corresponding to a first image frame in the video stream and a second mask corresponding to a second image frame after the first image frame; According to the image frame, generating an optical flow map matrix corresponding to the image frame; According to the mask and the optical flow map matrix, a target mask for segmenting a target object from the video stream is generated, including: generating a third mask according to the first mask and the optical flow map matrix; generating a fourth mask according to the third mask, the second mask, and the optical flow map; and generating the target mask according to the fourth mask.

2. The method according to claim 1, wherein: The first mask is used to segment the target object from the first frame image, and the second mask is used to segment the target object from the second frame image.

3. The method according to claim 2, wherein: The acquiring of a first mask corresponding to a first frame image in a video stream and a second mask corresponding to a second frame image after the first frame image comprises: when the first frame image is the first frame image in the video stream, obtaining the first mask by inputting the first frame image into a trained mask generation model; and when the first frame image is not the first frame image in the video stream, using a target mask of the first frame image obtained according to the first frame image and a frame image before the first frame image as the first mask; The second mask is obtained by inputting the second frame image into a trained mask generation model.

4. The method according to claim 1, wherein: The step of generating an optical flow map matrix corresponding to the image frame according to the image frame includes: An optical flow map matrix is ​​generated according to the first frame image and the second frame image, wherein each element in the optical flow map matrix is ​​an optical flow vector of a corresponding unit display block in the first frame image, and the unit display block includes one or more pixels.

5. The method according to claim 1, wherein: The step of generating a target mask for segmenting a target object from the video stream according to the mask and the optical flow map matrix comprises: A target mask corresponding to the second frame image is generated according to the first mask, the second mask and the optical flow map matrix, and the target mask is used to segment a target object from the second frame image.

6. The method according to claim 1, wherein: The generating a third mask according to the first mask and the optical flow map comprises: The first mask is pixel-remapped based on the optical flow map matrix to generate the third mask.

7. The method according to claim 1, wherein: The step of generating a target mask corresponding to the second frame image according to the first mask, the second mask and the optical flow map matrix further includes: An optical flow matrix is ​​obtained based on the optical flow map matrix, and each element of the optical flow matrix is ​​the modulus of the corresponding element in the optical flow map.

8. The method according to claim 7, wherein: The step of generating a fourth mask according to the third mask, the second mask, and the optical flow map comprises: Obtaining a first product obtained by performing a dot multiplication of the third mask and an inverse result of the optical flow matrix, wherein the inverse result of the optical flow matrix is ​​obtained by subtracting the optical flow matrix from the unit matrix; Obtaining a second product obtained by performing a dot product of the second mask and the optical flow matrix; The fourth mask is generated according to the sum of the first product and the second product.

9. The method according to claim 1, wherein: Generating the target mask according to the fourth mask includes: The fourth mask is corrected by using the second mask and the target object edge detection result of the second frame image, and the corrected fourth mask is used as the target mask.

10. The method according to claim 9, wherein: The step of correcting the fourth mask by using the second mask and the target object edge detection result of the second frame image includes: Obtaining a third product obtained by performing a dot multiplication of the fourth mask and an inverse result of a target object edge detection result of the second frame image, wherein the inverse result of the target object edge detection result is obtained by subtracting the inverse result of the target object edge detection result from the unit matrix; Obtaining a fourth product obtained by performing a dot product of the second mask and the target object edge detection result; The target mask is generated according to the sum of the third product and the fourth product.

11. The method according to claim 10, wherein: Generating the target mask according to the sum of the third product and the fourth product includes: The value of each unit display block in the sum of the third product and the fourth product is compared with a corresponding preset threshold to obtain the value of the corresponding unit display block in the target mask.

12. An image processing method, comprising: receiving a request for generating a target mask for segmenting a target object from a video stream, the request including video stream identification information and target object identification information; According to the video stream identification information, the video stream is acquired, and according to the target object identification information, a corresponding mask generation model is acquired; Acquiring a mask corresponding to an image frame in the video stream by using the mask generation model, including: acquiring a first mask corresponding to a first image frame in the video stream and a second mask corresponding to a second image frame after the first image frame; According to the image frame, generating an optical flow map matrix corresponding to the image frame; According to the mask and the optical flow map matrix, a target mask for segmenting a target object from the video stream is generated, including: generating a third mask according to the first mask and the optical flow map matrix; generating a fourth mask according to the third mask, the second mask, and the optical flow map; and generating the target mask according to the fourth mask.

13. An image processing device, comprising: A first acquisition module is configured to acquire a mask corresponding to an image frame in a video stream, including: acquiring a first mask corresponding to a first image frame in the video stream and a second mask corresponding to a second image frame after the first image frame; A first generating module is configured to generate an optical flow map matrix corresponding to the image frame according to the image frame; The second generating module is configured to generate a target mask for segmenting the target object from the video stream according to the mask and the optical flow map matrix, including: generating a third mask according to the first mask and the optical flow map matrix; generating a fourth mask according to the third mask, the second mask, and the optical flow map; and generating the target mask according to the fourth mask.

14. An image processing device, comprising: A receiving module is configured to receive a request for generating a target mask for segmenting a target object from a video stream, wherein the request includes video stream identification information and target object identification information; A second acquisition module is configured to acquire the video stream according to the video stream identification information, and acquire a corresponding mask generation model according to the target object identification information; A third acquisition module is configured to acquire a mask corresponding to an image frame in the video stream through the mask generation model, including: acquiring a first mask corresponding to a first image frame in the video stream and a second mask corresponding to a second image frame after the first image frame; A first generating module is configured to generate an optical flow map matrix corresponding to the image frame according to the image frame; The second generating module is configured to generate a target mask for segmenting the target object from the video stream according to the mask and the optical flow map matrix, including: generating a third mask according to the first mask and the optical flow map matrix; generating a fourth mask according to the third mask, the second mask, and the optical flow map; and generating the target mask according to the fourth mask.

15. An electronic device comprising a memory and a processor; wherein: The memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the method steps of any one of claims 1-12.

16. A readable storage medium having computer instructions stored thereon, wherein the computer instructions, when executed by a processor, implement the method steps of any one of claims 1-12.

Images