An image segmentation method, device, electronic equipment and storage medium

Abstract

The application discloses an image segmentation method and device, electronic equipment and storage medium. When performing image segmentation on a first image to be segmented, a first network branch of an image segmentation model is used to perform feature extraction on the first image to obtain a first feature map. A second feature map corresponding to a second image of a neighboring frame is combined to obtain a spliced third feature map. The second feature map is obtained by performing feature extraction on the second image by using the first network branch and a second network branch of the image segmentation model. Finally, a third network branch of the image segmentation model is used to perform feature extraction on the third feature map to obtain an image segmentation result. The process of performing feature extraction on the first image by using the second network branch of the image segmentation model is reduced, thereby reducing the calculation amount of image segmentation. The first feature map and the second feature map are combined to determine the final image segmentation result, thereby avoiding poor segmentation effect and flickering display between frames.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image segmentation method, device, electronic device and storage medium. Background Art

[0002] When livestreaming on mobile phones or other devices, livestreamers may need to perform image segmentation in some scenarios, for example, to remove a person's portrait and replace it with a prepared background image. Due to the limited hardware performance of mobile phones, using image segmentation models can result in adverse effects such as frame rate drops. Currently, livestreaming platforms generally adopt the following approach: designing models with varying computational requirements based on the type of mobile phone. High-end devices require 300M floating-point operations, mid-range devices 120M floating-point operations, and low-end devices 80M floating-point operations.

[0003] In the process of implementing image segmentation, it was found that there are at least the following problems in the existing technology: mid-range or low-end machines have poor segmentation effects due to computational limitations, and the image segmentation effect is poor. In addition, due to computational limitations, flickering is likely to occur between frames. For example, a small area of ​​clothing may be detected as a portrait in the previous frame and as a background in the next frame. In this case, there will be visible flickering. Summary of the Invention

[0004] The embodiments of the present invention provide an image segmentation method, device, electronic device and storage medium to solve the problem that the image segmentation solution in the related art has poor segmentation effect due to the limitation of computational complexity, poor image segmentation effect and easy flickering display.

[0005] An embodiment of the present invention provides an image segmentation method, the method comprising:

[0006] Acquire a first image to be segmented and a second image in an adjacent frame of the first image; input the first image into a pre-trained image segmentation model, and extract features of the first image based on a first network branch of the image segmentation model to obtain a first feature map;

[0007] Obtaining a second feature map corresponding to the second image, wherein the second feature map is obtained by inputting the second image into the image segmentation model and performing feature extraction on the second image based on the first network branch and the second network branch of the image segmentation model;

[0008] The first feature map and the second feature map are spliced ​​to obtain a third feature map, and features are extracted from the third feature map based on the third network branch of the image segmentation model to obtain an image segmentation result.

[0009] In another aspect, an embodiment of the present invention provides an image segmentation device, comprising:

[0010] A first acquisition module is configured to acquire a first image to be segmented and a second image in an adjacent frame of the first image; input the first image into a pre-trained image segmentation model, and perform feature extraction on the first image based on a first network branch of the image segmentation model to obtain a first feature map;

[0011] a second acquisition module, configured to acquire a second feature map corresponding to the second image, wherein the second feature map is obtained by inputting the second image into the image segmentation model and performing feature extraction on the second image based on the first network branch and the second network branch of the image segmentation model;

[0012] The first segmentation module is used to splice the first feature map and the second feature map to obtain a third feature map, and based on the third network branch of the image segmentation model, perform feature extraction on the third feature map to obtain an image segmentation result.

[0013] In another aspect, an embodiment of the present invention provides an electronic device, comprising a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus;

[0014] Memory for storing computer programs;

[0015] The processor is configured to implement any of the above method steps when executing a program stored in the memory.

[0016] On the other hand, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of the above are implemented.

[0017] An embodiment of the present invention provides an image segmentation method, device, electronic device and storage medium, the method comprising: obtaining a first image to be segmented and a second image of an adjacent frame of the first image; inputting the first image into a pre-trained image segmentation model, and performing feature extraction on the first image based on a first network branch of the image segmentation model to obtain a first feature map; obtaining a second feature map corresponding to the second image, wherein the second feature map is obtained by inputting the second image into the image segmentation model, and performing feature extraction on the second image based on the first network branch and the second network branch of the image segmentation model; splicing the first feature map and the second feature map to obtain a third feature map, and performing feature extraction on the third feature map based on the third network branch of the image segmentation model to obtain an image segmentation result.

[0018] The above technical solution has the following advantages or beneficial effects:

[0019] Because in the embodiment of the present invention, when performing image segmentation on the first image to be segmented, the first network branch of the image segmentation model extracts features from the first image to obtain a first feature map, and combines the second feature map corresponding to the second image of the adjacent frame to obtain a spliced ​​third feature map, wherein the second feature map is obtained by extracting features from the second image based on the first network branch and the second network branch of the image segmentation model. Finally, based on the third network branch of the image segmentation model, feature extraction is performed on the third feature map to obtain an image segmentation result. In the embodiment of the present invention, the process of extracting features from the first image by the second network branch of the image segmentation model is reduced, thereby reducing the amount of computation for image segmentation, and combining the first feature map corresponding to the first image and the second feature map corresponding to the second image of the adjacent frame to determine the final image segmentation result, so that the image segmentation effect is better, avoiding the problem of poor segmentation effect due to computational limitation in the prior art, and the problem of flickering display between frames due to computational limitation. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0021] Figure 1 Schematic diagram of the image segmentation process provided in Example 1 of the present invention;

[0022] Figure 2 Schematic diagram of the segmentation neural network model model1 provided in Example 4 of the present invention;

[0023] Figure 3 Schematic diagram of the structure of the separation convolution module provided in Example 4 of the present invention;

[0024] Figure 4 This is a rendering of the separation convolution module provided in Example 4 of the present invention;

[0025] Figure 5 This is a rendering of the deconvolution module provided in Example 4 of the present invention;

[0026] Figure 6 Schematic diagram of similarity determination model model 2 provided in Example 4 of the present invention;

[0027] Figure 7 Schematic diagram of the feature reuse neural network model model 3 provided in Example 4 of the present invention;

[0028] Figure 8 A schematic structural diagram of an image segmentation device provided in Example 5 of the present invention;

[0029] Figure 9 This is a schematic diagram of the structure of an electronic device provided in Example 6 of the present invention. DETAILED DESCRIPTION

[0030] The present invention will be further described in detail below with reference to the accompanying drawings. It is apparent that the embodiments described are only some, not all, of the present invention. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort are intended to fall within the scope of protection of the present invention.

[0031] Example 1:

[0032] Figure 1 A schematic diagram of an image segmentation process provided by an embodiment of the present invention, the process includes:

[0033] S101: Acquire a first image to be segmented and a second image in an adjacent frame of the first image; input the first image into a pre-trained image segmentation model, and perform feature extraction on the first image based on a first network branch of the image segmentation model to obtain a first feature map.

[0034] S102: Obtain a second feature map corresponding to the second image, wherein the second feature map is obtained by inputting the second image into the image segmentation model and performing feature extraction on the second image based on the first network branch and the second network branch of the image segmentation model.

[0035] S103: Concatenate the first feature map and the second feature map to obtain a third feature map, and perform feature extraction on the third feature map based on the third network branch of the image segmentation model to obtain an image segmentation result.

[0036] The image segmentation method provided in embodiments of the present invention is applicable to electronic devices, such as PCs, tablets, and servers. Image segmentation can be used to segment portraits or other target objects, such as vehicles. Different image segmentation models must be trained for different target objects.

[0037] In the embodiment of the present invention, the image to be segmented is referred to as the first image, and the adjacent frame image of the first image is referred to as the second image. The adjacent frame image of the first image can be the previous frame image of the first image, or can be an image preceding the first image and separated from the first image by a preset number of frames. The preset number of frames is, for example, 2 frames or 3 frames.

[0038] The first image is input into a pre-trained image segmentation model, and features of the first image are extracted based on the first network branch of the image segmentation model to obtain a first feature map. The second image is pre-trained into the image segmentation model, and features of the second image are extracted based on the first and second network branches of the image segmentation model to obtain a second feature map. The first and second feature maps are then concatenated to obtain a third feature map, and features are extracted from the third feature map based on the third network branch of the image segmentation model to obtain the image segmentation result.

[0039] Among them, the first network branch includes a first convolution module; the second network branch includes a second convolution module, a third convolution module, a first deconvolution module and a second deconvolution module in sequence; and the third network branch includes a third deconvolution module.

[0040] Because in the embodiment of the present invention, when performing image segmentation on the first image to be segmented, the first network branch of the image segmentation model extracts features from the first image to obtain a first feature map, and combines the second feature map corresponding to the second image of the adjacent frame to obtain a spliced ​​third feature map, wherein the second feature map is obtained by extracting features from the second image based on the first network branch and the second network branch of the image segmentation model. Finally, based on the third network branch of the image segmentation model, feature extraction is performed on the third feature map to obtain an image segmentation result. In the embodiment of the present invention, the process of extracting features from the first image by the second network branch of the image segmentation model is reduced, thereby reducing the amount of computation for image segmentation, and combining the first feature map corresponding to the first image and the second feature map corresponding to the second image of the adjacent frame to determine the final image segmentation result, so that the image segmentation effect is better, avoiding the problem of poor segmentation effect due to computational limitation in the prior art, and the problem of flickering display between frames due to computational limitation.

[0041] Example 2:

[0042] To ensure the accuracy of image segmentation, based on the above embodiment, in an embodiment of the present invention, before inputting the first image into a pre-trained image segmentation model and performing feature extraction on the first image based on the first network branch of the image segmentation model to obtain a first feature map, the method further includes:

[0043] Determine the similarity between the first image and the second image. If the similarity is greater than a preset similarity threshold, input the first image into a pre-trained image segmentation model, and perform feature extraction on the first image based on a first network branch of the image segmentation model to obtain a first feature map.

[0044] In this embodiment of the present invention, the similarity between the first image and the second image is first determined, and then a determination is made as to whether the similarity is greater than a preset similarity threshold. If so, segmentation of the first image is completed by combining the first feature map corresponding to the first image and the second feature map corresponding to the second image, thereby ensuring the accuracy of the image segmentation.

[0045] When determining the similarity between the first image and the second image, convolution processing is performed on the first image and the second image to obtain a sixth feature map and a seventh feature map. The sixth feature map and the seventh feature map are input into a similarity function to obtain the similarity between the first image and the second image. The similarity function may be a cosine function, and the similarity between the first image and the second image is obtained by calculating the cosine distance between the sixth feature map and the seventh feature map.

[0046] For example:

[0047] For example, three consecutive frames of images are image A, image B, and image C. Image A is based on the first network branch, the second network branch, and the third network branch of the image segmentation model, and the image segmentation result is obtained by extracting features from image A. After calculation, the similarity between image B and image A is greater than the preset similarity threshold. When image B is segmented, the first network branch of the image segmentation model is used to extract features from image B to obtain a B feature map, and the A feature map corresponding to image A is obtained by extracting features from image A based on the first network branch and the second network branch of the image segmentation model. The B feature map and the A feature map are then spliced ​​together, and finally, the spliced ​​feature map is extracted based on the third network branch to obtain the segmentation result of image B.

[0048] When image C is segmented, since the adjacent image B does not have a feature map output by the second network branch, the similarity between image C and image A can be calculated. If the similarity between image C and image A is greater than the preset similarity threshold, when image C is segmented, the first network branch of the image segmentation model is used to extract features from image C to obtain a C feature map, and the A feature map corresponding to image A is obtained by extracting features from image A using the first and second network branches of the image segmentation model. The C feature map and the A feature map are then spliced ​​together, and finally, feature extraction is performed on the spliced ​​feature map based on the third network branch to obtain the segmentation result of image C.

[0049] The training process of the image segmentation model includes:

[0050] For the sample image in the training set, the sample image is input into the image segmentation model, and the training segmentation result corresponding to the sample image is obtained through the first network branch, the second network branch and the third network branch in the image segmentation model. According to the training segmentation result and the labeled segmentation result corresponding to the sample image, the various parameters in the first network branch, the second network branch and the third network branch in the image segmentation model are trained.

[0051] In which, during the training process, the sample image is input into the image segmentation model, and when the training segmentation result corresponding to the sample image is obtained after passing through the first network branch, the second network branch and the third network branch in the image segmentation model, the feature map output by the first network branch and the feature map output by the second network branch are spliced, and then the spliced ​​feature map is input into the third network branch to obtain the training segmentation result corresponding to the sample image.

[0052] Example 3:

[0053] In a case where the similarity between the first image and the second image is not greater than a preset similarity threshold, in order to ensure accuracy of image segmentation of the first image, based on the above embodiments, in an embodiment of the present invention, if the similarity is not greater than the preset similarity threshold, the method further includes:

[0054] The first image is input into a pre-trained image segmentation model, and based on the first network branch, the second network branch and the third network branch of the image segmentation model, features of the first image are extracted to obtain an image segmentation result.

[0055] In the embodiment of the present invention, feature extraction may be performed on the first image based on the first network branch, the second network branch, and the third network branch in sequence to obtain an image segmentation result.

[0056] Preferably, based on the first network branch, the second network branch, and the third network branch of the image segmentation model, extracting features from the first image to obtain an image segmentation result includes:

[0057] Inputting the first image into a pre-trained image segmentation model, and performing feature extraction on the first image based on a first network branch of the image segmentation model to obtain a first feature map;

[0058] Performing feature extraction on the first feature map based on the second network branch of the image segmentation model to obtain a fourth feature map;

[0059] The first feature map and the fourth feature map are spliced ​​to obtain a fifth feature map, and features are extracted from the fifth feature map based on the third network branch of the image segmentation model to obtain an image segmentation result.

[0060] In an embodiment of the present invention, first, feature extraction is performed on the first image based on the first network branch of the image segmentation model to obtain a first feature map, and then feature extraction is performed on the first feature map based on the second network branch to obtain a fourth feature map. The first feature map and the fourth feature map are then spliced ​​together to obtain a fifth feature map, and finally, feature extraction is performed on the fifth feature map based on the third network branch to obtain an image segmentation result. In an embodiment of the present invention, before outputting the image segmentation result based on the third network branch, the first feature map and the fourth feature map are spliced ​​together, so that the fifth feature map input to the third network branch contains the feature information of the first feature map and the fourth feature map. Feature extraction is performed on the fifth feature map based on the third network branch, taking into account more feature information, thereby making the image segmentation result more accurate.

[0061] Example 4:

[0062] On the basis of the above embodiments, in an embodiment of the present invention, the method further includes:

[0063] According to the first image, the image segmentation result and a pre-saved background image, the first image and the background image are fused to obtain a third image.

[0064] For example, a data stream is obtained from the client; the data stream is parsed into a screenshot p1, and p1 is scaled into a picture of size 256*128*3 to obtain the first image p2; p2 is input into the image segmentation model and the segmentation result m1 is output, which is a 256*128 size, and each value is a decimal between 0-1, the value corresponding to the portrait part is a larger value close to 1, and the value corresponding to the background part is a smaller value close to 0; based on the background picture selected by the anchor, that is, the second image b1 and the image segmentation result m1, a new picture, that is, the third image p3 = b1*(1-m1)+p2*m1, is generated, and the third image p3 is scaled to the size of the screenshot p1 and sent to the user.

[0065] The following describes this case in detail with reference to the accompanying drawings, taking portrait segmentation as an example.

[0066] The embodiment of the present invention includes three models: 1. a specially designed segmentation neural network model model 1; 2. a similarity determination model model 2; and 3. a feature reuse neural network model model 3. The details are as follows:

[0067] 1. As Figure 2As shown, the specially designed segmentation neural network model model1 has 12 channels in the first layer block1, which is much smaller than 24 in the related art; 48 channels in the second layer block2, which is larger than 32 in the related art; 96 channels in the third layer block3; 96 channels in the fourth layer block4; 48 channels in the fifth layer block5; and 12 channels in the sixth layer block6. Figure 2 In the example, the concatenation of block 1 and block 5 means that the two parts of features are put together to form a 128*64*24 feature as the input of block 6. This concatenation is an important component for realizing feature reuse, which can retain some features of the latest frame while reusing features.

[0068] Among them, block1, block2 and block3 are separate convolution modules, and their structure diagram is as follows Figure 3 As shown in Figure 1, it includes Conv 1×1 Relu6 layer, Dwise 3×3 stride=2 Relu6 layer and Conv 1×1 Linear layer. The corresponding effect diagrams of block1, block2 and block3 are shown in Figure 1. Figure 4 As shown in Figure 2, the image scale becomes smaller after convolution. Block4, block5 and block6 are deconvolution modules. The corresponding effect diagrams of block4, block5 and block6 are shown in Figure 2. Figure 5 As shown in Figure 3, the image scale becomes larger after deconvolution, achieving scale restoration.

[0069] 2. Similarity determination model model2 Figure 6 As shown in the figure below, the adjacent frame image is resized to a 256*128*3 image q0, and the current frame image is resized to a 256*128*3 image p2. The similarity determination model model2 is shown in the figure below. The convolutional layer conv: 9*9*3*1 is used to convolve q0 and p2, respectively, to obtain y0: 128*64 and y1: 128*64. The similarity between q0 and p2 is then calculated based on the similarity function, that is, score = cos(y0, y1). If score is greater than the preset similarity threshold s, the feature reuse neural network model model3 is executed, otherwise the segmentation neural network model model1 is executed.

[0070] 3. Feature reuse neural network model model3 Figure 7 As shown in the figure, the output of Block 1 in model 1 corresponding to the current image is concatenated with the output of block 5 in model 1 corresponding to the adjacent frame image to obtain a 128*64*24 feature, and deconvolution is performed to obtain the image segmentation result.

[0071] The present invention adopts a portrait segmentation system and method for similarity judgment and feature reuse, which reduces the computational complexity by 30%, improves the inference speed by 10%, effectively reduces the frame rate drop, and the segmentation results of similar frames are basically consistent, effectively reducing the flickering phenomenon.

[0072] Example 5:

[0073] Figure 8 This is a schematic diagram of the structure of an image segmentation device provided by an embodiment of the present invention, the device comprising:

[0074] The first acquisition module 81 is configured to acquire a first image to be segmented and a second image in an adjacent frame of the first image; input the first image into a pre-trained image segmentation model, and perform feature extraction on the first image based on a first network branch of the image segmentation model to obtain a first feature map;

[0075] A second acquisition module 82 is configured to acquire a second feature map corresponding to the second image, wherein the second feature map is obtained by inputting the second image into the image segmentation model and performing feature extraction on the second image based on the first network branch and the second network branch of the image segmentation model;

[0076] The first segmentation module 83 is used to splice the first feature map and the second feature map to obtain a third feature map, and perform feature extraction on the third feature map based on the third network branch of the image segmentation model to obtain an image segmentation result.

[0077] The device further comprises:

[0078] The judgment module 84 is configured to determine the similarity between the first image and the second image, and execute the first acquisition module if the similarity is greater than a preset similarity threshold.

[0079] If the similarity is not greater than a preset similarity threshold, the device further includes:

[0080] The second segmentation module 85 is used to input the first image into a pre-trained image segmentation model, and perform feature extraction on the first image based on the first network branch, the second network branch and the third network branch of the image segmentation model to obtain an image segmentation result.

[0081] The second segmentation module 85 is specifically used to input the first image into a pre-trained image segmentation model, perform feature extraction on the first image based on the first network branch of the image segmentation model to obtain a first feature map; perform feature extraction on the first feature map based on the second network branch of the image segmentation model to obtain a fourth feature map; splice the first feature map and the fourth feature map to obtain a fifth feature map, and perform feature extraction on the fifth feature map based on the third network branch of the image segmentation model to obtain an image segmentation result.

[0082] The device further comprises:

[0083] The fusion module 86 is configured to fuse the first image with the background image according to the first image, the image segmentation result and a pre-saved background image to obtain a third image.

[0084] The judgment module is specifically used to perform convolution processing on the first image and the second image respectively to obtain a sixth feature map and a seventh feature map; and input the sixth feature map and the seventh feature map into a similarity function to obtain the similarity between the first image and the second image.

[0085] The device further comprises:

[0086] The training module 87 is used to input the sample image in the training set into the image segmentation model, obtain the training segmentation result corresponding to the sample image through the first network branch, the second network branch and the third network branch in the image segmentation model, and train the various parameters in the first network branch, the second network branch and the third network branch in the image segmentation model based on the training segmentation result and the labeled segmentation result corresponding to the sample image.

[0087] Example 6:

[0088] On the basis of the above embodiments, an electronic device is further provided in an embodiment of the present invention, such as Figure 9 As shown, it includes: a processor 301, a communication interface 302, a memory 303 and a communication bus 304, wherein the processor 301, the communication interface 302, and the memory 303 communicate with each other through the communication bus 304;

[0089] The memory 303 stores a computer program. When the program is executed by the processor 301, the processor 301 performs the following steps:

[0090] Acquire a first image to be segmented and a second image in an adjacent frame of the first image; input the first image into a pre-trained image segmentation model, and extract features of the first image based on a first network branch of the image segmentation model to obtain a first feature map;

[0091] Obtaining a second feature map corresponding to the second image, wherein the second feature map is obtained by inputting the second image into the image segmentation model and performing feature extraction on the second image based on the first network branch and the second network branch of the image segmentation model;

[0092] The first feature map and the second feature map are spliced ​​to obtain a third feature map, and features are extracted from the third feature map based on the third network branch of the image segmentation model to obtain an image segmentation result.

[0093] Based on the same inventive concept, an electronic device is also provided in an embodiment of the present invention. Since the principle of solving the problem by the above electronic device is similar to that of the image segmentation method, the implementation of the above electronic device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0094] The electronic device provided in the embodiment of the present invention may specifically be a desktop computer, a portable computer, a smart phone, a tablet computer, a personal digital assistant (PDA), a network side device, etc.

[0095] The communication bus mentioned in the electronic device mentioned above may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is used in the figure, but this does not mean that there is only one bus or only one type of bus.

[0096] The communication interface 302 is used for communication between the electronic device and other devices.

[0097] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk memory. Alternatively, the memory may be at least one storage device located away from the processor.

[0098] The above-mentioned processor can be a general-purpose processor, including a central processing unit, a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc.

[0099] Example 7:

[0100] Based on the above embodiments, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program executable by an electronic device. When the program is executed on the electronic device, the electronic device implements the following steps:

[0101] Acquire a first image to be segmented and a second image in an adjacent frame of the first image; input the first image into a pre-trained image segmentation model, and extract features of the first image based on a first network branch of the image segmentation model to obtain a first feature map;

[0102] Obtaining a second feature map corresponding to the second image, wherein the second feature map is obtained by inputting the second image into the image segmentation model and performing feature extraction on the second image based on the first network branch and the second network branch of the image segmentation model;

[0103] The first feature map and the second feature map are spliced ​​to obtain a third feature map, and features are extracted from the third feature map based on the third network branch of the image segmentation model to obtain an image segmentation result.

[0104] Based on the same inventive concept, a computer-readable storage medium is also provided in an embodiment of the present invention. Since the principle of solving the problem when the processor executes the computer program stored on the above-mentioned computer-readable storage medium is similar to the image segmentation method, the implementation of the processor when executing the computer program stored on the above-mentioned computer-readable storage medium can refer to the implementation of the method, and the repeated parts will not be repeated.

[0105] The above-mentioned computer-readable storage medium can be any available medium or data storage device that can be accessed by the processor in the electronic device, including but not limited to magnetic storage such as floppy disks, hard disks, magnetic tapes, magneto-optical disks (MO), etc., optical storage such as CDs, DVDs, BDs, HVDs, etc., and semiconductor storage such as ROMs, EPROMs, EEPROMs, non-volatile memories (NANDFLASH), solid-state drives (SSDs), etc.

[0106] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0107] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0108] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0109] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0110] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. An image segmentation method, characterized in that: The method comprises: Acquire a first image to be segmented and a second image of an adjacent frame of the first image; input the first image into a pre-trained image segmentation model, and extract features of the first image based on a first network branch of the image segmentation model to obtain a first feature map; Obtaining a second feature map corresponding to the second image, wherein the second feature map is obtained by inputting the second image into the image segmentation model and performing feature extraction on the second image based on the first network branch and the second network branch of the image segmentation model; splicing the first feature map and the second feature map to obtain a third feature map, and performing feature extraction on the third feature map based on the third network branch of the image segmentation model to obtain an image segmentation result; Before inputting the first image into a pre-trained image segmentation model and performing feature extraction on the first image based on a first network branch of the image segmentation model to obtain a first feature map, the method further includes: Determine the similarity between the first image and the second image. If the similarity is greater than a preset similarity threshold, input the first image into a pre-trained image segmentation model, and perform feature extraction on the first image based on a first network branch of the image segmentation model to obtain a first feature map.

2. The method according to claim 1, wherein If the similarity is not greater than a preset similarity threshold, the method further includes: The first image is input into a pre-trained image segmentation model, and based on the first network branch, the second network branch and the third network branch of the image segmentation model, features of the first image are extracted to obtain an image segmentation result.

3. The method according to claim 2, wherein Based on the first network branch, the second network branch, and the third network branch of the image segmentation model, extracting features from the first image to obtain an image segmentation result includes: Inputting the first image into a pre-trained image segmentation model, and performing feature extraction on the first image based on a first network branch of the image segmentation model to obtain a first feature map; Performing feature extraction on the first feature map based on the second network branch of the image segmentation model to obtain a fourth feature map; The first feature map and the fourth feature map are spliced ​​to obtain a fifth feature map, and features are extracted from the fifth feature map based on the third network branch of the image segmentation model to obtain an image segmentation result.

4. The method according to claim 1, wherein The method further comprises: According to the first image, the image segmentation result and a pre-saved background image, the first image and the background image are fused to obtain a third image.

5. The method according to claim 1, wherein The first network branch includes a first convolution module; the second network branch includes a second convolution module, a third convolution module, a first deconvolution module and a second deconvolution module in sequence; the third network branch includes a third deconvolution module.

6. The method according to claim 1, wherein The training process of the image segmentation model includes: For the sample image in the training set, the sample image is input into the image segmentation model, and the training segmentation result corresponding to the sample image is obtained through the first network branch, the second network branch and the third network branch in the image segmentation model. According to the training segmentation result and the labeled segmentation result corresponding to the sample image, the various parameters in the first network branch, the second network branch and the third network branch in the image segmentation model are trained.

7. An image segmentation device, characterized in that: The device comprises: A first acquisition module is configured to acquire a first image to be segmented and a second image in an adjacent frame of the first image; input the first image into a pre-trained image segmentation model, and perform feature extraction on the first image based on a first network branch of the image segmentation model to obtain a first feature map; a second acquisition module, configured to acquire a second feature map corresponding to the second image, wherein the second feature map is obtained by inputting the second image into the image segmentation model and performing feature extraction on the second image based on the first network branch and the second network branch of the image segmentation model; A first segmentation module is configured to concatenate the first feature map and the second feature map to obtain a third feature map, and perform feature extraction on the third feature map based on a third network branch of the image segmentation model to obtain an image segmentation result; The device further comprises: A judgment module is used to determine the similarity between the first image and the second image. If the similarity is greater than a preset similarity threshold, the first image is input into a pre-trained image segmentation model, and features of the first image are extracted based on the first network branch of the image segmentation model to obtain a first feature map.

8. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; Memory for storing computer programs; A processor, configured to implement the method steps described in any one of claims 1 to 6 when executing a program stored in a memory.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps according to any one of claims 1 to 6 are implemented.

Images