Foreground and background segmentation method, electronic device, computer readable medium

Abstract

The application provides a foreground and background segmentation method, an electronic device and a computer readable medium. The foreground and background segmentation method comprises: obtaining first feature data of to-be-segmented image data; performing feature compression and aggregation processing on the first feature data by using an encoder in a trained foreground and background segmentation network to obtain third feature data; wherein the foreground and background segmentation network comprises an encoder, a network bottom block and a decoder; calculating the third feature data by using the encoder through a self-attention mechanism to obtain channel adaptive weights; calculating fourth feature data according to the channel adaptive weights and the first feature data by using the encoder; and inputting the fourth feature data obtained by processing the first feature data through the encoder into the network bottom block to perform processing through the network bottom block and the decoder to obtain a first foreground and background segmentation result.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to foreground / background segmentation methods, electronic devices, and computer-readable media. Background Technology

[0002] In image processing, foreground and background segmentation is necessary in certain scenarios for subsequent image processing and analysis. For example, in sports, accurately tracking players is crucial for generating relevant statistics such as ball control, player speed, and distance traveled. These statistics are useful to professionals, such as coaches, and enhance the entertainment value for viewers. Before tracking, each frame of the input video typically needs to be segmented to extract the object of interest within each frame. This is considered a preprocessing stage upon which other higher-order vision tasks rely, and for segmentation algorithms, ensuring above-real-time processing speed and low error resolution is paramount.

[0003] Current segmentation algorithms do not achieve ideal segmentation results. Summary of the Invention

[0004] This application provides a foreground / background segmentation method, an electronic device, and a computer-readable medium.

[0005] In a first aspect, embodiments of this application provide a foreground / background segmentation method, comprising: acquiring first feature data of image data to be segmented; performing feature compression and aggregation processing on the first feature data using an encoder in a trained foreground / background segmentation network to obtain third feature data; wherein the foreground / background segmentation network includes: an encoder, a network bottom block, and a decoder; using the encoder to calculate channel adaptive weights on the third feature data through a self-attention mechanism; using the encoder to calculate fourth feature data based on the channel adaptive weights and the first feature data; and inputting the fourth feature data obtained after processing the first feature data by the encoder into the network bottom block for processing by the network bottom block and the decoder to obtain a first foreground / background segmentation result.

[0006] Secondly, embodiments of this application provide an electronic device, including: at least one processor; and a memory storing at least one program, which, when executed by the at least one processor, implements any of the foreground / background segmentation methods described above.

[0007] Thirdly, embodiments of this application provide a computer-readable medium storing a computer program, which, when executed by a processor, implements any of the foreground / background segmentation methods described above.

[0008] The foreground and background segmentation method provided in this application performs feature compression and aggregation on the second feature data, and then uses a self-attention mechanism to obtain adaptive channel weights. This establishes the correlation between channels and adaptively re-optimizes the response of the feature channels, thereby improving the ability to capture long-distance contextual information in foreground and background segmentation and thus improving the accuracy of foreground and background segmentation. Attached Figure Description

[0009] Figure 1 A flowchart of a foreground / background segmentation method provided in one embodiment of this application;

[0010] Figure 2 This is a schematic diagram illustrating the processing of coded blocks in a U-shaped network according to an embodiment of this application;

[0011] Figure 3 A block diagram of a foreground / background segmentation device provided in another embodiment of this application. Detailed Implementation

[0012] To enable those skilled in the art to better understand the technical solutions of this application, the foreground and background segmentation method, electronic equipment, and computer-readable medium provided in this application will be described in detail below with reference to the accompanying drawings.

[0013] Exemplary embodiments will be described more fully below with reference to the accompanying drawings; however, these exemplary embodiments may be embodied in different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this application will be thorough and complete, and will enable those skilled in the art to fully understand the scope of this application.

[0014] Where there is no conflict, the various embodiments of this application and the features thereof may be combined with each other.

[0015] As used herein, the term “and / or” includes any and all combinations of at least one related enumerated entry.

[0016] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application. As used herein, the singular forms “a” and “the” are also intended to include the plural forms unless the context clearly indicates otherwise. It will also be understood that when the terms “comprising” and / or “made of” are used in this specification, the presence of the stated feature, integral, step, operation, element, and / or component is specified, but the presence or addition of at least one other feature, integral, step, operation, element, component, and / or group thereof is not excluded.

[0017] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and this application, and will not be interpreted as having an idealized or overly formal meaning, unless expressly so defined herein.

[0018] Figure 1 This is a flowchart of a foreground / background segmentation method provided in one embodiment of this application.

[0019] Firstly, referring to Figure 1 One embodiment of this application provides a foreground / background segmentation method, including:

[0020] Step 100: Obtain the first feature data of the image data to be segmented.

[0021] In some exemplary embodiments, the first feature data can be obtained by performing multiple convolution operations on the image data to be segmented.

[0022] Assuming the size of the image to be segmented is C×H×W, where C is the number of channels, H is the height, and W is the width, then the size of the first feature data is also C×H×W.

[0023] Step 101: The encoder in the trained foreground-background segmentation network is used to perform feature compression and aggregation on the first feature data to obtain the third feature data; wherein, the foreground-background segmentation network includes: encoder, network bottom block and decoder; the encoder is used to calculate the channel adaptive weights on the third feature data through a self-attention mechanism; the encoder is used to calculate the fourth feature data based on the channel adaptive weights and the first feature data.

[0024] In some exemplary embodiments, the encoder includes at least one coding block, each coding block performing the following processing on the input second feature data: performing feature compression and aggregation processing on the second feature data to obtain N third feature data; where N is an integer greater than or equal to 3; calculating the channel adaptive weights on the N third feature data through a self-attention mechanism; calculating the fourth feature data based on the channel adaptive weights and the second feature data; wherein, when the coding block is the first coding block, the second feature data is the first feature data; when the coding block is the i-th coding block, where i is an integer greater than or equal to 2, the second feature data is the fourth feature data output by the (i-1)-th coding block.

[0025] In some exemplary embodiments, the foreground and background segmentation network can be any U-shaped network well known to those skilled in the art. For example, a U-shaped network includes an encoder, a decoder, and a network bottom block. The encoder includes at least one encoding block, and the decoder includes at least one decoding block. The output of the encoder is input to the network bottom block as its input, and the output of the network bottom block is input to the decoder as its input. The output of the decoder is the foreground and background segmentation result, such as the first and second foreground and background segmentation results mentioned in this application. The output of the preceding encoding block in the encoder is input to the following encoding block as its input, and the output of the preceding decoding block in the decoder is input to the following decoding block as its input.

[0026] In some exemplary embodiments, such as Figure 2 As shown, the process of compressing and aggregating the second feature data to obtain N third feature data includes: performing global average pooling on the second feature data to obtain global information; performing convolution and activation function operations on the global information with the k-th convolution kernel to obtain the k-th third feature data; where k is an integer greater than or equal to 1 and less than or equal to N. In other words, the second feature data is divided into N branches along the channel dimension.

[0027] In some exemplary embodiments, performing global average pooling on the second feature data to obtain global information includes: calculating the global information according to the formula w = g(x); where w is the global information, g(x) is the global average pooling function, and x is the second feature data.

[0028] In some exemplary embodiments, g(x) can be the average value of the image data corresponding to each channel in the second feature data. Then, the size of the global information obtained after average pooling is C×1×1.

[0029] In some exemplary embodiments, the convolution kernels corresponding to different channels may be the same or different; that is, the values ​​of k are different, and the k-th convolution kernel may be the same or different.

[0030] In some exemplary embodiments, the size of the k-th convolutional kernel is 1×1.

[0031] In some exemplary embodiments, performing convolution operations and activation function operations on global information with the k-th convolution kernel to obtain the k-th third feature data includes: according to the formula The k-th third feature data is calculated; where X k For the kth third feature data, σ is the third convolution function, and σ() is the first activation function.

[0032] In some exemplary embodiments, the first activation function may be the sigmoid activation function.

[0033] In some exemplary embodiments, such as Figure 2 As shown, the channel adaptive weights are calculated by using a self-attention mechanism on N third feature data, including: calculating the k-th normalized weight of the k-th third feature data based on the k-th third feature data; and determining the channel adaptive weights based on the 1st to Nth normalized weights.

[0034] In some exemplary embodiments, based on the k-th third feature data according to formula X k =σ(W2{δ(W1{x k})}) Calculate the k-th normalized weight; where X k Let be the k-th normalized weight, σ() be the first activation function, W1{} be the first convolution function, W2{} be the second convolution function, δ() be the second activation function, and x be the k-th normalized weight. k This is the third feature data corresponding to the k-th channel.

[0035] In some exemplary embodiments, the size of the convolution kernels corresponding to W1{} and W2{} is 1×1.

[0036] In some exemplary embodiments, the second activation function is the ReLU activation function.

[0037] In some exemplary embodiments, such as Figure 2 As shown, N is 3. Determining the channel adaptive weights based on the first to the Nth normalized weights includes: performing a dot product operation, scaling, and normalization on the first and second normalized weights to obtain the channel autocorrelation weight matrix; and performing a dot product operation on the channel autocorrelation weight matrix and the third normalized weight to obtain the channel adaptive weights.

[0038] In some exemplary embodiments, such as Figure 2 As shown, the channel autocorrelation weight matrix is ​​obtained by performing a dot product operation, scaling, and normalization on the first and second normalized weights. This includes: calculating the channel autocorrelation weight matrix according to the formula based on the first and second normalized weights. The channel autocorrelation weight matrix is ​​calculated; where X T d1 is the channel autocorrelation weight matrix, softmax() is the third activation function, X1 is the first normalized weight, X2 is the second normalized weight, and d2 is the scale ratio.

[0039] In this embodiment, the purpose of scaling is to prevent scale explosion in the dot product operation, and the third activation function is used to normalize the weights.

[0040] In some exemplary embodiments, such as Figure 2 As shown, the channel adaptive weights are obtained by performing a dot product operation on the channel autocorrelation weight matrix and the third normalized weights, including: according to formula X s =X T • X3 calculates the channel adaptive weights; where X s For channel adaptive weights, X T X is the channel autocorrelation weight matrix, and X3 is the third normalized weight.

[0041] In some exemplary embodiments, such as Figure 2 As shown, the fourth feature data is obtained by performing a dot product operation on the channel adaptive weights and the second feature data, including: according to formula X c =X s The fourth feature data is obtained by calculating x; where X c X is the fourth feature data. s is the channel adaptive weight, and x is the second feature data.

[0042] Step 102: Input the fourth feature data obtained after the first feature data is processed by the encoder into the network bottom block. The network bottom block and decoder process the data to obtain the first foreground and background segmentation result.

[0043] In some exemplary embodiments, the step of inputting the fourth feature data obtained after processing the first feature data by the encoder into the lower layer block of the network and processing it by the lower layer block of the network and the decoder to obtain a first foreground-background segmentation result includes: inputting the fourth feature data output from the last encoding block into the lower layer block of the network and processing it by the lower layer block of the network and the decoder to obtain a first foreground-background segmentation result.

[0044] In some exemplary embodiments, before obtaining the first feature data of the image data to be segmented, the method further includes: obtaining a training image dataset; performing enhancement preprocessing on the training image dataset; and training a foreground and background segmentation network model based on the enhanced preprocessed training image dataset.

[0045] In some exemplary embodiments, the COCO dataset can be used as the training image dataset. The COCO dataset is a dataset used for human visual cognition and visual understanding, which includes multiple scene categories. Each category in the COCO dataset includes 400 images, which meets the requirement of large-scale data for training in deep learning methods.

[0046] In some exemplary embodiments, enhancing the preprocessing of the training image dataset includes: performing foreground and background segmentation on each training image data in the training image dataset to obtain a second foreground and background segmentation result; and labeling the training image data with the second foreground and background segmentation result.

[0047] In some exemplary embodiments, the enhanced preprocessing of the training image dataset further includes: rotating, cropping, or performing other processing on the training image data to obtain more training image data.

[0048] In some exemplary embodiments, training a foreground-background segmentation network model based on an enhanced preprocessed training image dataset includes: acquiring first feature data for each enhanced preprocessed training image data in the enhanced preprocessed training image dataset, using the first feature data as input to the foreground-background segmentation network, and using a second foreground-background segmentation result as output to train the foreground-background segmentation network model.

[0049] In some exemplary embodiments, training methods well known to those skilled in the art can be used to train the foreground and background segmentation network, which will not be elaborated here.

[0050] It should be noted that the foreground and background segmentation method in this application embodiment segments the foreground data and background data in the image data. Foreground data refers to the image data of interest, and background data refers to the image data of no interest. For example, in a sports competition, the image data corresponding to the players and the ball are foreground data, and the other image data are background data.

[0051] The foreground and background segmentation method provided in this application performs feature compression and aggregation on the second feature data, and then uses a self-attention mechanism to obtain adaptive channel weights. This establishes the correlation between channels and adaptively re-optimizes the response of the feature channels, thereby improving the ability to capture long-distance contextual information in foreground and background segmentation and thus improving the accuracy of foreground and background segmentation.

[0052] Secondly, another embodiment of this application provides an electronic device, including: at least one processor; and a memory storing at least one program, which, when executed by at least one processor, implements any of the foreground / background segmentation methods described above.

[0053] Among them, the processor is a device with data processing capabilities, including but not limited to the central processing unit (CPU); the memory is a device with data storage capabilities, including but not limited to random access memory (RAM, more specifically such as SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), and flash memory (FLASH).

[0054] In some embodiments, the processor and memory are interconnected via a bus, and thus connected to other components of the computing device.

[0055] Thirdly, another embodiment of this application provides a computer-readable medium storing a computer program, which, when executed by a processor, implements any of the foreground / background segmentation methods described above.

[0056] Figure 3 A block diagram of a foreground / background segmentation device provided in another embodiment of this application.

[0057] Fourthly, refer to Figure 3 Another embodiment of this application provides a foreground / background segmentation apparatus, comprising: a feature data acquisition module 301, used to acquire first feature data of image data to be segmented; and a foreground / background segmentation module 302, used to perform feature compression and aggregation processing on the first feature data using an encoder in a trained foreground / background segmentation network to obtain third feature data; wherein the foreground / background segmentation network includes: an encoder, a network bottom block, and a decoder; the encoder calculates channel adaptive weights on the third feature data using a self-attention mechanism; the encoder calculates fourth feature data based on the channel adaptive weights and the first feature data; and the fourth feature data obtained after processing the first feature data by the encoder is input into the network bottom block, and processed by the network bottom block and the decoder to obtain a first foreground / background segmentation result.

[0058] In some exemplary embodiments, the system further includes: a dataset acquisition module 303 for acquiring a training image dataset; a preprocessing module 304 for performing enhanced preprocessing on the training image dataset; and a model training module 305 for training a foreground-background segmentation network based on the enhanced preprocessed training image dataset.

[0059] In some exemplary embodiments, the preprocessing module 304 is specifically used to: perform foreground and background segmentation on each training image data in the training image dataset to obtain a second foreground and background segmentation result; and label the training image data with the second foreground and background segmentation result.

[0060] In some exemplary embodiments, the encoder includes at least one coding block, each coding block performing the following processing on the input second feature data: performing feature compression and aggregation processing on the second feature data to obtain N third feature data; where N is an integer greater than or equal to 3; calculating the channel adaptive weights on the N third feature data using a self-attention mechanism; calculating the fourth feature data based on the channel adaptive weights and the second feature data; wherein, when the coding block is the first coding block, the second feature data is the first feature data; when the coding block is the i-th coding block, where i is an integer greater than or equal to 2, the second feature data is the fourth feature data output by the (i-1)-th coding block;

[0061] The foreground and background segmentation module 302 is specifically used to implement the following method: inputting the fourth feature data obtained after the first feature data is processed by the encoder into the lower layer block of the network, and processing it by the lower layer block of the network and the decoder to obtain the first foreground and background segmentation result: inputting the fourth feature data output by the last encoding block into the lower layer block of the network, and processing it by the lower layer block of the network and the decoder to obtain the first foreground and background segmentation result.

[0062] In some exemplary embodiments, the foreground and background segmentation module 302 is specifically used to perform feature compression and aggregation processing on the second feature data to obtain N third feature data in the following manner: performing global average pooling processing on the second feature data to obtain global information; performing convolution operation and activation function operation on the global information with the kth convolution kernel to obtain the kth third feature data; where k is an integer greater than or equal to 1 and less than or equal to N.

[0063] In some exemplary embodiments, the foreground and background segmentation module 302 is specifically used to calculate the channel adaptive weights by means of a self-attention mechanism on N third feature data: calculate the k-th normalized weight of the k-th third feature data according to the k-th third feature data; determine the channel adaptive weights according to the 1st normalized weight to the Nth normalized weight.

[0064] In some exemplary embodiments, the foreground and background segmentation module 302 is specifically used to calculate the k-th normalized weight of the k-th third feature data based on the k-th third feature data in the following manner: based on the k-th third feature data according to formula X k =σ(W2{δ(W1{x k})}) Calculate the k-th normalized weight; where X k Let x be the k-th normalized weight, σ() be the first activation function, W1{} be the first convolution function, W2{} be the second convolution function, δ() be the second activation function, and x be the k-th normalized weight. kThis is the kth third feature data.

[0065] In some exemplary embodiments, N is 3, and the foreground and background segmentation module 302 is specifically used to determine the channel adaptive weights based on the first to Nth normalized weights in the following manner: performing dot product operation, scale scaling and normalization on the first and second normalized weights to obtain the channel autocorrelation weight matrix; and performing dot product operation on the channel autocorrelation weight matrix and the third normalized weight to obtain the channel adaptive weights.

[0066] In some exemplary embodiments, the foreground and background segmentation module 302 is specifically used to obtain the channel autocorrelation weight matrix by performing dot product operation, scale scaling, and normalization on the first normalized weight and the second normalized weight: according to the formula... The channel autocorrelation weight matrix is ​​calculated; where X T d1 is the channel autocorrelation weight matrix, softmax() is the third activation function, X1 is the first normalized weight, X2 is the second normalized weight, and d2 is the scale ratio.

[0067] The specific implementation process of the foreground and background segmentation device described above is the same as that of the foreground and background segmentation method in the aforementioned embodiments, and will not be repeated here.

[0068] It will be understood by those skilled in the art that all or some of the steps, systems, or apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software may be distributed on a computer-readable medium, which may include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0069] Example embodiments have been disclosed herein, and while specific terminology has been used, it is for illustrative purposes only and should be construed as such, and is not intended to be limiting. In some instances, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in connection with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in connection with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of this application as set forth by the appended claims.

Claims

1. A foreground / background segmentation method, comprising: Obtain the first feature data of the image data to be segmented; The first feature data is compressed and aggregated using the encoder in the trained foreground-background segmentation network to obtain the third feature data; wherein, the foreground-background segmentation network includes: an encoder, a network bottom layer block and a decoder; The encoder uses a self-attention mechanism to calculate the channel adaptive weights from the third feature data. The encoder calculates the fourth feature data based on the channel adaptive weights and the first feature data. The fourth feature data obtained after the first feature data is processed by the encoder is input into the network bottom block, and the network bottom block and the decoder process it to obtain the first foreground and background segmentation result. The encoder includes at least one encoding block, and each encoding block performs the following processing on the input second feature data: The second feature data is subjected to feature compression and aggregation to obtain N third feature data; where N is an integer greater than or equal to 3; The channel adaptive weights are calculated from the N third feature data using a self-attention mechanism. The fourth feature data is calculated based on the channel adaptive weights and the second feature data; Wherein, when the coding block is the first coding block, the second feature data is the first feature data; when the coding block is the i-th coding block, where i is an integer greater than or equal to 2, the second feature data is the fourth feature data output by the (i-1)-th coding block. The step of inputting the fourth feature data obtained after processing the first feature data by the encoder into the lower layer block of the network, and then processing it by the lower layer block of the network and the decoder to obtain the first foreground and background segmentation result includes: The fourth feature data output from the last coding block is input into the network's bottom layer block, and processed by the network's bottom layer block and the decoder to obtain the first foreground and background segmentation result.

2. The foreground and background segmentation method according to claim 1, before obtaining the first feature data of the image data to be segmented, the method further includes: Obtain the training image dataset; Enhanced preprocessing is performed on the training image dataset; The foreground and background segmentation network is trained using the enhanced preprocessed training image dataset.

3. The foreground / background segmentation method according to claim 2, wherein, The enhanced preprocessing of the training image dataset includes: Foreground and background segmentation is performed on each training image in the training image dataset to obtain a second foreground and background segmentation result. The second foreground and background segmentation results are labeled on the training image data.

4. The foreground / background segmentation method according to claim 1, wherein, The process of compressing and aggregating the second feature data to obtain N third feature data includes: Global average pooling is performed on the second feature data to obtain global information; The global information is convolved with the k-th convolution kernel and then activated to obtain the k-th third feature data; where k is an integer greater than or equal to 1 and less than or equal to N.

5. The foreground / background segmentation method according to claim 1, wherein, The process of calculating the channel adaptive weights from the N third feature data using a self-attention mechanism includes: Calculate the k-th normalized weight of the k-th third feature data based on the k-th third feature data; The channel adaptive weights are determined based on the first normalized weights to the Nth normalized weights.

6. The foreground / background segmentation method according to claim 5, wherein, According to the kth third feature data, according to formula X k =σ(W2{δ(W1{x k })})Calculate the k-th normalized weight; Among them, X k Let be the k-th normalized weight, σ() be the first activation function, W1{} be the first convolution function, W2{} be the second convolution function, δ() be the second activation function, and x be the k-th normalized weight. k This refers to the kth third feature data.

7. The foreground / background segmentation method according to claim 5, wherein, When N is 3, determining the channel adaptive weights based on the first to Nth normalized weights includes: The channel autocorrelation weight matrix is ​​obtained by performing dot product operation, scaling, and normalization on the first and second normalized weights. The channel adaptive weights are obtained by performing a dot product operation on the channel autocorrelation weight matrix and the third normalized weight.

8. The foreground / background segmentation method according to claim 7, wherein, The step of performing a dot product operation, scaling, and normalization on the first and second normalized weights to obtain the channel autocorrelation weight matrix includes: According to the first normalized weight and the second normalized weight, according to the formula The channel autocorrelation weight matrix is ​​calculated. Among them, X T Let be the channel autocorrelation weight matrix, softmax() be the third activation function, X1 be the first normalized weight, X2 be the second normalized weight, and d2 be the scale ratio.

9. An electronic device, comprising: At least one processor; A memory, wherein at least one program is stored thereon, which, when executed by the at least one processor, implements the foreground / background segmentation method according to any one of claims 1-8.

10. A computer-readable medium storing a computer program that, when executed by a processor, implements the foreground / background segmentation method according to any one of claims 1-8.

Images