Image processing method and device, computer device and storage medium

Abstract

This application relates to an image processing method, apparatus, computer device, storage medium, and computer program product. The method includes: acquiring a depth feature matrix and a semantic feature matrix corresponding to an image to be segmented; obtaining a front-lighting matrix corresponding to the image to be segmented based on the semantic feature matrix; the elements in the front-lighting matrix represent the probability that a corresponding pixel in the image to be front-lighted; fusing the depth feature matrix and the semantic feature matrix to obtain a fusion matrix; obtaining a back-lighting matrix and a shadow matrix corresponding to the image to be segmented based on the fusion matrix; and dividing the image to be segmented into at least one region based on the front-lighting matrix, back-lighting matrix, and shadow matrix, wherein the at least one region includes at least one of a front-lighting region, a back-lighting region, and a shadow region. This method can improve the accuracy of image segmentation.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to an image processing method, apparatus, computer equipment, storage medium, and computer program product. Background Technology

[0002] With the development of computer technology, image processing technology has emerged. In the process of image processing, it is necessary to divide the image into regions and then apply different processing methods to different regions to improve the image clarity.

[0003] In traditional techniques, images are divided into shadow and non-shadow regions based on the brightness values ​​of pixels and a threshold. This simple division method results in low accuracy. Summary of the Invention

[0004] Therefore, it is necessary to provide an image processing method, apparatus, computer device, computer-readable storage medium, and computer program product that can improve the accuracy of image segmentation in response to the above-mentioned technical problems.

[0005] Firstly, this application provides an image processing method. The method includes:

[0006] Obtain the depth feature matrix and semantic feature matrix corresponding to the image to be segmented;

[0007] Based on the semantic feature matrix, the front lighting matrix corresponding to the image to be segmented is obtained; the elements in the front lighting matrix represent the probability that the corresponding pixel in the image to be segmented is a front lighting point.

[0008] The depth feature matrix and the semantic feature matrix are fused to obtain a fusion matrix. Based on the fusion matrix, the backlight matrix and shadow matrix corresponding to the image to be segmented are obtained. The elements in the backlight matrix represent the backlight probability of the corresponding pixel in the image to be segmented being a backlight point. The elements in the shadow matrix represent the shadow probability of the corresponding pixel in the image to be segmented being a shadow point.

[0009] Based on the aforementioned front lighting matrix, back lighting matrix, and shadow matrix, the image to be segmented is divided into at least one region, wherein the at least one region includes at least one of a front lighting region, a back lighting region, and a shadow region.

[0010] In one embodiment, dividing the image to be segmented into at least one region based on the front lighting matrix, back lighting matrix, and shadow matrix includes:

[0011] Obtain the position information of the target pixels in the image to be segmented;

[0012] Based on the location information, the front lighting probability corresponding to the target pixel is obtained from the front lighting matrix, the back lighting probability corresponding to the target pixel is obtained from the back lighting matrix, and the shadow probability corresponding to the target pixel is obtained from the shadow matrix.

[0013] The category of the target pixel is determined based on the probability of front lighting, back lighting, and shadow corresponding to the target pixel;

[0014] Based on the category corresponding to each pixel in the image to be segmented, the image to be segmented is divided into at least one region.

[0015] In one embodiment, determining the category of the target pixel based on the probability of front lighting, backlighting, and shadow corresponding to the target pixel includes:

[0016] The probability of direct sunlight is compared with a first threshold to obtain a first comparison result;

[0017] The backlight probability is compared with a second threshold to obtain a second comparison result;

[0018] The shadow probability is compared with a third threshold to obtain a third comparison result;

[0019] Based on the first comparison result, the second comparison result, and the third comparison result, the category of the target pixel is determined.

[0020] In one embodiment, the probability of front lighting, the probability of backlighting, and the probability of shadow are category probabilities; determining the category of the target pixel based on the first comparison result, the second comparison result, and the third comparison result includes:

[0021] If among the first comparison result, the second comparison result, and the third comparison result, there is only one comparison result that is greater than, then the category corresponding to the category probability of the comparison result being greater than is determined as the category of the target pixel.

[0022] If at least two of the first, second, and third comparison results are greater than, then the largest category probability is obtained from the category probabilities of the comparison results being greater than, and the category corresponding to the largest category probability is determined as the category of the target pixel.

[0023] In one embodiment, obtaining the front lighting matrix corresponding to the image to be segmented based on the semantic feature matrix includes:

[0024] The semantic feature matrix is ​​input into the first segmentation neural network model;

[0025] Based on the semantic feature matrix, the probability of each pixel in the image to be front-lit is calculated by the first segmentation neural network model. Based on the probability of front-lit for each pixel in the image to be segmented, the front-lit matrix corresponding to the image to be segmented is obtained.

[0026] In one embodiment, fusing the depth feature matrix and the semantic feature matrix to obtain a fusion matrix, and obtaining the backlight matrix and shadow matrix corresponding to the image to be segmented based on the fusion matrix, includes:

[0027] Subtracting the deep feature matrix from the semantic feature matrix yields the fusion matrix;

[0028] The fusion matrix is ​​input into the second segmentation neural network model;

[0029] Based on the fusion matrix, the backlight probability of each pixel in the image to be segmented being a backlight point and the shadow probability of each pixel being a shadow point are calculated using the second segmentation neural network model. Based on the backlight probability of each pixel in the image to be segmented, the backlight matrix corresponding to the image to be segmented is obtained, and based on the shadow probability of each pixel in the image to be segmented, the shadow matrix corresponding to the image to be segmented is obtained.

[0030] In one embodiment, obtaining the depth feature matrix and semantic feature matrix corresponding to the image to be segmented includes:

[0031] Obtain the image to be segmented;

[0032] Based on the depth information of the pixels in the image to be divided, depth features are extracted from the image to be divided to obtain the depth feature matrix corresponding to the image to be divided.

[0033] Based on the semantic information of the pixels in the image to be divided, semantic features are extracted from the image to be divided to obtain the semantic feature matrix corresponding to the image to be divided.

[0034] Secondly, this application also provides an image processing apparatus. The apparatus includes:

[0035] The acquisition module is used to acquire the depth feature matrix and semantic feature matrix corresponding to the image to be segmented;

[0036] The first prediction module is used to obtain the front lighting matrix corresponding to the image to be segmented based on the semantic feature matrix; the elements in the front lighting matrix represent the probability that the corresponding pixel in the image to be segmented is a front lighting point.

[0037] The second prediction module is used to fuse the depth feature matrix and the semantic feature matrix to obtain a fusion matrix, and based on the fusion matrix, to obtain the backlight matrix and shadow matrix corresponding to the image to be segmented; the elements in the backlight matrix represent the backlight probability of the corresponding pixel in the image to be segmented being a backlight point; the elements in the shadow matrix represent the shadow probability of the corresponding pixel in the image to be segmented being a shadow point.

[0038] The segmentation module is used to divide the image to be segmented into at least one region based on the front lighting matrix, back lighting matrix, and shadow matrix, wherein the at least one region includes at least one of the front lighting region, back lighting region, and shadow region.

[0039] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0040] Obtain the depth feature matrix and semantic feature matrix corresponding to the image to be segmented;

[0041] Based on the semantic feature matrix, the front lighting matrix corresponding to the image to be segmented is obtained; the elements in the front lighting matrix represent the probability that the corresponding pixel in the image to be segmented is a front lighting point.

[0042] The depth feature matrix and the semantic feature matrix are fused to obtain a fusion matrix. Based on the fusion matrix, the backlight matrix and shadow matrix corresponding to the image to be segmented are obtained. The elements in the backlight matrix represent the backlight probability of the corresponding pixel in the image to be segmented being a backlight point. The elements in the shadow matrix represent the shadow probability of the corresponding pixel in the image to be segmented being a shadow point.

[0043] Based on the aforementioned front lighting matrix, back lighting matrix, and shadow matrix, the image to be segmented is divided into at least one region, wherein the at least one region includes at least one of a front lighting region, a back lighting region, and a shadow region.

[0044] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0045] Obtain the depth feature matrix and semantic feature matrix corresponding to the image to be segmented;

[0046] Based on the semantic feature matrix, the front lighting matrix corresponding to the image to be segmented is obtained; the elements in the front lighting matrix represent the probability that the corresponding pixel in the image to be segmented is a front lighting point.

[0047] The depth feature matrix and the semantic feature matrix are fused to obtain a fusion matrix. Based on the fusion matrix, the backlight matrix and shadow matrix corresponding to the image to be segmented are obtained. The elements in the backlight matrix represent the backlight probability of the corresponding pixel in the image to be segmented being a backlight point. The elements in the shadow matrix represent the shadow probability of the corresponding pixel in the image to be segmented being a shadow point.

[0048] Based on the aforementioned front lighting matrix, back lighting matrix, and shadow matrix, the image to be segmented is divided into at least one region, wherein the at least one region includes at least one of a front lighting region, a back lighting region, and a shadow region.

[0049] The aforementioned image processing method, apparatus, computer equipment, storage medium, and computer program product acquire the depth feature matrix and semantic feature matrix corresponding to the image to be segmented. Based on the semantic feature matrix, a front lighting matrix corresponding to the image to be segmented is obtained. The depth feature matrix and semantic feature matrix are fused to obtain a fusion matrix. Based on the fusion matrix, a backlight matrix and a shadow matrix corresponding to the image to be segmented are obtained. Based on the front lighting matrix, backlight matrix, and shadow matrix, the image to be segmented is divided into at least one region, which includes at least one of a front lighting region, a backlight region, and a shadow region. The front lighting matrix is ​​obtained based on the semantic feature matrix corresponding to the image to be segmented. The semantic feature matrix contains rich semantic features of the image to be processed, improving the accuracy of the front lighting matrix. The depth feature matrix and semantic feature matrix are fused to obtain a fusion matrix, which contains both semantic and depth features of the image to be segmented. Based on the fusion matrix, a backlight matrix and a shadow matrix are obtained, improving the accuracy of the backlight matrix and shadow matrix. Based on the front lighting matrix, backlight matrix, and shadow matrix, the category corresponding to each pixel in the image to be segmented is determined. The image to be segmented is then segmented according to the category corresponding to each pixel, making the segmentation of image regions accurate to the pixel level, thus improving the accuracy of image segmentation. Attached Figure Description

[0050] Figure 1 This is an application environment diagram of an image processing method in one embodiment;

[0051] Figure 2 This is a flowchart illustrating an image processing method in one embodiment;

[0052] Figure 3 This is a flowchart illustrating the image segmentation steps in one embodiment;

[0053] Figure 4 This is a flowchart illustrating the steps for obtaining the backlight matrix and shadow matrix in one embodiment;

[0054] Figure 5 This is a flowchart illustrating the steps for obtaining the depth feature matrix and semantic feature matrix in one embodiment;

[0055] Figure 6 This is a flowchart illustrating an image segmentation method in one embodiment;

[0056] Figure 7 Here is a block diagram of the image partitioning network in one embodiment;

[0057] Figure 8 This is a structural block diagram of an image processing device in one embodiment;

[0058] Figure 9 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0060] The image processing method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated on server 104 or placed on a cloud or other network server. Both the terminal and the server can be used independently to execute the image processing method provided in this embodiment. The terminal and server can also be used collaboratively to execute the image processing method provided in this embodiment. For example, a computer device obtains the depth feature matrix and semantic feature matrix corresponding to the image to be segmented, obtains the front lighting matrix corresponding to the image to be segmented based on the semantic feature matrix, fuses the depth feature matrix and semantic feature matrix to obtain a fusion matrix, obtains the backlight matrix and shadow matrix corresponding to the image to be segmented based on the fusion matrix, and divides the image to be segmented into at least one region based on the front lighting matrix, backlight matrix, and shadow matrix. The at least one region includes at least one of a front lighting region, a backlight region, and a shadow region. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can be smart speakers, smart TVs, smart air conditioners, smart vehicle devices, etc. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted devices. Server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0061] In one embodiment, such as Figure 2As shown, an image processing method is provided. This method can be applied to a computer device, which can be a terminal or a server. The method can be executed independently by the terminal or server, or it can be implemented through interaction between the terminal and the server. This embodiment illustrates the application of this method to a computer device, including steps 202 to 208.

[0062] Step 202: Obtain the depth feature matrix and semantic feature matrix corresponding to the image to be segmented.

[0063] The image to be segmented refers to the image from which regions will be divided. This image can be a single image or multiple images. The depth feature matrix is ​​a matrix containing image depth information. Depth information refers to the distance information between pixels in the image to be segmented and the camera. The depth feature matrix can be a two-dimensional or three-dimensional matrix, but a three-dimensional depth feature matrix is ​​generally used to improve the accuracy of image segmentation. The semantic feature matrix is ​​a matrix containing semantic information of the image to be segmented. Semantic information includes, but is not limited to, color information, brightness information, saturation information, and hue information. The semantic feature matrix can also be a two-dimensional or three-dimensional matrix, but a three-dimensional semantic feature matrix is ​​generally used to improve the accuracy of image segmentation.

[0064] For example, a computer device acquires an image to be segmented, and then acquires the depth feature matrix and semantic feature matrix corresponding to the image to be segmented.

[0065] In one embodiment, a computer device acquires an image to be segmented and then inputs the image to be segmented into a feature extraction neural network model. The feature extraction neural network model extracts the depth features and semantic features of the image to be segmented to obtain a three-dimensional depth feature matrix and a three-dimensional semantic feature matrix corresponding to the image to be segmented.

[0066] In one embodiment, a computer device acquires an image to be segmented, and then inputs the image to be segmented into a semantic neural network model and a deep neural network model. The semantic neural network model extracts semantic features from the image to be segmented to obtain a semantic feature matrix corresponding to the image to be segmented, and the deep neural network model extracts depth features from the image to be segmented to obtain a depth feature matrix corresponding to the image to be segmented.

[0067] Step 204: Based on the semantic feature matrix, obtain the front lighting matrix corresponding to the image to be segmented; the elements in the front lighting matrix represent the probability that the corresponding pixel in the image to be segmented is a front lighting point.

[0068] The front-lighting matrix is ​​a matrix composed of the probabilities of each pixel being illuminated by the light. It can be understood that the number of elements in the front-lighting matrix is ​​the same as the number of pixels in the image to be segmented, and each element represents the probability that a pixel in the image is illuminated by the light. For example, if the image to be segmented contains 256 pixels, with 16 pixels per row and 16 rows in total, then the front-lighting matrix is ​​a 16×16 two-dimensional matrix. A pixel is a pixel whose probability of being illuminated by the light is greater than a threshold. The probability of being illuminated by the light is the probability that a pixel is illuminated by the light. The probability of being illuminated by the light is a decimal or percentage between 0 and 1. The higher the probability of being illuminated by the light, the greater the likelihood that the pixel is illuminated by the light. The front-lighting matrix is ​​a two-dimensional matrix.

[0069] For example, the computer device processes the semantic feature matrix to obtain the front lighting matrix corresponding to the image to be segmented, wherein the elements in the front lighting matrix represent the probability that the corresponding pixel in the image to be segmented is a front lighting point.

[0070] In one embodiment, the computer device calculates the probability that each pixel in the image to be segmented is a front-lit point based on the semantic feature matrix, and generates a front-lit matrix corresponding to the image to be segmented based on the position of the pixel and the corresponding front-lit probability of the pixel.

[0071] In one embodiment, the computer device inputs the semantic feature matrix into the front-lighting neural network model to obtain the front-lighting matrix corresponding to the image to be segmented.

[0072] Step 206: The depth feature matrix and the semantic feature matrix are fused to obtain a fusion matrix. Based on the fusion matrix, the backlight matrix and shadow matrix corresponding to the image to be segmented are obtained. The elements in the backlight matrix represent the backlight probability of the corresponding pixel in the image to be segmented being a backlight point. The elements in the shadow matrix represent the shadow probability of the corresponding pixel in the image to be segmented being a shadow point.

[0073] The backlight matrix is ​​a matrix composed of the backlight probabilities corresponding to each pixel. It can be understood that the number of elements in the backlight matrix is ​​the same as the number of pixels in the image to be segmented, and each element in the backlight matrix represents the probability that a corresponding pixel in the image is a backlight point. The backlight matrix is ​​a two-dimensional matrix. The shadow matrix is ​​a matrix composed of the shadow probabilities corresponding to each pixel. The shadow matrix is ​​also a two-dimensional matrix.

[0074] For example, the computer device fuses the depth feature matrix and the semantic feature matrix to obtain a fusion matrix, and then obtains the backlight matrix and shadow matrix corresponding to the image to be segmented based on the fusion matrix. The elements in the backlight matrix represent the probability that a corresponding pixel in the image to be segmented is a backlight point, and the elements in the shadow matrix represent the probability that a corresponding pixel in the image to be segmented is a shadow point.

[0075] In one embodiment, the computer device adds the depth feature matrix to the semantic feature matrix to obtain a fusion matrix. Based on the fusion matrix, the probability of each pixel in the image to be segmented being a backlight point is calculated to obtain the backlight matrix corresponding to the image to be segmented. Then, based on the fusion matrix, the probability of each pixel in the image to be segmented being a shadow point is calculated to obtain the shadow matrix corresponding to the image to be segmented.

[0076] In one embodiment, the computer device inputs the fusion matrix into the backlight / shadow prediction neural network model to obtain the backlight matrix and shadow matrix corresponding to the image to be segmented.

[0077] Step 208: Based on the front lighting matrix, back lighting matrix, and shadow matrix, divide the image to be segmented into at least one region, where the at least one region includes at least one of the front lighting region, back lighting region, and shadow region.

[0078] In this context, the front-lit region refers to the region composed of front-lit points in the image to be segmented. The back-lit region refers to the region composed of back-lit points in the image to be segmented. The shadow region refers to the region composed of shadow points in the image to be segmented.

[0079] For example, a computer device divides an image to be segmented based on a front lighting matrix, a back lighting matrix, and a shadow matrix, and the segmentation result includes at least one region among the front lighting region, the back lighting region, and the shadow region.

[0080] In one embodiment, a computer device acquires target pixels in an image to be segmented, obtains the front lighting probability, back lighting probability, and shadow probability corresponding to the target pixels from the front lighting matrix, back lighting matrix, and shadow matrix, respectively, compares the front lighting probability, back lighting probability, and shadow probability, obtains the maximum probability among the front lighting probability, back lighting probability, and shadow probability, takes the category corresponding to the maximum probability as the category corresponding to the target pixel, and segments the image to be segmented based on the category corresponding to each pixel in the image to be segmented. The segmentation result includes at least one region among the front lighting region, back lighting region, and shadow region.

[0081] In the above image processing method, a front lighting matrix is ​​obtained based on the semantic feature matrix corresponding to the image to be segmented. The semantic feature matrix contains rich semantic features of the image to be processed, which improves the accuracy of the front lighting matrix. The depth feature matrix and the semantic feature matrix are fused to obtain a fusion matrix, which contains the semantic and depth features of the image to be segmented. The backlight matrix and shadow matrix are obtained based on the fusion matrix, which improves the accuracy of the backlight matrix and shadow matrix. Based on the front lighting matrix, backlight matrix, and shadow matrix, the category corresponding to each pixel in the image to be segmented is determined. The image to be segmented is segmented according to the category corresponding to each pixel, and the segmentation of the image region is accurate to the pixel, which improves the accuracy of image segmentation.

[0082] In one embodiment, such as Figure 3 As shown, based on the front lighting matrix, back lighting matrix, and shadow matrix, dividing the image to be segmented into at least one region includes:

[0083] Step 302: Obtain the position information of the target pixels in the image to be segmented.

[0084] In this context, a pixel is an indivisible unit or element in an image. It can be understood as an image composed of multiple pixels, with each pixel being the smallest unit that makes up the image. Position information refers to the location of a pixel within the image. Position information can be represented by a number or two-dimensional coordinates. For example, if an image consists of 256 pixels, and each pixel is numbered from left to right and from top to bottom, then the position information of each pixel can be represented by its pixel number.

[0085] For example, a computer device obtains a pixel from a plurality of pixels in an image to be segmented, uses the obtained pixel as a target pixel, and then obtains the position information of the target pixel.

[0086] Step 304: Based on the location information, obtain the front lighting probability corresponding to the target pixel from the front lighting matrix, obtain the back lighting probability corresponding to the target pixel from the back lighting matrix, and obtain the shadow probability corresponding to the target pixel from the shadow matrix.

[0087] For example, the computer device obtains the front lighting probability corresponding to the position information from the front lighting matrix, the back lighting probability corresponding to the position information from the back lighting matrix, and the shadow probability corresponding to the position information from the shadow matrix.

[0088] Step 306: Determine the category of the target pixel based on the probability of front lighting, back lighting, and shadow corresponding to the target pixel.

[0089] Here, "category" refers to pixel attributes. Categories include front-lit pixels, back-lit pixels, and shadow pixels. Categories can be divided according to actual needs.

[0090] For example, the computer device determines the category of the target pixel based on the probability of front lighting, back lighting, and shadow corresponding to the target pixel.

[0091] In one embodiment, the computer device compares the probability of front lighting, backlighting, and shadow corresponding to the target pixel with preset threshold values. If only one of the probabilities of front lighting, backlighting, and shadow is greater than the preset threshold value, the category corresponding to the probability greater than the preset threshold value is taken as the category of the target pixel. If two or three of the probabilities of front lighting, backlighting, and shadow are greater than the preset threshold value, the two or three probabilities greater than the preset threshold value are compared, the maximum probability among the two or three probabilities is determined, and the category corresponding to the maximum probability is taken as the category of the target pixel.

[0092] Step 308: Based on the category corresponding to each pixel in the image to be segmented, divide the image to be segmented into at least one region.

[0093] In this context, a region refers to an image patch composed of pixels of the same category.

[0094] For example, a computer device divides an image into segments based on the category corresponding to each pixel in the image to be segmented, and the segmentation result includes at least one region.

[0095] In one embodiment, the computer device identifies each type of pixel, for example, identifying front-lit pixels as 1, backlit pixels as 1, and shadow pixels as 3, and grouping pixels with the same identification in the image into a region, the resulting partitioning includes at least one region.

[0096] In this embodiment, based on the front lighting matrix, backlighting matrix, and shadow matrix, the category corresponding to each pixel in the image to be segmented is determined. The image to be segmented is then segmented according to the category corresponding to each pixel, thus making the segmentation of the image region accurate to the pixel level and improving the accuracy of image segmentation.

[0097] In one embodiment, determining the category of a target pixel based on its probability of being illuminated by sunlight, backlit by sunlight, and in shadow includes:

[0098] The probability of front lighting is compared with a first threshold to obtain a first comparison result; the probability of backlighting is compared with a second threshold to obtain a second comparison result; the probability of shadow is compared with a third threshold to obtain a third comparison result; based on the first comparison result, the second comparison result, and the third comparison result, the category of the target pixel is determined.

[0099] The first threshold refers to the minimum probability that a pixel is a front-lit point. The second threshold refers to the minimum probability that a pixel is a backlit point. The third threshold refers to the minimum probability that a pixel is a shadow point. The first, second, and third thresholds can be set according to the actual situation and can be different or the same. The first comparison result, the second comparison result, and the third comparison result are all greater than, less than, or equal to one of the following:

[0100] For example, the computer device compares the probability of the target pixel being illuminated with a first threshold to obtain a first comparison result, compares the probability of the target pixel being backlit with a second threshold to obtain a second comparison result, compares the probability of the target pixel being shadowed with a third threshold to obtain a third comparison result, and then determines the category of the target pixel based on the first comparison result, the second comparison result, and the third comparison result.

[0101] In this embodiment, the category corresponding to each pixel in the image to be segmented is determined based on the probability of front lighting, back lighting, and shadow. The image to be segmented is then segmented according to the category corresponding to each pixel, thus making the segmentation of the image region accurate to the pixel level and improving the accuracy of image segmentation.

[0102] In one embodiment, the probability of front lighting, the probability of backlighting, and the probability of shadow are category probabilities; determining the category of the target pixel based on the first comparison result, the second comparison result, and the third comparison result includes:

[0103] If only one of the first, second, and third comparison results is greater than, then the category corresponding to the category probability of the comparison result being greater than is determined as the category of the target pixel. If at least two of the first, second, and third comparison results are greater than, then the category probability with the highest comparison result being greater than is obtained, and the category corresponding to the highest category probability is determined as the category of the target pixel.

[0104] For example, the computer device judges the first comparison result, the second comparison result, and the third comparison result. If there is only one comparison result that is greater than the first comparison result, the category corresponding to the category probability that is greater than the comparison result is determined to be the category of the target pixel. If there are at least two comparison results that are greater than the first comparison result, the largest category probability is obtained from the category probabilities that are greater than the comparison results, and the category corresponding to the largest category probability is determined to be the category of the target pixel.

[0105] In this embodiment, based on the first comparison result, the second comparison result, and the third comparison result, the category corresponding to each pixel in the image to be divided is determined, and the image to be divided is divided according to the category corresponding to each pixel, thereby making the division of image regions accurate to the pixel and improving the accuracy of image division.

[0106] In one embodiment, the forward-lighting matrix corresponding to the image to be segmented is obtained based on the semantic feature matrix, including:

[0107] The semantic feature matrix is ​​input into the first segmentation neural network model; based on the semantic feature matrix, the first segmentation neural network model calculates the probability that each pixel in the image to be segmented is a front-lit point, and based on the front-lit probability of each pixel in the image to be segmented, the front-lit matrix corresponding to the image to be segmented is obtained.

[0108] The first segmentation neural network model refers to a model that predicts the probability that a pixel in the image to be a front-lit point is a front-lit point. It can be understood as a pre-trained neural network model that can accurately estimate the probability that each pixel is a front-lit point.

[0109] For example, the computer device inputs the semantic feature matrix into the first segmentation neural network model, the first segmentation neural network model calculates the probability that each pixel in the image to be segmented is a front-lit point, and obtains the front-lit matrix corresponding to the image to be segmented based on the front-lit probability of each pixel in the image to be segmented.

[0110] In one embodiment, the computer device inputs the semantic feature matrix into a first segmentation neural network model. The first segmentation neural network model calculates the probability that each pixel in the image to be segmented is a front-lit point. The front-lit probabilities of each pixel in the image to be segmented are arranged according to the position information of each pixel to obtain the front-lit matrix corresponding to the image to be segmented.

[0111] In this embodiment, the first segmentation neural network model calculates the probability that each pixel in the image to be a front-lit point is a front-lit point. The first segmentation neural network model is a pre-trained neural network model, which can accurately calculate the probability that a pixel is a front-lit point, thus improving the accuracy of the probability of a pixel corresponding to the front-lit point. Based on the probability of the front-lit point of each pixel in the image to be segmented, the front-lit matrix corresponding to the image to be segmented is obtained, thus improving the accuracy of the front-lit matrix.

[0112] In one embodiment, such as Figure 4 As shown, the depth feature matrix and semantic feature matrix are fused to obtain a fusion matrix. Based on the fusion matrix, the inflection matrix and shadow matrix corresponding to the image to be segmented are obtained, including:

[0113] Step 402: Subtract the deep feature matrix from the semantic feature matrix to obtain the fusion matrix.

[0114] For example, a computer device subtracts a deep feature matrix from a semantic feature matrix to obtain a fusion matrix.

[0115] In one embodiment, the computer device performs a dot product operation on the semantic feature matrix and the deep feature matrix to obtain a fusion matrix.

[0116] Step 404: Input the fusion matrix into the second segmentation neural network model.

[0117] The second segmentation neural network model refers to a model that predicts the probability of a pixel in the image being a backlight point and the probability of a pixel being a shadow point. It can be understood as a pre-trained neural network model that can accurately estimate the probability of each pixel being a front-lit point and the probability of each pixel being a shadow point.

[0118] For example, the computer device inputs the fusion matrix into the second segmentation neural network model.

[0119] Step 406: Based on the fusion matrix, the backlight probability of each pixel in the image to be segmented as a backlight point and the shadow probability of each pixel as a shadow point are calculated through the second segmentation neural network model. Based on the backlight probability of each pixel in the image to be segmented, the backlight matrix corresponding to the image to be segmented is obtained, and based on the shadow probability of each pixel in the image to be segmented, the shadow matrix corresponding to the image to be segmented is obtained.

[0120] For example, the second segmentation neural network model calculates the backlight probability of each pixel in the image to be segmented as a backlight point and the shadow probability of each pixel as a shadow point based on the fusion matrix. Then, it obtains the backlight matrix corresponding to the image to be segmented based on the backlight probability of each pixel in the image to be segmented, and obtains the shadow matrix corresponding to the image to be segmented based on the shadow probability of each pixel in the image to be segmented.

[0121] In this embodiment, the second segmentation neural network model calculates the backlight matrix and shadow matrix of the image to be segmented based on the fusion matrix. The fusion matrix contains the depth information and semantic information of the image to be segmented. Based on the depth information and semantic features, the trained second segmentation neural network model is used to obtain the backlight matrix and shadow matrix, thereby improving the accuracy of the backlight matrix and shadow matrix.

[0122] In one embodiment, such as Figure 5 As shown, obtaining the depth feature matrix and semantic feature matrix corresponding to the image to be segmented includes:

[0123] Step 502: Obtain the image to be segmented.

[0124] For example, a computer device acquires an image to be segmented.

[0125] In one embodiment, a computer device acquires a target video, identifies a target object in the target video, extracts video frames containing the target object, and uses the video frames as images to be segmented.

[0126] In one embodiment, a computer device acquires an original image, preprocesses the original image, including but not limited to cropping and scaling, to obtain an image to be segmented.

[0127] Step 504: Based on the depth information of the pixels in the image to be divided, perform depth feature extraction on the image to be divided to obtain the depth feature matrix corresponding to the image to be divided.

[0128] For example, a computer device extracts features from the depth information of the image to be segmented to obtain a depth feature matrix corresponding to the image to be segmented.

[0129] In one embodiment, the computer device performs feature extraction on the depth information of the image to be segmented a preset number of times. Each feature extraction yields a two-dimensional depth feature matrix. Based on the preset number of two-dimensional depth feature matrices, a three-dimensional depth feature matrix corresponding to the image to be segmented is obtained.

[0130] Step 506: Based on the semantic information of the pixels in the image to be divided, semantic features are extracted from the image to be divided to obtain the semantic feature matrix corresponding to the image to be divided.

[0131] For example, a computer device extracts features from the semantic information of the image to be segmented to obtain a semantic feature matrix corresponding to the image to be segmented.

[0132] In one embodiment, the computer device extracts features from multiple semantic features of the image to be segmented, obtaining a two-dimensional semantic feature matrix corresponding to each semantic feature. Based on the multiple two-dimensional semantic feature matrices corresponding to the semantic features, a three-dimensional semantic feature matrix corresponding to the image to be segmented is obtained.

[0133] In this embodiment, a depth feature matrix and a semantic feature matrix are obtained based on the image to be segmented. The depth feature matrix contains rich depth features of the image to be segmented, and the semantic feature matrix contains rich semantic features of the image to be segmented, providing basic data for image segmentation and improving the accuracy of image segmentation.

[0134] In an exemplary embodiment, in checkpoints, electronic police systems, and large-scale scene monitoring, the main target objects in the image may be in various states such as front lighting, backlighting, and shadow. The target object appears brighter in front lighting and darker in backlighting or shadow, resulting in the target object not being clearly displayed in the image, leading to poor image quality and affecting subsequent functional processing. To improve image quality, it is necessary to divide the image into front lighting, backlighting, and shadow regions. The flowchart for image division is as follows... Figure 6 As shown, the image segmentation process used the following... Figure 7 The image segmentation network shown includes a feature extraction module, a depth estimation module, a front lighting segmentation module, a fusion module, and a backlight / shadow segmentation module.

[0135] First, the image segmentation network was trained. 8000 sets of RGB-D images of checkpoints, electronic police vehicles, and large-scene surveillance cameras were collected using a depth camera under various lighting conditions (front lighting, backlighting, and no direct sunlight), including images of areas with direct light, shadow, and backlight. A segmentation annotation tool was used to label the front-lit, shadow, and backlit regions in each of these 8000 sets of images. 2000 sets were randomly selected from these 8000 sets as the validation set, and the remaining 6000 sets were used as the training set. The front-lit segmentation module and the backlight / shadow segmentation module used the Dice loss function, a set similarity metric typically used to calculate the similarity between two samples, with values ​​ranging from [0,1]. The image segmentation network was trained using the 6000 sets of data until the Dice loss function converged. Then, the trained image segmentation network was validated using the 2000 sets of data, resulting in the trained image segmentation network.

[0136] The computer device acquires the original image, preprocesses the original image to obtain the image to be segmented. The preprocessing can be cropping, scaling or other processing methods, and the preprocessing is set according to the actual needs. Then the image to be segmented is input into the trained image segmentation network.

[0137] The image segmentation network acquires the image to be segmented. The feature extraction module extracts features from the image to be segmented, obtaining a feature matrix containing information about the image. This feature matrix is ​​then input into the front-lighting segmentation module to obtain the front-lighting matrix and the 3D semantic feature matrix. The 3D semantic feature matrix can be understood as intermediate parameters obtained during the front-lighting segmentation module's calculation. The feature matrix is ​​then input into the depth estimation module to obtain the corresponding depth image and the 3D depth feature matrix. The 3D depth feature matrix can also be understood as intermediate parameters obtained during the depth estimation module's calculation. The fusion module subtracts the 3D depth feature matrix from the 3D semantic feature matrix to obtain a fusion matrix. This fusion matrix is ​​then input into the backlighting / shadow segmentation module to obtain the backlighting matrix and the shadow matrix.

[0138] The computer device obtains the front-light probability of the target pixel from the front-light matrix, the back-light probability of the target pixel from the back-light matrix, and the shadow probability of the target pixel from the shadow matrix. It compares the front-light probability of the target pixel with a first threshold to obtain a first comparison result, compares the back-light probability of the target pixel with a second threshold to obtain a second comparison result, and compares the shadow probability of the target pixel with a third threshold to obtain a third comparison result. If only one of the first, second, and third comparison results is greater than a certain threshold, the category corresponding to the category with the greater-than-threshold comparison result is determined as the category of the target pixel. If at least two of the first, second, and third comparison results are greater than a certain threshold, the category with the highest greater-than-threshold comparison result is obtained, and the category corresponding to the highest-than-threshold comparison result is determined as the category of the target pixel.

[0139] Based on the category corresponding to each pixel in the image to be segmented, the image is segmented. The segmentation result includes at least one region: front lighting region, backlighting region, and shadow region, resulting in a segmented image. Different image processing methods are applied to different regions in the segmented image, and the processed image is output.

[0140] In this embodiment, the depth feature matrix and semantic feature matrix corresponding to the image to be segmented are obtained. Based on the semantic feature matrix, the front lighting matrix corresponding to the image to be segmented is obtained. The depth feature matrix and semantic feature matrix are fused to obtain a fusion matrix. Based on the fusion matrix, the backlight matrix and shadow matrix corresponding to the image to be segmented are obtained. Based on the front lighting matrix, backlight matrix and shadow matrix, the image to be segmented is divided into at least one region. The at least one region includes at least one of the front lighting region, backlight region and shadow region. The front lighting matrix is ​​obtained based on the semantic feature matrix corresponding to the image to be segmented. The semantic feature matrix contains rich semantic features of the image to be processed, which improves the accuracy of the front lighting matrix. The depth feature matrix and the semantic feature matrix are fused to obtain a fusion matrix. The fusion matrix contains the semantic and depth features of the image to be segmented. The backlight matrix and shadow matrix are obtained based on the fusion matrix, which improves the accuracy of the backlight matrix and shadow matrix. Based on the front lighting matrix, backlight matrix, and shadow matrix, the category corresponding to each pixel in the image to be segmented is determined. The image to be segmented is segmented according to the category corresponding to each pixel, and the segmentation of image regions is accurate to the pixel point, which improves the accuracy of image segmentation. Different image processing methods are applied to different regions in the segmented image to improve the image clarity.

[0141] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0142] Based on the same inventive concept, this application also provides an image processing apparatus for implementing the image processing method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more image processing apparatus embodiments provided below can be found in the limitations of the image processing method described above, and will not be repeated here.

[0143] In one embodiment, such as Figure 8 As shown, an image processing apparatus is provided, including: an acquisition module 802, a first prediction module 804, a second prediction module 806, and a segmentation module 808, wherein:

[0144] The acquisition module 802 is used to acquire the depth feature matrix and semantic feature matrix corresponding to the image to be segmented;

[0145] The first prediction module 804 is used to obtain the front lighting matrix corresponding to the image to be segmented based on the semantic feature matrix; the elements in the front lighting matrix represent the probability that the corresponding pixel in the image to be segmented is a front lighting point.

[0146] The second prediction module 806 is used to fuse the depth feature matrix and the semantic feature matrix to obtain a fusion matrix. Based on the fusion matrix, the backlight matrix and shadow matrix corresponding to the image to be segmented are obtained. The elements in the backlight matrix represent the backlight probability of the corresponding pixel in the image to be segmented being a backlight point. The elements in the shadow matrix represent the shadow probability of the corresponding pixel in the image to be segmented being a shadow point.

[0147] The segmentation module 808 is used to divide the image to be segmented into at least one region based on the front lighting matrix, the back lighting matrix, and the shadow matrix. The at least one region includes at least one of the front lighting region, the back lighting region, and the shadow region.

[0148] In one embodiment, the segmentation module 808 is further configured to: obtain the position information of target pixels in the image to be segmented; based on the position information, obtain the front lighting probability corresponding to the target pixel from the front lighting matrix, obtain the back lighting probability corresponding to the target pixel from the back lighting matrix, and obtain the shadow probability corresponding to the target pixel from the shadow matrix; determine the category of the target pixel based on the front lighting probability, back lighting probability, and shadow probability corresponding to the target pixel; and divide the image to be segmented into at least one region based on the category corresponding to each pixel in the image to be segmented.

[0149] In one embodiment, the segmentation module 808 is further configured to: compare the probability of front lighting with a first threshold to obtain a first comparison result; compare the probability of backlighting with a second threshold to obtain a second comparison result; compare the probability of shadow with a third threshold to obtain a third comparison result; and determine the category of the target pixel based on the first comparison result, the second comparison result, and the third comparison result.

[0150] In one embodiment, the segmentation module 808 is further configured to: if only one of the first comparison results, the second comparison result, and the third comparison result is greater than, then determine the category corresponding to the category probability of the category with the greater than comparison result as the category of the target pixel; if at least two of the first comparison results, the second comparison result, and the third comparison result are greater than, then obtain the largest category probability from the category probabilities of the category with the greater than comparison result, and determine the category corresponding to the largest category probability as the category of the target pixel.

[0151] In one embodiment, the first prediction module 804 is further configured to: input the semantic feature matrix into the first segmentation neural network model; calculate the probability of each pixel in the image to be segmented being a front-lit point through the first segmentation neural network model based on the semantic feature matrix; and obtain the front-lit matrix corresponding to the image to be segmented based on the probability of each pixel in the image to be segmented.

[0152] In one embodiment, the second prediction module 806 is further configured to: subtract the depth feature matrix from the semantic feature matrix to obtain a fusion matrix; input the fusion matrix into a second segmentation neural network model; based on the fusion matrix, calculate the backlight probability of each pixel in the image to be segmented as a backlight point and the shadow probability of each pixel as a shadow point through the second segmentation neural network model; obtain the backlight matrix corresponding to the image to be segmented based on the backlight probability of each pixel in the image to be segmented; and obtain the shadow matrix corresponding to the image to be segmented based on the shadow probability of each pixel in the image to be segmented.

[0153] In one embodiment, the acquisition module 802 is further configured to: acquire the image to be segmented; extract depth features from the image to be segmented based on the depth information of the pixels in the image to be segmented to obtain a depth feature matrix corresponding to the image to be segmented; and extract semantic features from the image to be segmented based on the semantic information of the pixels in the image to be segmented to obtain a semantic feature matrix corresponding to the image to be segmented.

[0154] Each module in the aforementioned image processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0155] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 9 As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements an image processing method. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0156] Those skilled in the art will understand that Figure 9 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0157] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0158] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0159] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0160] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0161] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0162] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0163] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. An image processing method, characterized by, The method includes: Obtain the depth feature matrix and semantic feature matrix corresponding to the image to be segmented; The semantic feature matrix is ​​input into the first segmentation neural network model; based on the semantic feature matrix, the first segmentation neural network model calculates the probability that each pixel in the image to be segmented is a front-lit point, and based on the front-lit probability of each pixel in the image to be segmented, the front-lit matrix corresponding to the image to be segmented is obtained; the elements in the front-lit matrix represent the probability that the corresponding pixel in the image to be segmented is a front-lit point. Subtracting the depth feature matrix from the semantic feature matrix yields a fusion matrix; the fusion matrix is ​​then input into a second segmentation neural network model; based on the fusion matrix, the second segmentation neural network model calculates the backlight probability of each pixel in the image to be segmented as a backlight point and the shadow probability of each pixel as a shadow point. Based on the backlight probabilities of each pixel in the image to be segmented, a backlight matrix corresponding to the image to be segmented is obtained; based on the shadow probabilities of each pixel in the image to be segmented, a shadow matrix corresponding to the image to be segmented is obtained. Elements in the backlight matrix represent the backlight probability of a corresponding pixel in the image to be segmented as a backlight point; elements in the shadow matrix represent the shadow probability of a corresponding pixel in the image to be segmented as a shadow point. Based on the aforementioned front lighting matrix, back lighting matrix, and shadow matrix, the image to be segmented is divided into at least one region, wherein the at least one region includes at least one of a front lighting region, a back lighting region, and a shadow region.

2. The method of claim 1, wherein, The step of dividing the image to be segmented into at least one region based on the front lighting matrix, back lighting matrix, and shadow matrix includes: Obtain the position information of the target pixels in the image to be segmented; Based on the location information, the front lighting probability corresponding to the target pixel is obtained from the front lighting matrix, the back lighting probability corresponding to the target pixel is obtained from the back lighting matrix, and the shadow probability corresponding to the target pixel is obtained from the shadow matrix. The category of the target pixel is determined based on the probability of front lighting, back lighting, and shadow corresponding to the target pixel; Based on the category corresponding to each pixel in the image to be segmented, the image to be segmented is divided into at least one region.

3. The method of claim 2, wherein, The process of determining the category of the target pixel based on its corresponding front lighting probability, backlighting probability, and shadow probability includes: The probability of direct sunlight is compared with a first threshold to obtain a first comparison result; The backlight probability is compared with a second threshold to obtain a second comparison result; The shadow probability is compared with a third threshold to obtain a third comparison result; Based on the first comparison result, the second comparison result, and the third comparison result, the category of the target pixel is determined.

4. The method of claim 3, wherein, The probabilities of front lighting, backlighting, and shadow are category probabilities; determining the category of the target pixel based on the first comparison result, the second comparison result, and the third comparison result includes: If among the first comparison result, the second comparison result, and the third comparison result, there is only one comparison result that is greater than, then the category corresponding to the category probability of the comparison result being greater than is determined as the category of the target pixel. If at least two of the first, second, and third comparison results are greater than, then the largest category probability is obtained from the category probabilities of the comparison results being greater than, and the category corresponding to the largest category probability is determined as the category of the target pixel.

5. The method of claim 1, wherein, The step of obtaining the front lighting matrix corresponding to the image to be segmented based on the front lighting probability of each pixel in the image to be segmented includes: The front lighting probability of each pixel in the image to be segmented is arranged according to the position information of each pixel to obtain the front lighting matrix corresponding to the image to be segmented.

6. The method of claim 1, wherein, The process of obtaining the depth feature matrix and semantic feature matrix corresponding to the image to be segmented includes: The image to be segmented is input into a semantic neural network model and a deep neural network model. The semantic neural network model extracts semantic features from the image to be segmented to obtain a semantic feature matrix corresponding to the image to be segmented. The deep neural network model extracts depth features from the image to be segmented to obtain a depth feature matrix corresponding to the image to be segmented.

7. The method of claim 1, wherein, The process of obtaining the depth feature matrix and semantic feature matrix corresponding to the image to be segmented includes: Obtain the image to be segmented; Based on the depth information of the pixels in the image to be divided, depth features are extracted from the image to be divided to obtain the depth feature matrix corresponding to the image to be divided. Based on the semantic information of the pixels in the image to be divided, semantic features are extracted from the image to be divided to obtain the semantic feature matrix corresponding to the image to be divided.

8. An image processing apparatus, characterized in that, The device includes: The acquisition module is used to acquire the depth feature matrix and semantic feature matrix corresponding to the image to be segmented; The first prediction module is used to input the semantic feature matrix into the first segmentation neural network model; based on the semantic feature matrix, the first segmentation neural network model calculates the probability that each pixel in the image to be segmented is a front-lit point, and based on the front-lit probability of each pixel in the image to be segmented, obtains the front-lit matrix corresponding to the image to be segmented; the elements in the front-lit matrix represent the probability that the corresponding pixel in the image to be segmented is a front-lit point. The second prediction module is used to subtract the depth feature matrix from the semantic feature matrix to obtain a fusion matrix; input the fusion matrix into a second segmentation neural network model; based on the fusion matrix, through the second segmentation neural network model, calculate the backlight probability of each pixel in the image to be segmented as a backlight point and the shadow probability of each pixel as a shadow point; based on the backlight probability of each pixel in the image to be segmented, obtain the backlight matrix corresponding to the image to be segmented; based on the shadow probability of each pixel in the image to be segmented, obtain the shadow matrix corresponding to the image to be segmented; the elements in the backlight matrix represent the backlight probability of the corresponding pixel in the image to be segmented as a backlight point; the elements in the shadow matrix represent the shadow probability of the corresponding pixel in the image to be segmented as a shadow point. The segmentation module is used to divide the image to be segmented into at least one region based on the front lighting matrix, back lighting matrix, and shadow matrix, wherein the at least one region includes at least one of the front lighting region, back lighting region, and shadow region.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images